Bug #51083 Crashing with signal 11 and problems connecting
Submitted: 11 Feb 2010 10:12 Modified: 13 Feb 2010 11:42
Reporter: Peter Hansen Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.1.31 OS:Linux (ubuntu jaunty)
Assigned to: CPU Architecture:Any

[11 Feb 2010 10:12] Peter Hansen
Description:
Server is crashing, often i see this
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Error log shows many entries like this:

100211 10:01:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
100211 10:01:37  InnoDB: Started; log sequence number 0 1823069313
100211 10:01:37 [Note] Event Scheduler: Loaded 0 events
100211 10:01:37 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.31-1ubuntu2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
100211 10:01:47 [ERROR] /usr/sbin/mysqld: Table './virtualusers/dspam_token_data' is marked as crashed and should be repaired
100211 10:01:47 [Warning] Checking table:   './virtualusers/dspam_token_data'
100211 10:01:47 [Warning] Recovering table: './virtualusers/dspam_token_data'
100211 10:02:53 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=9
max_threads=300
threads_connected=4
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 672230 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x7f408c0c1d40
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f409a87b0d0 thread_stack 0x20000
/usr/sbin/mysqld(my_print_stacktrace+0x29) [0x8b4f89]
/usr/sbin/mysqld(handle_segfault+0x383) [0x5f8f03]
/lib/libpthread.so.0 [0x7f409a4bf080]
/lib/libc.so.6(memcpy+0xd2) [0x7f4098f9f092]
/usr/sbin/mysqld(Field_blob::pack(unsigned char*, unsigned char const*, unsigned int, bool)+0x9c) [0x5dc55c]
/usr/sbin/mysqld(ha_archive::pack_row(unsigned char*)+0xd6) [0x767d26]
/usr/sbin/mysqld(ha_archive::real_write_row(unsigned char*, azio_stream*)+0x1e) [0x767dde]
/usr/sbin/mysqld(ha_archive::optimize(THD*, st_ha_check_opt*)+0x131) [0x768a51]
/usr/sbin/mysqld(ha_archive::repair(THD*, st_ha_check_opt*)+0x14) [0x766da4]
/usr/sbin/mysqld(ha_archive::check_and_repair(THD*)+0x34) [0x767094]
/usr/sbin/mysqld [0x63f966]
/usr/sbin/mysqld(open_table(THD*, TABLE_LIST*, st_mem_root*, bool*, unsigned int)+0x626) [0x641e16]
/usr/sbin/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int)+0x5db) [0x6429cb]
/usr/sbin/mysqld(open_normal_and_derived_tables(THD*, TABLE_LIST*, unsigned int)+0x1e) [0x642b0e]
/usr/sbin/mysqld(get_all_tables(THD*, TABLE_LIST*, Item*)+0x962) [0x70e0f2]
/usr/sbin/mysqld(get_schema_tables_result(JOIN*, enum_schema_table_state)+0x1f7) [0x6fe1c7]
/usr/sbin/mysqld(JOIN::exec()+0x605) [0x66eba5]
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x124) [0x66acf4]
/usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x16c) [0x67080c]
/usr/sbin/mysqld [0x60522f]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x10a3) [0x607833]
/usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x212) [0x60c092]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xdcf) [0x60d57f]
/usr/sbin/mysqld(do_command(THD*)+0xe8) [0x60dda8]
/usr/sbin/mysqld(handle_one_connection+0x226) [0x601426]
/lib/libpthread.so.0 [0x7f409a4b73ba]
/lib/libc.so.6(clone+0x6d) [0x7f4099000fcd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7f408c043ee0 is an invalid pointer
thd->thread_id=1058
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100211 10:02:54 mysqld_safe Number of processes running now: 0
100211 10:02:54 mysqld_safe mysqld restarted
100211 10:02:54  InnoDB: Started; log sequence number 0 1823069313
100211 10:02:54 [Note] Event Scheduler: Loaded 0 events
100211 10:02:54 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.31-1ubuntu2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
100211 10:04:44 [ERROR] /usr/sbin/mysqld: Table './virtualusers/dspam_token_data' is marked as crashed and should be repaired
100211 10:04:44 [Warning] Checking table:   './virtualusers/dspam_token_data'
100211 10:04:44 [Warning] Recovering table: './virtualusers/dspam_token_data'

mysqlchk shows corrupted tables but can run long enough to complete.
tried to shutdown the mysql server and run myisamchk as the documentation suggested, but no luck. The server has been upgraded from 5.0 to 5.1.

How to repeat:
happens all the time at the moment, can point at anything specific
[11 Feb 2010 10:28] Sveta Smirnova
Thank you for the report.

This looks like duplicate of bug #32880 which was fixed in version 5.1.34. Please upgrade to current version 5.1.43, try with it and inform us if this solves the problem.
[11 Feb 2010 10:35] Peter Hansen
Hi, 
Thanks for your quick reply.

After looking at the bug, I cant tell if the data where recovered or not? Can you tell?
[11 Feb 2010 10:39] Sveta Smirnova
Thank you for the feedback.

After fixing bug #32880 repairing archive table started working again. If this would be recovered in your case depends from corruption which you experienced though. Most likely it will.
[11 Feb 2010 11:00] Peter Hansen
Okay, ive got newest version on a clone machine running now.

Seems not to be crashing now.

When running mysqlchk -Asrp --auto-repair 
dont seem to repair anything? Just tells me to Please do "REPAIR TABLE ...." ?
With "error: Corrupt"

Should it not repair? 

Thanks
[11 Feb 2010 11:02] Peter Hansen
seems only to be ARCHIVE tables that cant be fixed?
[11 Feb 2010 11:09] Peter Hansen
Manually running "REPAIR TABLE db.eventlog"

Just tells me to repair it using REPAIR TABLE... hmm?
[11 Feb 2010 17:00] Sveta Smirnova
Thank you for the feedback.

REPAIR works with ARCHIVE tables. But as I wrote already REPAIR would work only if it can repair a table. This depends from corruption. For example, if some external application erased part of data REPAIR could not do anything. If you think your case is different and REPAIR must repair the table please upload to our FTP server *ARZ, *ARM and *frm files
[12 Feb 2010 13:40] Peter Hansen
OK, upgraded the server and it seems to work.

But we had to delete all archive tables (including data - 300 tables ... )

But.. 5.0 had .ARM files with ARCHIVE tables but they are now deprecated? 
This then results in error #1010 cant drop database... (cant rmdir ... err 17)
Because those files are still present.. why dont mysql check for this before issuing a rmdir command?!

Anyway, can i safely delete ALL *.ARM files without any harm?
[13 Feb 2010 11:42] Sveta Smirnova
Thank you for the feedback.

Closing as duplicate of bug #32880 as you can not repeat crash after upgrade.

Regarding to your questions:

> Because those files are still present.. why dont mysql check for this before issuing a
rmdir command?!

This was done for purpose: MySQL server can not decide if user left such files per own purpose or just by mistake. So it prefers to delegate this action to the user.