MySQL Bugs: #32849: mysql dies on slave during replication.

Bug #32849	mysql dies on slave during replication.
Submitted:	29 Nov 2007 15:48	Modified:	1 Jul 2010 20:07
Reporter:	Dimitrij HIlt	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Federated storage engine	Severity:	S2 (Serious)
Version:	5.0.32-Debian_7etch3-log, 5.0.54	OS:	Linux (Debian Etch/x86_64)
Assigned to:		CPU Architecture:	Any
Tags:	replication, Signal 11

Description:
During replication slave dies witch signal 11 daily and says:
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=2147483648
read_buffer_size=131072
max_used_connections=4
max_connections=2600
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 775473
1 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x11463400
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x4408b198, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
New value of fp=0x11463400 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2aaaac649be5 is invalid pointer
thd->thread_id=49
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

After that replication is brocken due loosed temporaery tables (one as MyISAM and one as FEDERATED).

How to repeat:
It's difficult. We have komplexly dataware house with lot of data (400GB) and all Engines (MyISAM, InnoDB, Federated)

Thank you for a problem report. Do you have any coredumps after these crashes?

no, no crash dumps because ulimit -c is 0. I will try to make a dump on next crash.

Dimi

Please, add a comment if you'll get any coredump. Also, if possible, try to repeat with a newer version, 5.0.45.

So. Now we have a core dump, but it is 11 GB big.

Here ist back trace:
(gdb) bt
#0  0x00002b09b8996797 in pthread_kill () from /lib/libpthread.so.0
#1  0x000000000059c575 in handle_segfault ()
#2  <signal handler called>
#3  0x0000000000852ee8 in my_lengthsp_8bit ()
#4  0x0000000000587ab7 in Field_enum::store ()
#5  0x00000000006bc5a9 in ha_federated::convert_row_to_internal_format ()
#6  0x00000000006bc69d in ha_federated::read_next ()
#7  0x00000000006bd8b3 in ha_federated::index_read_idx_with_result_set ()
#8  0x00000000006bd9e2 in ha_federated::index_read ()
#9  0x00000000005e3e1b in report_error ()
#10 0x00000000005f0403 in sub_select ()
#11 0x00000000005f0762 in sub_select_cache ()
#12 0x00000000005fac88 in JOIN::exec ()
#13 0x00000000005fc800 in mysql_select ()
#14 0x00000000005fd0b7 in handle_select ()
#15 0x00000000005b3dc2 in mysql_execute_command ()
#16 0x00000000005b5194 in mysql_parse ()
#17 0x000000000061e747 in Query_log_event::exec_event ()
#18 0x000000000068bb7f in handle_slave_sql ()
#19 0x00002b09b8992f1a in start_thread () from /lib/libpthread.so.0
#20 0x00002b09b92456c2 in clone () from /lib/libc.so.6
#21 0x0000000000000000 in ?? ()

On the Master tis query runs witchout any Problem.

Do you need anytink else from Core Dump?

Regards,

Dimi

Thank you for the feedback.

> On the Master tis query runs witchout any Problem.

Please provide this query and output of SHOW CREATE TABLE for all underlying tables.

See attachment table_schema_and_queries.txt

New coredump same issue:
Program terminated with signal 11, Segmentation fault.
#0  0x00002b677f2f0797 in pthread_kill () from /lib/libpthread.so.0
(gdb) bt
#0  0x00002b677f2f0797 in pthread_kill () from /lib/libpthread.so.0
#1  0x000000000059c575 in handle_segfault ()
#2  <signal handler called>
#3  0x0000000000852ee8 in my_lengthsp_8bit ()
#4  0x0000000000587ab7 in Field_enum::store ()
#5  0x00000000006bc5a9 in ha_federated::convert_row_to_internal_format ()
#6  0x00000000006bc69d in ha_federated::read_next ()
#7  0x00000000006bd8b3 in ha_federated::index_read_idx_with_result_set ()
#8  0x00000000006bd9e2 in ha_federated::index_read ()
#9  0x00000000005e3e1b in report_error ()
#10 0x00000000005f0403 in sub_select ()
#11 0x00000000005f0762 in sub_select_cache ()
#12 0x00000000005fac88 in JOIN::exec ()
#13 0x00000000005fc800 in mysql_select ()
#14 0x00000000005fd0b7 in handle_select ()
#15 0x00000000005b3dc2 in mysql_execute_command ()
#16 0x00000000005b5194 in mysql_parse ()
#17 0x000000000061e747 in Query_log_event::exec_event ()
#18 0x000000000068bb7f in handle_slave_sql ()
#19 0x00002b677f2ecf1a in start_thread () from /lib/libpthread.so.0
#20 0x00002b677fb9f6c2 in clone () from /lib/libc.so.6
#21 0x0000000000000000 in ?? ()

Regards,

Dimi

I'v installed now mysql-5.0.54 from debien backports and will say you if it does not crash.

Dimi

mysql 5.0.45 does not solve this issue.
May be a bug in federated engine?

Dimi

Today crashes the master host on same query. 

Dimitrij

May be realy a trouble with federated and temporaery table. After we switched the table FED_ARCHIVED from temporaery to 'normal' table type mysql does not dies anymore.

Regards,

Dimi

Thank you for the feedback.

I could not repeat described behavior using various tests. So closing as "Can't repeat", although stack trace looks similar to bug #41786 and can be same problem.