Bug #45203 Falcon crash noticed when executing random queries with replication enabled
Submitted: 29 May 2009 23:12 Modified: 5 Aug 2009 16:37
Reporter: Hema Sridharan Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: Partitions Severity:S2 (Serious)
Version:mysql-6.0-backup, mysql-6.0 OS:Linux
Assigned to: Assigned Account CPU Architecture:Any

[29 May 2009 23:12] Hema Sridharan
Description:
When executing random queries with falcon_replication using RQG load, mysqld crashed as follows (back trace may not be complete)

Program terminated with signal 6, Aborted.
#0  0x00000036a3c0b122 in pthread_kill () from /lib64/libpthread.so.0
#0  0x00000036a3c0b122 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000d8cdb7 in my_write_core (sig=6) at stacktrace.c:309
#2  0x0000000000731ad3 in handle_segfault (sig=6) at mysqld.cc:2713
#3  <signal handler called>
#4  0x00000036a3030045 in raise () from /lib64/libc.so.6
#5  0x00000036a3031ae0 in abort () from /lib64/libc.so.6
#6  0x00000036a3029756 in __assert_fail () from /lib64/libc.so.6
#7  0x0000000000897e82 in ha_partition::end_bulk_insert (this=0x9e42928, 
abort=false) at ha_partition.cc:3342
#8  0x00000000007f4f82 in handler::ha_end_bulk_insert (this=0x9e42928, 
abort=false) at handler.h:1609
#9  0x00000000007ed0e0 in select_insert::abort (this=0x9b9b368)
at sql_insert.cc:3324
#10 0x00000000007e8abc in handle_select (thd=0x9b7b4f0, lex=0x9b7ced8, 
result=0x9b9b368, setup_tables_done_option=1073741824) at sql_select.cc:316
#11 0x0000000000745e33 in mysql_execute_command (thd=0x9b7b4f0)
at sql_parse.cc:3282
#12 0x000000000074b1dc in mysql_parse (thd=0x9b7b4f0, 
inBuf=0x9c46898 "INSERT INTO `table10_falcon_key_pk_parts_2_int_autoinc` SELECT * FROM `table0_falcon` LIMIT 5", length=93, found_semicolon=0x4ec2cf20)
at sql_parse.cc:5937
#13 0x000000000074c44b in dispatch_command (command=COM_QUERY, thd=0x9b7b4f0, 
packet=0x9b86691 "INSERT INTO `table10_falcon_key_pk_parts_2_int_autoinc` SELECT * FROM `table0_falcon` LIMIT 5", packet_length=93) at sql_parse.cc:1049
#14 0x000000000074d8b3 in do_command (thd=0x9b7b4f0) at sql_parse.cc:731
#15 0x000000000073a8a4 in handle_one_connection (arg=0x9b7b4f0)
at sql_connect.cc:1146
#16 0x00000036a3c062e7 in start_thread () from /lib64/libpthread.so.0
#17 0x00000036a30ce3bd in clone () from /lib64/libc.so.6
Using host libthread_db library "/lib64/libthread_db.so.1".

mysqld: ha_partition.cc:3342: virtual int ha_partition::end_bulk_insert(bool): Assertion `bitmap_is_set(&m_bulk_insert_started, m_tot_parts)' failed.
090530  1:51:11 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=11
max_threads=151
thread_count=11
connection_count=11
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60735 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x9b7b4f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4ec2d0f0 thread_stack 0x40000
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(my_print_stacktrace+0x32) [0xd8cd0b]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(handle_segfault+0x2a6) [0x7318ea]
/lib64/libpthread.so.0 [0x36a3c0de60]
/lib64/libc.so.6(gsignal+0x35) [0x36a3030045]
/lib64/libc.so.6(abort+0x110) [0x36a3031ae0]
/lib64/libc.so.6(__assert_fail+0xf6) [0x36a3029756]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(ha_partition::end_bulk_insert(bool)+0x66) [0x897e82]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(handler::ha_end_bulk_insert(bool)+0x36) [0x7f4f82]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(select_insert::abort()+0x5e) [0x7ed0e0]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x246) [0x7e8abc]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(mysql_execute_command(THD*)+0x372d) [0x745e33]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x278) [0x74b1dc]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x939) [0x74c44b]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(do_command(THD*)+0x227) [0x74d8b3]
/export/home/tmp/wl4195c/mysql-6.0-backup/sql/mysqld(handle_one_connection+0x11a) [0x73a8a4]
/lib64/libpthread.so.0 [0x36a3c062e7]
/lib64/libc.so.6(clone+0x6d) [0x36a30ce3bd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9c46898 = INSERT INTO `table10_falcon_key_pk_parts_2_int_autoinc` SELECT * FROM `table0_falcon` LIMIT 5
thd->thread_id=10
thd->killed=NOT_KILLED

How to repeat:
perl runall.pl --basedir=/export/home/tmp/wl4195c/mysql-6.0-backup/ --grammar=/export/home/tmp/perl_new/mysql-test-extra-6.0/mysql-test/gentest/conf/falcon_replication.yy --gendata=/export/home/tmp/perl_new/mysql-test-extra-6.0/mysql-test/gentest/conf/falcon_replication.zz

Note that test will pass when --validator=Falcon option is included while execution.

The server crash is noticed in mysql-6.0 tree also.
[30 May 2009 5:30] Kevin Lewis
This is an assertion in ha_partition::end_bulk_insert.  Can a partitioning expert like Mattias look into this before it is assigned to Falcon?
[5 Aug 2009 16:37] Mattias Jonsson
Duplicate of bug#44108.