MySQL Bugs: #44735: The mysql cluster data node crash when dropping two big table

Bug #44735	The mysql cluster data node crash when dropping two big table
Submitted:	8 May 2009 3:50	Modified:	17 Oct 2009 12:39
Reporter:	Steven Fan	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Disk Data	Severity:	S3 (Non-critical)
Version:	mysql-5.1-telco-6.3	OS:	Linux (Linux acm-dev10-n1-int 2.6.15-1.2054_FC5smp #1 SMP Tue Mar 14 16:05:46 EST 2006 i686 i686 i386 GNU/L)
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	5.1.30-ndb-6.3.20-innodb, crash cluster

Description:
Hi,

I am doing the performance test to merge two tables data into one big table. The three tables has the same schema and they are disk-based table. But i find the mysql cluster data node down every time after i execute SQL to drop the old table after merging. 

The logis in the attachment.

How to repeat:
1. Create two disk-based tables with 2000000 records for each one. They have the same schema. (call_log_active_small_unsorted_2009050621, call_log_active_small_sorted_2009050618_2009050620_2)
2. Create the third table with the same schema. (call_log_tmp_sorted)
3. Execute "insert into third_table select ....." to copy the records from the first and second table to the third table.
4. Execute "rename table call_log_tmp_sorted to call_log_active_small_sorted_2009050618_2009050621_3"

"drop table call_log_active_small_unsorted_2009050621, call_log_active_small_sorted_2009050618_2009050620_2"
5. Then you find the connection loses when executing another SQL and you find the data node is down when checking the database status.

Suggested fix:
I don't know the root cause.

The log file

Attachment: log.txt (text/plain), 11.78 KiB.

Can you provide the failing data nodes error and trace log files?

And as the node fails with a segmentation fault signal: could you redo the test with ndbd processes started with the --core-file option and core files enabled in your system environment (e.g. "ulimit -c unlimited")? Seeing where the segfault signal was actually raised would help to track this one down ...

the failing data nodes error and trace log files

Attachment: ndb_2_out.zip (application/x-zip-compressed, text), 10.74 KiB.

the failing data nodes error and trace log files

Attachment: ndb_2_error.log (application/octet-stream, text), 2.02 KiB.

the failing data nodes error and trace log files

Attachment: mysqllog.zip (application/x-zip-compressed, text), 48.51 KiB.

Here is the core information:

GNU gdb Red Hat Linux (6.3.0.0-1.132.EL3rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Reading symbols from shared object read from target memory...(no debugging symbols found)...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Loaded system supplied DSO at 0xdb0000
Core was generated by `/opt/proxy/mysql/libexec/ndbd --core-file'.
Program terminated with signal 6, Aborted.
#0  0x00db0410 in __kernel_vsyscall ()
(gdb) bt
#0  0x00db0410 in __kernel_vsyscall ()
#1  0x00129159 in ?? ()
#2  0x0022fff4 in ?? ()
#3  0xb7ef86b0 in ?? ()
#4  0xbfd0bce8 in ?? ()
#5  0x0012a6e3 in ?? ()
#6  0x00000006 in ?? ()
#7  0xbfd0bc5c in ?? ()
#8  0x00000000 in ?? ()

Steven,

So your backtrace using the core-file doesn't show much. Have you built the binary yourself? In that case, could you please rebuild and enable debugging? 

Are you able to reproduce the crash with newer cluster version?

Can you upload the core file to ftp://ftp.mysql.com/pub/mysql/upload with bug44735 somewhere in the filename?

Can you tell us a bit more about your schema? What kind of attributes and indexes do you have on it? I'd like to see if it's possible to reproduce it on our site.

Could you please also provide your config.ini and my.cnf

/Gustaf

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".