Bug #48748 mysqld crash when I excute the update operation on master
Submitted: 13 Nov 2009 1:54 Modified: 22 Aug 2010 8:44
Reporter: run sun Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Replication Severity:S2 (Serious)
Version:mysql-5.1-telco-7.0 OS:Linux
Assigned to: CPU Architecture:Any
Tags: MySQL5.1.34 and ndb7.0.6

[13 Nov 2009 1:54] run sun
Description:
I setup a cluster with 2 mgmt, 4 ndb and 4 mysqld.
MySQL version is 5.1.34 and ndb version is 7.0.6
OS version is RHEL5.1

Now I enable log-bin on one of mysqld and want to configure it to master.
The command line is show below:

/opt/mysql/current/bin/mysqld --basedir=/opt/mysql/current --user=bloglines--datadir=/ask/bloglines/db/mysql-data --ndbcluster --ndb-connectstring=192.168.0.201,192.168.0.202 --server-id=1 --log-bin

But when I excute update operation via mysql client, it crash. Select operation is ok, and disable log-bin, all are ok.

Below is the mysql crash log:
@400000004ae563a532ee1024 091026 1:53:47 - mysqld got signal 11 ;
@400000004ae563a532ee1fc4 This could be because you hit a bug. It is also possible that this binary
@400000004ae563a532ee2b7c or one of the libraries it was linked against is corrupt, improperly built,
@400000004ae563a532ee3734 or misconfigured. This error can also be caused by malfunctioning hardware.
@400000004ae563a532ee3f04 We will try our best to scrape up some info that will hopefully help diagnose
@400000004ae563a532ee4abc the problem, but since we have already crashed, something is definitely wrong
@400000004ae563a532eee6fc and this may fail.
@400000004ae563a532eeeecc
@400000004ae563a532ef48a4 key_buffer_size=8384512
@400000004ae563a532ef5074 read_buffer_size=131072
@400000004ae563a532ef545c max_used_connections=3
@400000004ae563a532ef5c2c max_threads=151
@400000004ae563a532ef6014 threads_connected=2
@400000004ae563a532ef67e4 It is possible that mysqld could use up to
@400000004ae563a532ef739c key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337726 K
@400000004ae563a532ef7b6c bytes of memory
@400000004ae563a532ef7f54 Hope that's ok; if not, decrease some variables in the equation.
@400000004ae563a532efbdd4
@400000004ae563a532efc1bc thd: 0x9b01f50
@400000004ae563a532efc98c Attempting backtrace. You can use the following information to find out
@400000004ae563a532efd15c where mysqld died. If you see no messages after this, something went
@400000004ae563a532efdd14 terribly wrong...
@400000004ae563a532f03304 stack_bottom = 0xb09bb3a4 thread_stack 0x30000
@400000004ae563a532faed1c /opt/mysql/current/bin/mysqld(my_print_stacktrace+0x2d) [0x85bcd2d]
@400000004ae563a532fb79bc /opt/mysql/current/bin/mysqld(handle_segfault+0x381) [0x8247de1]
@400000004ae563a532fb818c [0x6be420]
@400000004ae563a532fb895c /lib/libpthread.so.0 [0x33143b]
@400000004ae563a532fb912c /lib/libc.so.6(clone+0x5e) [0x2b1fde]
@400000004ae563a532fb98fc Trying to get some variables.
@400000004ae563a532fb9ce4 Some pointers may be invalid and cause the dump to abort...
@400000004ae563a532fc15fc thd->query at (nil) is an invalid pointer
@400000004ae563a532fc1dcc thd->thread_id=1
@400000004ae563a532fc2984 thd->killed=NOT_KILLED
@400000004ae563a532fc2d6c The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
@400000004ae563a532fc3924 information that should help you find out what is causing the crash.
@400000004ae563a5348ea894 091026 1:53:47 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=bldev003mwh-bin' to avoid this problem.
@400000004ae563a53721e634 InnoDB: The log sequence number in ibdata files does not match
@400000004ae563a53721f5d4 InnoDB: the log sequence number in the ib_logfiles!
@400000004ae563a537229dcc 091026 1:53:47 InnoDB: Database was not shut down normally!
@400000004ae563a53722a984 InnoDB: Starting crash recovery.
@400000004ae563a53722b154 InnoDB: Reading tablespace information from the .ibd files...
@400000004ae563a5372fa5bc InnoDB: Restoring possible half-written data pages from the doublewrite
@400000004ae563a5372fb174 InnoDB: buffer...
@400000004ae563a6005cfea4 091026 1:53:48 InnoDB: Started; log sequence number 0 46409
@400000004ae563a603e0baac 091026 1:53:48 [Note] NDB: NodeID is 7, management server '10.76.96.201:1186'
@400000004ae563a703e08fb4 091026 1:53:49 [Note] NDB[0]: NodeID: 7, no storage nodes connected (timed out)
@400000004ae563a703e34ed4 091026 1:53:49 [Note] Starting Cluster Binlog Thread
@400000004ae563a703eb2e74 091026 1:53:49 [Note] Recovering after a crash using bldev003mwh-bin
@400000004ae563a703ebb72c 091026 1:53:49 [Note] Starting crash recovery...
@400000004ae563a703ec95d4 091026 1:53:49 [Note] Crash recovery finished.
@400000004ae563a70865298c 091026 1:53:49 [Note] Event Scheduler: Loaded 0 events
@400000004ae563a70865a68c 091026 1:53:49 [Note] /opt/mysql/current/bin/mysqld: ready for connections.
@400000004ae563a70865ae5c Version: '5.1.34-ndb-7.0.6-cluster-gpl-log' socket: '/tmp/mysql.sock' port: 3306 MySQL Cluster Server (GPL)
@400000004ae563a8040a5ed4 091026 1:53:50 [Note] NDB Binlog: Ndb tables initially read only.
@400000004ae563a9028d389c 091026 1:53:51 [Note] table './bloglines/unreads' opened read only
@400000004ae563aa094e704c 091026 1:53:52 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema
@400000004ae563aa097539fc 091026 1:53:52 [Note] NDB Binlog: logging ./mysql/ndb_schema (UPDATED,USE_WRITE)
@400000004ae563aa0bae788c 091026 1:53:52 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_apply_status
@400000004ae563aa1100da14 091026 1:53:52 [Note] NDB Binlog: logging ./mysql/ndb_apply_status (UPDATED,USE_WRITE)
@400000004ae563aa14a12e6c 091026 1:53:52 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$bloglines/unreads
@400000004ae563aa16f5b944 091026 1:53:52 [Note] NDB Binlog: logging ./bloglines/unreads (UPDATED,USE_WRITE)
@400000004ae563aa1d15c33c 2009-10-26 01:53:52 [NdbApi] INFO -- Flushing incomplete GCI:s < 26576/1
@400000004ae563aa1d1601bc 2009-10-26 01:53:52 [NdbApi] INFO -- Flushing incomplete GCI:s < 26576/1
@400000004ae563aa1d16674c 091026 1:53:52 [Note] NDB Binlog: starting log at epoch 26576/1
@400000004ae563aa1d16a1e4 091026 1:53:52 [Note] NDB Binlog: ndb tables writable

How to repeat:
You can set the same cluster, create a simple table, then insert more than 1 millions record into the table. Then you enable binlog on one mysqld, restart.
Then you can do update operation via mysql client.
[16 Nov 2009 14:19] Jørgen Austvik
Have you run an upgrade? Can you please supply the full error logs from mysqld?
[17 Nov 2009 0:55] run sun
Full mysqld error log

Attachment: bldev003.mysql.error (application/octet-stream, text), 5.28 KiB.

[17 Dec 2009 8:08] run sun
Any update?
[22 Feb 2010 19:33] Jonas Oreland
it looks like a charset issue,
1) have you compiled this your self, or is it a binary package
2) which options do you use to start mysqld ?
   (e.g you'r my.cnf)
3) do you disable/enable any charsets?

/Jonas
[2 Mar 2010 4:05] run sun
1. I use binary package
2. Options when start mysqld
   /opt/mysql/current/bin/mysqld --basedir=/opt/mysql/current
   --user=bloglines--datadir=/ask/bloglines/db/mysql-data --ndbcluster
   --ndb-connectstring=192.168.0.201,192.168.0.202 --server-id=1 --log-bin
3. I don't configure any charsets, may be they are set by default
[22 Jul 2010 8:44] Sveta Smirnova
Thank you for the feedback.

Please send us options you start MySQL Cluster with too.
[22 Aug 2010 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".