Bug #7342 API died
Submitted: 16 Dec 2004 2:16 Modified: 22 Jan 2005 8:37
Reporter: Brian Leung Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:4.1.7-max OS:Linux (Red Hat Lunix Enterprise)
Assigned to: Assigned Account CPU Architecture:Any

[16 Dec 2004 2:16] Brian Leung
Description:
I installed the cluster 4.1.7 version and running in the following configuration :
machine a :
1 ndb_mgmd
1 ndbd
1 mysqld
machine b:
1 ndbd
1 mysqld

The mysqld daemon died suddenly in which I didnt perform any operation on the mysql at all.
 
The following is the error I got from the err log file :

Number of processes running now: 0
041215 16:37:53  mysqld restarted
041215 16:37:53 [Warning] Ignoring user change to 'ser=mysql' because the user was set to 'mysql' earlier on the command line

041215 16:37:53  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
041215 16:37:53  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 54372.
InnoDB: Doing recovery: scanned up to log sequence number 0 54372
InnoDB: Last MySQL binlog file position 0 79, file name ./dev2-bin.000069
041215 16:37:53  InnoDB: Flushing modified pages from the buffer pool...
041215 16:37:53  InnoDB: Started; log sequence number 0 54372
/usr/local/mysql/bin/mysqld: ready for connections.
Version: '4.1.7-max-log'  socket: '/tmp/mysql.sock'  port: 3306  Official MySQL-max binary
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=258048
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 92783 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8a27e90
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xaffa67dc, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8131043
0xb75c6e48
0x83972fb
0x83972fb
0x8397150
0x81b38b2
0x81b0c3e
0x81c24ad
0x8142830
0x8144b89
0x813f13f
0x813eac8
0x813e1f8
0xb75c0dac
0xb755fa8a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8a1cc48 = desc bboard
thd->thread_id=3
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
041215 16:43:56  mysqld restarted
041215 16:43:57 [Warning] Ignoring user change to 'ser=mysql' because the user was set to 'mysql' earlier on the command line

041215 16:43:57  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
041215 16:43:57  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 54404.
InnoDB: Doing recovery: scanned up to log sequence number 0 54404
InnoDB: Last MySQL binlog file position 0 79, file name ./dev2-bin.000070
041215 16:43:57  InnoDB: Flushing modified pages from the buffer pool...
041215 16:43:57  InnoDB: Started; log sequence number 0 54404
/usr/local/mysql/bin/mysqld: ready for connections.
Version: '4.1.7-max-log'  socket: '/tmp/mysql.sock'  port: 3306  Official MySQL-max binary

And I extracted the stack code and performed the stack dump, the following is the stack dump result :

0x8131043 handle_segfault + 423
0xb75c6e48 _end + -1359456064
0x83972fb startTransactionLocal__3NdbUiUi + 43
0x83972fb startTransactionLocal__3NdbUiUi + 43
0x8397150 startTransaction__3NdbUiPCcUi + 100
0x81b38b2 ndb_get_table_statistics__FP3NdbPCcPUxT2 + 250
0x81b0c3e info__13ha_ndbclusterUi + 70
0x81c24ad mysqld_show_fields__FP3THDP13st_table_listPCcb + 121
0x8142830 mysql_execute_command__FP3THD + 10204
0x8144b89 mysql_parse__FP3THDPcUi + 169
0x813f13f dispatch_command__F19enum_server_commandP3THDPcUi + 1643
0x813eac8 do_command__FP3THD + 188
0x813e1f8 handle_one_connection + 616
0xb75c0dac _end + -1359480796
0xb755fa8a _end + -1359878910

I wonder if my configuration is not correct or what is causing it and if I can prevent this to happen again.

How to repeat:
the config.ini is as follows :

[ndbd default]
NoOfReplicas= 2
MaxNoOfOrderedIndexes=1000
DataMemory=1024M
IndexMemory=512M

[mysqld default]
[ndb_mgmd default]
[tcp default]

[ndbd]
HostName= a
DataDir: /var/lib/mysql-cluster/

[ndbd]
HostName= b
DataDir: /var/lib/mysql-cluster/

[ndb_mgmd]
HostName= a

[mysqld]
[mysqld]
[mysqld]

No action is performed when it is crashed.
[22 Dec 2004 6:03] Jonas Oreland
Hi,

I tried to reproduce this on 4.1.8
But failed.
Could you retry it on 4.1.8 if check if you still get the same problem?

/Jonas
[22 Dec 2004 6:35] Brian Leung
Thanks, I'll go and try it. I'll let you know if this problem still exists. But is 4.1.8 is in production ?
[22 Dec 2004 8:37] Brian Leung
Hi,

I downloaded the mysql-max-4.1.8-pc-linux-i686 and installed on my server. I configured as same as the original setting. But I encounterd the following problems :
Shell> ndbd
Unable to connect with connect string: nodeid=0,10.30.103.108:2200
Retrying every 5 seconds. Attempts left: 12 11 10 9 8

I found out that it will not work if I connected the ndb with the ip address and port number 2200; except the local host ip 127.0.0.1. I had the file Ndb.cfg that contains one line as host=10.30.103.108:2200.

Please advice
[22 Dec 2004 9:08] Jonas Oreland
Hi,

The default port for ndb has changed between 4.1.7 and 4.1.8
See http://lists.mysql.com/cluster/1083

The port used by ndb_mgmd is printed in the cluster log.

/Jonas
[14 Feb 2005 22:54] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".