Bug #28649 MySQLd node disconnects from NDBD Cluster

• Submitted: 24 May 2007 11:08
• Modified: 30 May 2007 4:34
• Reporter: Kris Buytaert (Candidate Quality Contributor)
• Status: No Feedback
• Category: MySQL Cluster: Cluster (NDB) storage engine
• Severity: S2 (Serious)
• Version: 5.1.12
• OS: Linux (Centos 4.X)
• Assigned to: Assigned Account
• CPU Architecture: Any
• Tags: cluster

[24 May 2007 11:08] Kris Buytaert

Description:
mysqld(API) node disconnects from cluster with no logging in ndb_mgm log 

070524  0:25:02 [ERROR] Error 626 in readAutoIncrementValue(): Tuple did not exist
[val2-root@DB-A mysql]# more DB-A.err  | grep  readA | wc -l 
11010

And after a while also 

070524  7:40:33 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ccmt/blade.frm'
070524  7:40:41 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ccmt/blade.frm'
070524  7:40:52 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ccmt/blade.frm'

(for almost all ndb tables) 

Restarting mysqld reconnects to the cluster and shows no errors 
070524 11:03:00 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ccmt/blade.frm'
070524 11:03:06 [Note] /usr/sbin/mysqld: Normal shutdown

070524 11:03:07 [Note] SCHEDULER: Purging queue. 0 events
070524 11:03:07 [Note] Stopping Cluster Binlog
070524 11:03:07 [Note] Stopping Cluster Utility thread
070524 11:03:09 [Warning] /usr/sbin/mysqld: Forcing close of thread 576  user: 'pmt'

070524 11:03:13  InnoDB: Starting shutdown...
070524 11:03:14  InnoDB: Shutdown completed; log sequence number 0 46439
070524 11:03:14 [Note] /usr/sbin/mysqld: Shutdown complete

070524 11:03:15  mysqld ended

070524 11:03:15  mysqld started
070524 11:03:15  InnoDB: Started; log sequence number 0 46439
070524 11:03:16 [Note] Starting MySQL Cluster Binlog Thread
070524 11:03:16 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.12-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
070524 11:03:17 [Note] SCHEDULER: Loaded 0 events

How to repeat:
Not yet.

[24 May 2007 14:18] Kris Buytaert

http://bugs.mysql.com/bug.php?id=24839 Seems similar

[24 May 2007 14:38] Kris Buytaert

The errors are coming every 5 minutes.

070524 13:55:05 [ERROR] Error 626 in readAutoIncrementValue(): Tuple did not exist
070524 14:00:03 [ERROR] Error 626 in readAutoIncrementValue(): Tuple did not exist
070524 14:05:02 [ERROR] Error 626 in readAutoIncrementValue(): Tuple did not exist

But now I haven't seen the database crashing.

[24 May 2007 14:43] Kris Buytaert

mysql> show create table isplistforserviceproxy;

+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Table                  | Create Table                                                                                                                                                                                                                                                                                                          |

+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| isplistforserviceproxy | CREATE TABLE `isplistforserviceproxy` (

  `ServiceProviderID` int(11) NOT NULL,

  `ServiceProxyID` int(11) NOT NULL,

  PRIMARY KEY (`ServiceProviderID`,`ServiceProxyID`),

  KEY `FKCB0E855E5BAAA7E2` (`ServiceProviderID`),

  KEY `FKCB0E855E3DFB5172` (`ServiceProxyID`)

) ENGINE=ndbcluster DEFAULT CHARSET=latin1 | 

+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

1 row in set (0.07 sec)

mysql> SHOW TABLE STATUS LIKE 'isplistforserviceproxy';

+------------------------+------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+-------------+-------------+------------+-------------------+----------+----------------+-----------------------+

| Name                   | Engine     | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation         | Checksum | Create_options | Comment               |

+------------------------+------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+-------------+-------------+------------+-------------------+----------+----------------+-----------------------+

| isplistforserviceproxy | NDBCLUSTER |      10 | Fixed      |   10 |             24 |      131072 |               0 |            0 |         0 |           NULL | NULL        | NULL        | NULL       | latin1_swedish_ci |     NULL |                | number_of_replicas: 2 | 

+------------------------+------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+-------------+-------------+------------+-------------------+----------+----------------+-----------------------+

1 row in set (0.54 sec)

Results in : 

070524 14:40:07 [ERROR] Error 626 in readAutoIncrementValue(): Tuple did not exist

[24 May 2007 14:53] Kris Buytaert

Unlike the other bug we are running into this on a non empty table.

mysql> select count(*) from isplistforserviceproxy;

+----------+

| count(*) |

+----------+

|       10 | 

+----------+

1 row in set (0.02 sec)

mysql> select * from isplistforserviceproxy;

+-------------------+----------------+

| ServiceProviderID | ServiceProxyID |

+-------------------+----------------+

|                 1 |             10 | 

|                 1 |              8 | 

|                 2 |              7 | 

|                 2 |              8 | 

|                 1 |              1 | 

|                 1 |              9 | 

|                 1 |              7 | 

|                 2 |              9 | 

|                 2 |              1 | 

|                 2 |             10 | 

+-------------------+----------------+

10 rows in set (0.42 sec)

[30 May 2007 4:34] Adam Dixon

Hi Kris,
Are you still running 5.1.12 ? I am trying to reproduce this on the latest (5.1.20 from BK) however am not having any luck. Perhaps this particular problem has been fixed.

Are you able to upgrade to the latest beta release and try to reproduce from then?

Or if I have missed something in order to reproduce this, please let me know.