Bug #45119 Segmentation Fault on update of indexed column in mixed disk-memory column table
Submitted: 27 May 2009 8:44 Modified: 21 Oct 2009 9:30
Reporter: Nathan Thera Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Disk Data Severity:S3 (Non-critical)
Version:mysql-5.1-telco-7.06 OS:Linux
Assigned to: Assigned Account CPU Architecture:Any
Tags: mysql-5.1.32 ndb-7.0.6, segmentation fault ndb disk error 6000 ndbd.cpp
Triage: Triaged: D2 (Serious) / R6 (Needs Assessment) / E6 (Needs Assessment)

[27 May 2009 8:44] Nathan Thera
Description:
Hi 

I am updating the columns of ndb mixed disk-memory tables. When I run a query that changes the value of an indexed column one of the ndb data nodes will die.
The queries are in the format of:
UPDATE table SET indexedColumn = "value" WHERE id = "idnum"
This error did not happen when all columns of the tables are in memory.

I am running mysql-5.1.32 ndb-7.0.5b freshly inserted ndb cluster with 4 nodes in a 2 replica setup.

Please let me know if you require any additional logs or test situations.

Below is some output from the logs and the trace file attached in the next comment.

Nathan

From out log:
2009-05-26 14:00:12 [ndbd] INFO     -- Received signal 11. Running error handler.
2009-05-26 14:00:12 [ndbd] INFO     -- Signal 11 received; Segmentation fault
2009-05-26 14:00:12 [ndbd] INFO     -- ndbd.cpp
2009-05-26 14:00:12 [ndbd] INFO     -- Error handler signal restarting system
2009-05-26 14:00:12 [ndbd] INFO     -- Error handler shutdown completed - exiting
2009-05-26 14:00:14 [ndbd] ALERT    -- Node 4: Forced node shutdown completed, restarting. Initiated by signal 11. Caused by error 6000: 'Error OS signal received(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2009-05-26 14:00:14 [ndbd] INFO     -- Ndb has terminated (pid 25720) restarting
2009-05-26 14:00:18 [ndbd] INFO     -- Configuration fetched from '172.29.71.119:1186', generation: 4
2009-05-26 14:00:18 [ndbd] INFO     -- Angel pid: 25719 ndb pid: 10306
NDBMT: non-mt
2009-05-26 14:00:18 [ndbd] INFO     -- NDB Cluster -- DB node 4
2009-05-26 14:00:18 [ndbd] INFO     -- mysql-5.1.32 ndb-7.0.5-beta --
2009-05-26 14:00:18 [ndbd] INFO     -- WatchDog timer is set to 6000 ms
2009-05-26 14:00:18 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 6148Mb initial: 6532Mb
Adding 6532Mb to ZONE_LO (1,209023)
2009-05-26 14:00:20 [ndbd] INFO     -- Start initiated (mysql-5.1.32 ndb-7.0.5)

From error log: 
Time: Tuesday 26 May 2009 - 14:00:12
Status: Temporary error, restart node
Message: Error OS signal received (Internal error, programming error or missing error message, please report a bug)
Error: 6000
Error data: Signal 11 received; Segmentation fault
Error object: ndbd.cpp
Program: ndbd
Pid: 25720
Trace: /var/lib/mysql-cluster/ndb_4_trace.log.15
Version: mysql-5.1.32 ndb-7.0.5-beta
***EOM***

How to repeat:
1) Create ndb table with mixed disk and memory columns.
2) Create an index for one of the columns (memory column; did not test disk column)
3) Populate table with rows.
4) Change the value of the column with index.

One ndb data node dies at this point. Repeat until the cluster runs out of nodes to fully crash the nodes. The data node assigned Master is usually the one that dies.

Suggested fix:
-Only use memory columns for the table.
-Do not update columns with an index.
-Add new row with updated column value and remove the previous one.
[27 May 2009 8:46] Nathan Thera
trace log of the node that crashed

Attachment: ndb_4_trace.log.zip (application/x-zip-compressed, text), 57.12 KiB.

[6 Jun 2009 22:44] Sveta Smirnova
Thank you for the report.

Please provide output of SHOW CREATE TABLE table
[4 Jul 2009 5:26] Nathan Thera
Hi

I have updated trace logs and structure of the same issue running on 7.0.6.
This is from a freshly inserted NDB cluster.

Nathan

Out_log
2009-07-03 22:50:42 [ndbd] INFO     -- Received signal 11. Running error handler.
2009-07-03 22:50:43 [ndbd] INFO     -- Signal 11 received; Segmentation fault
2009-07-03 22:50:43 [ndbd] INFO     -- ndbd.cpp
2009-07-03 22:50:43 [ndbd] INFO     -- Error handler signal restarting system
2009-07-03 22:50:43 [ndbd] INFO     -- Error handler shutdown completed - exiting
2009-07-03 22:50:43 [ndbd] ALERT    -- Node 6: Forced node shutdown completed, restarting. Initiated by sign
al 11. Caused by error 6000: 'Error OS signal received(Internal error, programming error or missing error me
ssage, please report a bug). Temporary error, restart node'.
2009-07-03 22:50:43 [ndbd] INFO     -- Ndb has terminated (pid 11271) restarting
2009-07-03 22:50:54 [ndbd] INFO     -- Configuration fetched from '172.29.71.97:1186', generation: 1
2009-07-03 22:50:54 [ndbd] INFO     -- Angel pid: 22541 ndb pid: 14577
NDBMT: non-mt
2009-07-03 22:50:54 [ndbd] INFO     -- NDB Cluster -- DB node 6
2009-07-03 22:50:54 [ndbd] INFO     -- mysql-5.1.34 ndb-7.0.6 --
2009-07-03 22:50:54 [ndbd] INFO     -- WatchDog timer is set to 6000 ms
2009-07-03 22:50:54 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 7172Mb initial: 7556Mb
Adding 7556Mb to ZONE_LO (1,241791)
2009-07-03 22:50:56 [ndbd] INFO     -- Start initiated (mysql-5.1.34 ndb-7.0.6)

Error_log
Time: Friday 3 July 2009 - 22:50:42
Status: Temporary error, restart node
Message: Error OS signal received (Internal error, programming error or missing error message, please report a bug)
Error: 6000
Error data: Signal 11 received; Segmentation fault
Error object: ndbd.cpp
Program: ndbd
Pid: 11271
Trace: /var/lib/mysql-cluster/ndb_6_trace.log.8
Version: mysql-5.1.34 ndb-7.0.6
***EOM***
[4 Jul 2009 5:32] Nathan Thera
Trace file of the 7.0.6 crash

Attachment: ndb_6_trace.log.zip (application/x-zip-compressed, text), 57.84 KiB.

[20 Aug 2009 7:14] Jonas Oreland
Hi,

I tried with your table, and did not manage to reproduce.
Could you provide a dump of the table in question ?

/Jonas
[12 Sep 2009 0:07] Nathan Thera
Unfortunately the table is much too large to send.

I will see if I can create a smaller data set.
[21 Oct 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".