Bug #45614 Query crashes data nodes, sometimes all of them, sometimes just one.
Submitted: 19 Jun 2009 12:51 Modified: 22 Jul 2009 12:43
Reporter: John Sabo Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Disk Data Severity:S2 (Serious)
Version:mysql-5.1-telco-7.0 OS:Linux
Assigned to: CPU Architecture:Any
Tags: crash, dbtc/DbtcMain.cpp, ndb 7.0.6, NDBCNTR
Triage: Triaged: D1 (Critical) / R6 (Needs Assessment) / E6 (Needs Assessment)

[19 Jun 2009 12:51] John Sabo
Description:
We have a cron that updates a table that runs every morning.. Since we've upgraded to mysql-3.1.34_ndb-7.0.6 it's been crashing data nodes or even our entire cluster (all data nodes go poof)

I have 2 different errors for 2 diff types of crashes. Both with the same query.

Full crash (all data nodes go poof):

Time: Thursday 18 June 2009 - 05:00:26
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming
Error: 2341
Error data: dbtc/DbtcMain.cpp
Error object: DBTC (Line: 9344) 0x0000000e
Program: /usr/sbin/ndbd
Pid: 10357
Trace: /db/datadir/ndb_2_trace.log.1
Version: mysql-5.1.34 ndb-7.0.6
***EOM***

Single node crash:

Time: Friday 19 June 2009 - 05:00:22
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal e
Error: 2303
Error data: Node 2 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 260) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 21093
Trace: /db/datadir/ndb_2_trace.log.2
Version: mysql-5.1.34 ndb-7.0.6
***EOM***

Here's the query: 

INSERT INTO st_transactions_summary

SELECT

NULL,
t1.billing_company_id,
t1.transaction_type,
/* t1.account_id, */
/* t1.subaccount_id, */
t1.transaction_id,
t1.subscription_id,
t1.transaction_date,
t2.transaction_date,
/* t2.partner_id, */
t1.amount,
t2.amount,
t2.initial_period,
t2.recurring_amount,
t2.recurring_period,
'', /* member status */
t2.card_type,
t2.affiliate_type,
t1.commission,
t2.payment_type,
u.uid,
ur.sales_rep_id,
sub.sid

FROM st_transactions t1
   LEFT JOIN st_transactions t2
    ON(t1.subscription_id = t2.subscription_id AND t1.billing_company_id = t2.bi
   LEFT JOIN user_subscriptions sub
    ON (sub.vendor_subs_id = t1.subscription_id AND t1.billing_company_id = sub.
   LEFT JOIN users_revshare ur
    ON (sub.rsid = ur.rsid AND sub.rsid != '')
   LEFT JOIN users u
    ON (ur.uid = u.uid)

These crashes did not happen with 7.0.6. 

How to repeat:
Run query.
[19 Jun 2009 13:00] John Sabo
Description:
We have a cron that updates a table that runs every morning.. Since we've upgraded to
mysql-3.1.34_ndb-7.0.6 it's been crashing data nodes or even our entire cluster (all data
nodes go poof)

I have 2 different errors for 2 diff types of crashes. Both with the same query.

Full crash (all data nodes go poof):

Time: Thursday 18 June 2009 - 05:00:26
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtc/DbtcMain.cpp
Error object: DBTC (Line: 9344) 0x0000000e
Program: /usr/sbin/ndbd
Pid: 10357
Trace: /db/datadir/ndb_2_trace.log.1
Version: mysql-5.1.34 ndb-7.0.6
***EOM***

Single node crash:

Time: Friday 19 June 2009 - 05:00:22
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 2 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 260) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 21093
Trace: /db/datadir/ndb_2_trace.log.2
Version: mysql-5.1.34 ndb-7.0.6
***EOM***

Here's the query: 

INSERT INTO st_transactions_summary

SELECT

NULL,
t1.billing_company_id,
t1.transaction_type,
/* t1.account_id, */
/* t1.subaccount_id, */
t1.transaction_id,
t1.subscription_id,
t1.transaction_date,
t2.transaction_date,
/* t2.partner_id, */
t1.amount,
t2.amount,
t2.initial_period,
t2.recurring_amount,
t2.recurring_period,
'', /* member status */
t2.card_type,
t2.affiliate_type,
t1.commission,
t2.payment_type,
u.uid,
ur.sales_rep_id,
sub.sid

FROM st_transactions t1
   LEFT JOIN st_transactions t2
    ON(t1.subscription_id = t2.subscription_id AND t1.billing_company_id = t2.billing_company_id AND t2.transaction_type = 'new')
   LEFT JOIN user_subscriptions sub
    ON (sub.vendor_subs_id = t1.subscription_id AND t1.billing_company_id = sub.billing_company_id)
   LEFT JOIN users_revshare ur
    ON (sub.rsid = ur.rsid AND sub.rsid != '')
   LEFT JOIN users u
    ON (ur.uid = u.uid)

These crashes did not happen with 7.0.6.
[19 Jun 2009 15:01] John Sabo
Excuse me.. did not happen with 7.0.5
[22 Jun 2009 12:43] Jonathan Miller
Hi, please attach the logs tar/gzip as they more then likely will be needed.

Thanks!
[22 Jun 2009 13:31] Jørgen Austvik
Please also include log files, trace files from the crash and the configuration files (mysql/cluster).
[22 Jul 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[23 Sep 2009 19:53] Matthew Bilek
I get this error all of the time also.  See bug #46985 for configuration information.