| Bug #45614 | Query crashes data nodes, sometimes all of them, sometimes just one. | ||
|---|---|---|---|
| Submitted: | 19 Jun 2009 12:51 | Modified: | 22 Jul 2009 12:43 |
| Reporter: | John Sabo | Email Updates: | |
| Status: | No Feedback | Impact on me: | |
| Category: | MySQL Cluster: Disk Data | Severity: | S2 (Serious) |
| Version: | mysql-5.1-telco-7.0 | OS: | Linux |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | crash, dbtc/DbtcMain.cpp, ndb 7.0.6, NDBCNTR | ||
| Triage: | Triaged: D1 (Critical) / R6 (Needs Assessment) / E6 (Needs Assessment) | ||
[19 Jun 2009 13:00]
John Sabo
Description:
We have a cron that updates a table that runs every morning.. Since we've upgraded to
mysql-3.1.34_ndb-7.0.6 it's been crashing data nodes or even our entire cluster (all data
nodes go poof)
I have 2 different errors for 2 diff types of crashes. Both with the same query.
Full crash (all data nodes go poof):
Time: Thursday 18 June 2009 - 05:00:26
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtc/DbtcMain.cpp
Error object: DBTC (Line: 9344) 0x0000000e
Program: /usr/sbin/ndbd
Pid: 10357
Trace: /db/datadir/ndb_2_trace.log.1
Version: mysql-5.1.34 ndb-7.0.6
***EOM***
Single node crash:
Time: Friday 19 June 2009 - 05:00:22
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 2 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 260) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 21093
Trace: /db/datadir/ndb_2_trace.log.2
Version: mysql-5.1.34 ndb-7.0.6
***EOM***
Here's the query:
INSERT INTO st_transactions_summary
SELECT
NULL,
t1.billing_company_id,
t1.transaction_type,
/* t1.account_id, */
/* t1.subaccount_id, */
t1.transaction_id,
t1.subscription_id,
t1.transaction_date,
t2.transaction_date,
/* t2.partner_id, */
t1.amount,
t2.amount,
t2.initial_period,
t2.recurring_amount,
t2.recurring_period,
'', /* member status */
t2.card_type,
t2.affiliate_type,
t1.commission,
t2.payment_type,
u.uid,
ur.sales_rep_id,
sub.sid
FROM st_transactions t1
LEFT JOIN st_transactions t2
ON(t1.subscription_id = t2.subscription_id AND t1.billing_company_id = t2.billing_company_id AND t2.transaction_type = 'new')
LEFT JOIN user_subscriptions sub
ON (sub.vendor_subs_id = t1.subscription_id AND t1.billing_company_id = sub.billing_company_id)
LEFT JOIN users_revshare ur
ON (sub.rsid = ur.rsid AND sub.rsid != '')
LEFT JOIN users u
ON (ur.uid = u.uid)
These crashes did not happen with 7.0.6.
[19 Jun 2009 15:01]
John Sabo
Excuse me.. did not happen with 7.0.5
[22 Jun 2009 12:43]
Jonathan Miller
Hi, please attach the logs tar/gzip as they more then likely will be needed. Thanks!
[22 Jun 2009 13:31]
Jørgen Austvik
Please also include log files, trace files from the crash and the configuration files (mysql/cluster).
[22 Jul 2009 23:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[23 Sep 2009 19:53]
Matthew Bilek
I get this error all of the time also. See bug #46985 for configuration information.
Description: We have a cron that updates a table that runs every morning.. Since we've upgraded to mysql-3.1.34_ndb-7.0.6 it's been crashing data nodes or even our entire cluster (all data nodes go poof) I have 2 different errors for 2 diff types of crashes. Both with the same query. Full crash (all data nodes go poof): Time: Thursday 18 June 2009 - 05:00:26 Status: Temporary error, restart node Message: Internal program error (failed ndbrequire) (Internal error, programming Error: 2341 Error data: dbtc/DbtcMain.cpp Error object: DBTC (Line: 9344) 0x0000000e Program: /usr/sbin/ndbd Pid: 10357 Trace: /db/datadir/ndb_2_trace.log.1 Version: mysql-5.1.34 ndb-7.0.6 ***EOM*** Single node crash: Time: Friday 19 June 2009 - 05:00:22 Status: Temporary error, restart node Message: System error, node killed during node restart by other node (Internal e Error: 2303 Error data: Node 2 killed this node because GCP stop was detected Error object: NDBCNTR (Line: 260) 0x0000000a Program: /usr/sbin/ndbd Pid: 21093 Trace: /db/datadir/ndb_2_trace.log.2 Version: mysql-5.1.34 ndb-7.0.6 ***EOM*** Here's the query: INSERT INTO st_transactions_summary SELECT NULL, t1.billing_company_id, t1.transaction_type, /* t1.account_id, */ /* t1.subaccount_id, */ t1.transaction_id, t1.subscription_id, t1.transaction_date, t2.transaction_date, /* t2.partner_id, */ t1.amount, t2.amount, t2.initial_period, t2.recurring_amount, t2.recurring_period, '', /* member status */ t2.card_type, t2.affiliate_type, t1.commission, t2.payment_type, u.uid, ur.sales_rep_id, sub.sid FROM st_transactions t1 LEFT JOIN st_transactions t2 ON(t1.subscription_id = t2.subscription_id AND t1.billing_company_id = t2.bi LEFT JOIN user_subscriptions sub ON (sub.vendor_subs_id = t1.subscription_id AND t1.billing_company_id = sub. LEFT JOIN users_revshare ur ON (sub.rsid = ur.rsid AND sub.rsid != '') LEFT JOIN users u ON (ur.uid = u.uid) These crashes did not happen with 7.0.6. How to repeat: Run query.