MySQL Bugs: #70783: Simultaneous nodes crash (Caused by error 2301: 'Assertion(Internal error, prog)

Bug #70783	Simultaneous nodes crash (Caused by error 2301: 'Assertion(Internal error, prog)
Submitted:	31 Oct 2013 10:20	Modified:	25 May 2016 14:33
Reporter:	Сергей Кукуев	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	7.3.2	OS:	Linux (2.6.39-200.24.1.el6uek.x86_64)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
All data nodes simultaneously crashed.
There wasn't any significant load on cluster.
Only one NDB API app node was running with 2 cluster connections established and
4 threads (4 Ndb objects) were working under load ~1500-1600 requests per sec.
Each request includes approx. 5 PK operations (reads and writes).

SELECT node_id, memory_type, used, total, used_pages, total_pages, 
round(used/1024/1024) as "Used (Mb)", 
round(total/1024/1024) as "Total (Mb)", 
round(used*100/total) as "Used (%)" 
FROM ndbinfo.memoryusage;

# node_id, memory_type, used, total, used_pages, total_pages, Used (Mb), Total (Mb), Used (%)
3, Data memory, 7083032576, 98784247808, 216157, 3014656, 6755, 94208, 7
3, Index memory, 882819072, 11813257216, 107766, 1442048, 842, 11266, 7
4, Data memory, 7083032576, 98784247808, 216157, 3014656, 6755, 94208, 7
4, Index memory, 882819072, 11813257216, 107766, 1442048, 842, 11266, 7
6, Data memory, 7081426944, 98784247808, 216108, 3014656, 6753, 94208, 7
6, Index memory, 882843648, 11813257216, 107769, 1442048, 842, 11266, 7

SELECT *#sum(t.DATA_LENGTH)/1024/1024
FROM information_schema.TABLES t
WHERE t.DATA_LENGTH > 1024*1024 order by DATA_LENGTH desc;

# TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE, ENGINE, VERSION, ROW_FORMAT, TABLE_ROWS, AVG_ROW_LENGTH, DATA_LENGTH, MAX_DATA_LENGTH, INDEX_LENGTH, DATA_FREE, AUTO_INCREMENT, CREATE_TIME, UPDATE_TIME, CHECK_TIME, TABLE_COLLATION, CHECKSUM, CREATE_OPTIONS, TABLE_COMMENT
def, BS, BL_Values, BASE TABLE, ndbcluster, 10, Fixed, 72890219, 104, 7631470592, 0, 0, 0, 72890220, , , , utf8mb4_bin, , , Primary balance value and attributes
def, BS, BL_PeriodicAttrs, BASE TABLE, ndbcluster, 10, Dynamic, 28844950, 68, 2563866624, 0, 0, 0, , , , , utf8mb4_bin, , , Periodic balances specific attributes
def, BS, BL_TDs, BASE TABLE, ndbcluster, 10, Dynamic, 6689035, 36, 1112178688, 0, 0, 0, , , , , utf8mb4_bin, , , Short classification by TD
def, BS, BL_PAnumbers, BASE TABLE, ndbcluster, 10, Dynamic, 9000007, 36, 548798464, 0, 0, 0, , , , , utf8mb4_bin, , , Short classification by PA number
def, BS, TL_Factors, BASE TABLE, ndbcluster, 10, Dynamic, 423, 68, 2097152, 0, 0, 0, 424, , , , utf8mb4_bin, , , Services on templates
def, BS, TL_Data, BASE TABLE, ndbcluster, 10, Dynamic, 349, 132, 2097152, 0, 0, 0, 350, , , , utf8mb4_bin, , , Templates of balances
def, BSPA, BL_Values, BASE TABLE, ndbcluster, 10, Fixed, 100000, 104, 10878976, 0, 0, 0, 100001, , , , utf8mb4_bin, , , Primary balance value and attributes

How to repeat:
Have no idea.

I runned much more higher load (4 APP nodes x 16 connections x 40 worker threads x ~6000 reqs per sec per app node) on the same configuration and there were no problems.

Can you provide ndb_error_reporter output (or manually attach the management and data node logs)?

http://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-programs-ndb-error-reporter.html

Logs are attached already.

ndb_error_report 1

Attachment: ndb_1.tar.gz (application/gzip, text), 381.48 KiB.

ndb_error_report 2

Attachment: ndb_2.tar.gz (application/gzip, text), 1.65 MiB.

ndb_error_report 3

Attachment: ndb_3.tar.gz (application/gzip, text), 1.41 MiB.

ndb_error_report 5

Attachment: ndb_4.tar.gz (application/gzip, text), 1.29 MiB.

ndb_error_report 6

Attachment: ndb_6.tar.gz (application/gzip, text), 1.46 MiB.

config.ini

Attachment: config.ini (application/octet-stream, text), 4.67 KiB.

Do you need any additional info?

This is a duplicate of the GSN 40 missing bug.
Not sure which bug number that has. Fairly certain it is fixed in a number of recent versions.

cant reproduce on any of the modern versions

duplicate: 18455971