Bug #44330 ndb-connection-pool set higher then NDB API slots makes db appear corrupt
Submitted: 16 Apr 2009 21:22 Modified: 23 Apr 2009 13:52
Reporter: Jonathan Miller Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.1-telco-7.0 OS:Any
Assigned to: CPU Architecture:Any

[16 Apr 2009 21:22] Jonathan Miller
Description:
Found during debugging of another issue.

MySQLD ndb-connection-pool=3

NDB API Slots = 1

TPCB database and stored procedures already loaded.

Start running tpcbSql

tpcbSql errors on:
for (m_i = 0; m_i < NUMBER_OF_SQL_STATEMENTS; m_i++)
  {
    stmt[m_i] = MysqlSimplePrepare(mysql, MySqlStmts[m_i]);
    CheckStmt(stmt[m_i]); <----Error
  }

MySQLD error log:
090416 22:51:10 [ERROR] /data0/cr_autotest/libexec/mysqld: Incorrect information in file: './TPCB/account.frm'
090416 22:51:10 [ERROR] /data0/cr_autotest/libexec/mysqld: Incorrect information in file: './TPCB/branch.frm' 090416 22:51:10 [ERROR] /data0/cr_autotest/libexec/mysqld: Incorrect information in file: './TPCB/history.frm'
090416 22:51:10 [ERROR] /data0/cr_autotest/libexec/mysqld: Incorrect information in file: './TPCB/sync.frm'
090416 22:51:10 [ERROR] /data0/cr_autotest/libexec/mysqld: Incorrect information in file: './TPCB/teller.frm'
090416 22:51:10 [ERROR] /data0/cr_autotest/libexec/mysqld: Incorrect information in file: './TPCB/trans.frm'
090416 22:51:16 [ERROR] /data0/cr_autotest/libexec/mysqld: Incorrect information in file: './TPCB/account.frm'

In side mysql:

mysql> use TPCB
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from account limit 1;
ERROR 1296 (HY000): Got error 157 'Unknown error code' from NDBCLUSTER
mysql> show errors;
+-------+------+-----------------------------------------------------+
| Level | Code | Message                                             |
+-------+------+-----------------------------------------------------+
| Error | 1296 | Got error 4009 'Cluster Failure' from NDB           |
| Error | 1296 | Got error 157 'Unknown error code' from NDBCLUSTER  |
| Error | 1033 | Incorrect information in file: './TPCB/account.frm' |
+-------+------+-----------------------------------------------------+

Looks corrupt!

From ndb tools:
./ndb_select_count -d TPCB account
100000 records in table account

./ndb_desc -d TPCB account
-- account --
Version: 2
Fragment type: 9
K Value: 6
Min load factor: 78
Max load factor: 80
Temporary table: no
Number of attributes: 4
Number of primary keys: 1
Length of frm data: 306
Row Checksum: 1
Row GCI: 1
SingleUserMode: 0
ForceVarPart: 1
FragmentCount: 1
TableStatus: Retrieved
-- Attributes --
aid Int PRIMARY KEY DISTRIBUTION KEY AT=FIXED ST=MEMORY
bid Int NULL AT=FIXED ST=MEMORY
balance Decimal(8,2) NULL AT=FIXED ST=MEMORY
filler Char(80;latin1_swedish_ci) NULL AT=FIXED ST=MEMORY

-- Indexes --
PRIMARY KEY(aid) - UniqueHashIndex
PRIMARY(aid) - OrderedIndex

hmmm, ndb seeing it okay,

Take down mysqld.
change CCP to # of slots (1)
restart.

mysql> use TPCB
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from account limit 1;
+-------+------+-----------+-----------------+
| aid   | bid  | balance   | filler          |
+-------+------+-----------+-----------------+
| 44980 | 4498 | 716594.00 | account updated |
+-------+------+-----------+-----------------+
1 row in set (0.00 sec)

Back to normal. That's better!

How to repeat:
1) Setup mysqld, and one NDBD with proper CCP = API Slots.
2) Load TPCB db and Stored Procs.
/usr/bin/perl -w /space/cluster_rep_auto/tests/tpcb/tpcb_load_db.pl --port=15001 --host=ndb13 --log=/space/cluster_rep_auto/logs/tpcb.log --numA=100000
3) Take down mysqld and change CCP to 3 or more and restart
4) log into mysql (might take a little bit before you can log in)
5) USE TPCB;
6) select * from account limit 1;
ERROR 1296 (HY000): Got error 157 'Unknown error code' from NDBCLUSTER
mysql> show errors;
+-------+------+-----------------------------------------------------+
| Level | Code | Message                                             |
+-------+------+-----------------------------------------------------+
| Error | 1296 | Got error 4009 'Cluster Failure' from NDB           |
| Error | 1296 | Got error 157 'Unknown error code' from NDBCLUSTER  |
| Error | 1033 | Incorrect information in file: './TPCB/account.frm' |
+-------+------+-----------------------------------------------------+

my.cnf
[mysqld]
#debug=d:t:i:O,/space/var/mysql.trace
server-id=8
loose-skip-grant-tables
loose-skip-innodb
ndb_use_exact_count=0
ndb-cluster-connection-pool=3
#ndb-cluster-connection-pool=1
core
port=15001
datadir=/space/var/
log-warnings
log-error=/space/var/err.ndb13.err
ndbcluster
ndb-connectstring=ndb13:14000  # location of MGM node

[ndbd]
core
ndb-connectstring=ndb13:14000  # location of MGM node

config.ini
[NDBD DEFAULT]
NoOfReplicas: 1
DataDir: /space/run
FileSystemPath: /space/run

DataMemory: 1G
IndexMemory: 200M

[MGM DEFAULT]

[NDB_MGMD]
Id: 1
HostName: ndb13
PortNumber: 14000
DataDir: /space/run
ArbitrationRank: 1

[NDBD]
Id: 2
HostName: ndb13

#[NDBD]
#Id: 3
#HostName: ndb14

[API]
Id: 4
HostName: ndb13

#[API]
#Id: 5
#HostName: ndb13

#[API]
#Id: 6
#HostName: ndb13

Suggested fix:
Unknown currently
[16 Apr 2009 21:23] Jonathan Miller
TPCB DB LOADER

Attachment: tpcb_load_db.pl (application/x-perl, text), 31.70 KiB.