Bug #46846 I can't create many undo logfile in mysql cluster
Submitted: 21 Aug 2009 2:54 Modified: 27 Aug 2009 10:11
Reporter: ws lee Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Cluster: Disk Data Severity:S1 (Critical)
Version:mysql5.1-telco-6.3, 5.1-telco-7.0 OS:Solaris (5.10)
Assigned to: Assigned Account CPU Architecture:Any
Tags: mysql5.1.32-ndb6.3.24
Triage: Triaged: D3 (Medium) / R6 (Needs Assessment) / E6 (Needs Assessment)

[21 Aug 2009 2:54] ws lee
Description:
I can't create many undo logfile.
In my environment, mysql clusgter occured error if undo file create over 237 number.

How to repeat:
config.ini
----------
[NDBD DEFAULT]
NoOfReplicas=1
DataMemory=500M
IndexMemory=50M

datadir=/var/lib/mysql-cluster5.1.32-ndb6.3.24

[NDBD]
hostname=localhost

[NDB_MGMD DEFAULT]
datadir=/var/lib/mysql-cluster5.1.32-ndb6.3.24

[NDB_MGMD]
hostname=localhost

[MYSQLD]
hostname=localhost
-----------------

my.cnf
--------------
[MYSQL_CLUSTER]
ndb-connectstring="host=localhost"

[MYSQLD]                        
ndbcluster
-----------------

create_unodfile.sh
-----------
MYSQL_DIR=/usr/local/mysql5.1.32-ndb6.3.24

i=0
while [ $i -le 500 ]
do
        i=`expr $i + 1`
        echo "i = $i"
        if [ $i = 1 ] 
        then
                CMD=`$MYSQL_DIR/bin/mysql -uroot -e"CREATE LOGFILE GROUP lg ADD UNDOFILE 'undo${i}.dat' ENGINE=ndbcluster" 2>&1`
        else
                CMD=`$MYSQL_DIR/bin/mysql -uroot -e"ALTER LOGFILE GROUP lg ADD UNDOFILE 'undo${i}.dat'ENGINE=ndbcluster" 2>&1`
        fi
        if [ -n "$CMD" ]
        then
                echo $CMD
                exit
        fi      
done
-----------------------------------

Run create_unodfile.sh
./create_unodfile.sh
i = 1
...
...
...
i = 235
i = 236
i = 237
ERROR 1533 (HY000) at line 1: Failed to alter: CREATE UNDOFILE

In my server, mysql occured error, in undofile 237 number
In this time, data node log is like below.

cat ndb_1_out.log
...
...
2009-08-21 11:22:29 [ndbd] INFO     -- DBDICT: File system open failed during FsConnectRecord state 1. OS errno: 24
2009-08-21 11:22:29 [ndbd] INFO     -- DBDICT (Line: 814) 0x0000000a
2009-08-21 11:22:29 [ndbd] INFO     -- Error handler shutting down system
2009-08-21 11:22:30 [ndbd] INFO     -- Error handler shutdown completed - exiting
2009-08-21 11:22:31 [ndbd] ALERT    -- Node 1: Forced node shutdown completed. Caused by error 2814: 'System reports no more file system resources(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
......
]
[27 Aug 2009 10:11] Sveta Smirnova
Thank you for the report.

Verified as described. Problem only repeatable on Solaris. On Linux everything works fine.