Bug #39947 Mysqld can't change tables after network recovery
Submitted: 9 Oct 2008 8:05 Modified: 12 Jan 2009 15:14
Reporter: li zhou Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:6.2.16 OS:Any
Assigned to: Don Kehn CPU Architecture:Any

[9 Oct 2008 8:05] li zhou
Description:
A customer issue about cluster.  

The configuration of cluster as follow:
3 hosts. 
host1: datanode1, mgm1; 
host2: datanode2, mysqld1; 
host3: mgm2, mysqld2.

After host3 lost connect with other hosts for several minutes, the network of host3 recover. When tried to connect mysqld2 to manage the mysql, it told that "table ** is read only". But everything work well throught mysqld1

How to repeat:
config.ini, test program , my.cnf and dump table will attach later.

1.
setup the mysql cluster:                                                 
host1: ndbd1, mgm1; 
host2: ndbd2, mysqld1; 
host3: mgm2, mysqld2.

2.
start the cluster

3.
compile the test program code readonly.c like:
gcc -g -o readonly readonly.c -I/usr/local/mysql/include -L/usr/local/mysql/lib/mysql -lmysqlclient -lz -lcrypt -lnsl -lm

4.
create the data files,log files, and the test database:
mysql < moodle.sql

5.                                                 
run the test program on host3, it will print messages to the terminal

6.
unplug the ethernet cable of host3.
After few seconds( three times of db-api heartbeat interval), the test program will print error message on the terminal;
then, plug the cable.
Now, the test program will abort and print "table is read only" on the terminal,                                                                 
mysqld will log "table './moodle/mdl_user_1014' opened read only" to its error log file.
if the test program does not abort, unplug and plug the cable more times, the exception will occur.

7.
now,the mdl_user_1014 table is read only through mysql server on host3, but it can be changed through mysql server on host2.

If we turn off log-bin, comment out log-bin in my.cnf, we can change table throughm mysqld2 on host3.
Disk table and memory table have the same problem.
[9 Oct 2008 8:08] li zhou
test files

Attachment: readonly.zip (application/x-zip-compressed, text), 81.51 KiB.

[10 Oct 2008 2:19] li zhou
Sorry for my incautious description. about bin log, it should be:

If we *TURN ON* log-bin, add log-bin in my.cnf, we can change table through
mysqld2 on host3. If we *TURN OFF* log-bin, comment out log-bin, it can't work.
[15 Oct 2008 22:50] Don Kehn
This is confirmed and was introduced with the global schema lock. hints: global_flag_skip_invalidate_cache, m_cluster_state == CS_waiting_for_clean_cache in the ClusterMgr.cpp area.