MySQL Bugs: #39947: Mysqld can't change tables after network recovery

Bug #39947	Mysqld can't change tables after network recovery
Submitted:	9 Oct 2008 8:05	Modified:	12 Jan 2009 15:14
Reporter:	li zhou	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	6.2.16	OS:	Any
Assigned to:	Don Kehn	CPU Architecture:	Any

Description:
A customer issue about cluster.

The configuration of cluster as follow:
3 hosts.
host1: datanode1, mgm1;
host2: datanode2, mysqld1;
host3: mgm2, mysqld2.

After host3 lost connect with other hosts for several minutes, the network of host3 recover. When tried to connect mysqld2 to manage the mysql, it told that "table ** is read only". But everything work well throught mysqld1

How to repeat:
config.ini, test program , my.cnf and dump table will attach later.

1.
setup the mysql cluster:
host1: ndbd1, mgm1;
host2: ndbd2, mysqld1;
host3: mgm2, mysqld2.

2.
start the cluster

3.
compile the test program code readonly.c like:
gcc -g -o readonly readonly.c -I/usr/local/mysql/include -L/usr/local/mysql/lib/mysql -lmysqlclient -lz -lcrypt -lnsl -lm

4.
create the data files,log files, and the test database:
mysql < moodle.sql

5.
run the test program on host3, it will print messages to the terminal

6.
unplug the ethernet cable of host3.
After few seconds( three times of db-api heartbeat interval), the test program will print error message on the terminal;
then, plug the cable.
Now, the test program will abort and print "table is read only" on the terminal,
mysqld will log "table './moodle/mdl_user_1014' opened read only" to its error log file.
if the test program does not abort, unplug and plug the cable more times, the exception will occur.

7.
now,the mdl_user_1014 table is read only through mysql server on host3, but it can be changed through mysql server on host2.

If we turn off log-bin, comment out log-bin in my.cnf, we can change table throughm mysqld2 on host3.
Disk table and memory table have the same problem.

test files

Attachment: readonly.zip (application/x-zip-compressed, text), 81.51 KiB.

Sorry for my incautious description. about bin log, it should be:

If we *TURN ON* log-bin, add log-bin in my.cnf, we can change table through
mysqld2 on host3. If we *TURN OFF* log-bin, comment out log-bin, it can't work.

This is confirmed and was introduced with the global schema lock. hints: global_flag_skip_invalidate_cache, m_cluster_state == CS_waiting_for_clean_cache in the ClusterMgr.cpp area.