MySQL Bugs: #44693: MYSQL Daemon hangs when rotating and deleting binlogs

Bug #44693	MYSQL Daemon hangs when rotating and deleting binlogs
Submitted:	6 May 2009 13:11	Modified:	16 Dec 2009 9:58
Reporter:	Etienne SAmmut	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: Locking	Severity:	S2 (Serious)
Version:	mysql-5.1.31	OS:	Linux (Centos 4.4)
Assigned to:		CPU Architecture:	Any

Description:
I have installed mysql-cluster version 6.3.22 with mysql-5.1.31. I have setup the binlog size to 128MB and I set it up to keep logs only for 7 days. As you know binlogs are cleaned when mysql starts or it creates a new binlog. What is happening in my case is that when log rotation occurs and it has more than 1 binlog file to delete the mysql instance stals. No updates/inserts are allowed on the Databases and I have to restart the mysql daemon so that we can continue use the database. Infacts the queries on hte db are marked as locked state

Also I noticed that in the mysql.bin.index file the file name of the second binlog file is deleted but physically the binlog file is still on the machine

How to repeat:
Setup the bin log file size to 64/128 MB
setup the expiry_logs_days to a small value
Bombard the DB with inserts to create more than one bin file in a day
When the expiry_logs_days pass bombard the DB to rotate log files and the problem should occur 

Suggested fix:
disabled the expiry_logs_days

Can be related to bug #41751

I always started the mysql daemon with the expiry_logs_days parameter setup and never gave me the error. 

I have just tried re-enable the parameter on our development environment and the mysql daemon did not startup. Below is the log extract. When I tried to start it again then it started. After enabling the expiry_log_days stoping and starting multiple times mysql daemon never failed to start

090506 15:28:38 mysqld_safe Starting mysqld daemon with databases from /opt/mysql-cluster/datafiles/
090506 15:28:38  InnoDB: Started; log sequence number 0 336554466
090506 15:28:38 [Note] NDB: NodeID is 4, management server '80.85.108.2:1189'
090506 15:28:39 [Note] NDB[0]: NodeID: 4, all storage nodes connected
090506 15:28:39 [Note] Starting Cluster Binlog Thread
090506 15:28:39 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=10485760
max_used_connections=0
max_threads=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2311165 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld(my_print_stacktrace+0x2d) [0x931dbd]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld(handle_segfault+0x324) [0x5f93d4]
/lib64/tls/libpthread.so.0 [0x3e0740c430]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld [0x7988fe]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld(ha_binlog_index_purge_file(THD*, char const*)+0x9e) [0x6e822e]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld(MYSQL_BIN_LOG::purge_logs(char const*, bool, bool, bool, unsigned long long*)+0x502) [0x6941e2]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld(MYSQL_BIN_LOG::purge_logs_before_date(long)+0x245) [0x694615]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld [0x5fac7d]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld(main+0x7c5) [0x5fe265]
/lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x3e0691c3fb]
/opt/mysql-cluster-6.3.22/executable//libexec/mysqld [0x5309ca]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
090506 15:28:39 mysqld_safe mysqld from pid file /opt/mysql-cluster/datafiles/mt-ce-bwa-dev02.pid ended
 

Thanks 
Etienne Sammut

Thank you for the feedback.

Trace looked similar to bug #41751. So I close this report as duplicate.

As problem is only repeatable with NDB distribution and this is regression bug is not surprising you never saw this error before.