Bug #46915 long semaphore wait
Submitted: 25 Aug 2009 11:16 Modified: 25 Sep 2009 11:43
Reporter: Joel Flickinger Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.0.77 OS:Linux (Red Hat Enterprise Linux Server release 5.3 (Tikanga))
Assigned to: CPU Architecture:Any

[25 Aug 2009 11:16] Joel Flickinger
Description:
Swap usage and eventual system crash.  Disabled swap to get an output of possible logs to reveal anything useful.  I had been looking around for simular issues with other users and haven't come up with anything other than a possible bug.

mysql.err

InnoDB: Warning: a long semaphore wait:
--Thread 1164409152 has waited at btr0cur.c line 424 for 455.00 seconds the semaphore:
X-lock on RW-latch at 0x2aad51637f98 created in file buf0buf.c line 497
a writer (thread id 1099688256) has reserved it in mode  exclusive
number of readers 0, waiters flag 1
Last time read locked in file ./../include/btr0btr.ic line 28
Last time write locked in file btr0cur.c line 424
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 1, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
--Thread 1164409152 has waited at btr0cur.c line 424 for 618.00 seconds the semaphore:
X-lock on RW-latch at 0x2aad51637f98 created in file buf0buf.c line 497
a writer (thread id 1099688256) has reserved it in mode  exclusive
number of readers 0, waiters flag 1
Last time read locked in file ./../include/btr0btr.ic line 28
Last time write locked in file btr0cur.c line 424
InnoDB: Warning: a long semaphore wait:
--Thread 1143429440 has waited at srv0srv.c line 1959 for 312.00 seconds the semaphore:
Mutex at 0x2aaabb1622b8 created file srv0srv.c line 872, lock var 0
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 1184110912 has waited at trx0trx.c line 213 for 312.00 seconds the semaphore:
Mutex at 0x2aaabb1622b8 created file srv0srv.c line 872, lock var 0
waiters flag 0
InnoDB: Warning: a long semaphore wait:
--Thread 1209137472 has waited at trx0trx.c line 213 for 312.00 seconds the semaphore:
Mutex at 0x2aaabb1622b8 created file srv0srv.c line 872, lock var 0
waiters flag 0
InnoDB: Warning: a long semaphore wait:
--Thread 1218722112 has waited at trx0trx.c line 213 for 308.00 seconds the semaphore:
Mutex at 0x2aaabb1622b8 created file srv0srv.c line 872, lock var 0
waiters flag 0
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 1, pwrites 1

my.cnf 

[client]
# default_character_set           = utf8
user                            = root
password                        = 
# socket                        = /var/lib/mysql/logs/mysql.sock
socket                          = /tmp/mysql.sock

[mysqld_safe]
default_character_set           = utf8
basedir                         = /var/lib/mysql/data

[mysqld]
auto_increment_increment        = 1
auto_increment_offset           = 1
default_character_set           = utf8
key_buffer                      = 256M
long_query_time                 = 10
max_allowed_packet              = 256M
max_connections                 = 256
myisam_sort_buffer_size         = 64M
port                            = 3306
read_buffer_size                = 2M
read_rnd_buffer_size            = 8M
query_cache_size                = 256M
server-id                       = 19507506
skip-locking
sort_buffer_size                = 2M
table_cache                     = 512
thread_cache                    = 32
thread_concurrency              = 16
default_storage_engine          = innodb
group_concat_max_len            = 131072

datadir                         = /var/lib/mysql/data
log-bin                         = /var/lib/mysql/binlog/binlog
# log                           = /var/lib/mysql/logs/mysql.log
log-error                       = /var/lib/mysql/logs/mysql.error
pid-file                        = /var/lib/mysql/logs/mysql.pid
log-slow-queries                = /var/lib/mysql/logs/mysql.slow
# socket                        = /var/lib/mysql/logs/mysql.sock
socket                          = /tmp/mysql.sock
tmpdir                          = /dev/shm

log-slave-updates
skip-slave-start
# master-host                   = 
# master-user                   = 
# master-password               = 
# master-port                   = 
# replicate-do-db               = 
replicate-ignore-db             = mysql
report-host                     = av-db01-p06
# report-port                   = 
master-info-file                = /var/lib/mysql/relay/master.info
relay-log                       = /var/lib/mysql/relay/relay
relay-log-index                 = /var/lib/mysql/relay/relay.index
relay-log-info-file             = /var/lib/mysql/relay/relay.info

innodb_additional_mem_pool_size = 64M
innodb_buffer_pool_size         = 10G
innodb_data_file_path           = ibdata01:16G;ibdata02:16G;ibdata03:16G;ibdata04:16G;ibdata05:16G;
innodb_data_home_dir            = /var/lib/mysql/ib
innodb_flush_log_at_trx_commit  = 1
innodb_lock_wait_timeout        = 20
innodb_log_buffer_size          = 8M
innodb_log_file_size            = 256M
innodb_log_files_in_group       = 2
innodb_log_group_home_dir       = /var/lib/mysql/ib

How to repeat:
Not sure, system has been operational for about 8 months with no problems this is our 4th crash.

Suggested fix:
N/A
[25 Aug 2009 11:43] Miguel Solorzano
Thank you for the bug report. Could you please provide the stacktrace of the crash?. Thanks in advance.
[25 Aug 2009 17:07] Joel Flickinger
This is all I have from dmesg

Attachment: dmesg (application/octet-stream, text), 7.83 KiB.

[25 Sep 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".