Bug #68047 DB hung with InnoDB: Warning: a long semaphore wait when optimize table is going
Submitted: 7 Jan 2013 10:01 Modified: 22 Jan 2013 19:48
Reporter: Praneeth Reddy Kuchavaram Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.1.47 OS:Linux (X86_64)
Assigned to: CPU Architecture:Any
Tags: InnoDB: Warning: a long semaphore wait

[7 Jan 2013 10:01] Praneeth Reddy Kuchavaram
Description:
Hi,

When table optimize of 200Gb odd table was running. DB was hung and couldn't connect to it.
Interestingly when checked for Temporary table..I didn't find *#sql* file under data directory. DB is hung after 6 hrs table optimize has started.

Below are the errors.

MySQL thread id 14993921, query id 3369742277 localhost root rename result table
optimize table private_unmatched_detections
---TRANSACTION A8020C55, ACTIVE 288 sec, process no 16280, OS thread id 2129779008
5 lock struct(s), heap size 1248, 40 row lock(s), undo log entries 19
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
--Thread 1096558912 has waited at row/row0purge.c line 541 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x2aaaabd2a880 created in file dict/dict0dict.c line 667
a writer (thread id 1557363008) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row/row0purge.c line 541
Last time write locked in file row/row0mysql.c line 3040
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
--Thread 1096558912 has waited at row/row0purge.c line 541 for 303.00 seconds the semaphore:
S-lock on RW-latch at 0x2aaaabd2a880 created in file dict/dict0dict.c line 667
a writer (thread id 1557363008) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row/row0purge.c line 541
Last time write locked in file row/row0mysql.c line 3040
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream

How to repeat:
NA
[7 Jan 2013 13:12] Praneeth Reddy Kuchavaram
So, waited for sometime to bring down DB.Nothing responded, then terminated background mysql process & started database. But, is no crashing again & again after recovery with below errors.

130106 08:04:54 mysqld_safe Starting mysqld daemon with databases from /myhome/vdc/globalaction/node3/data
130106  8:04:54 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
130106  8:04:59  InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 28390316547722
130106  8:05:00  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 28390321790464
InnoDB: Doing recovery: scanned up to log sequence number 28390327033344
InnoDB: Doing recovery: scanned up to log sequence number 28390332276224
.
.
.
.
InnoDB: Doing recovery: scanned up to log sequence number 28393457032704
InnoDB: Doing recovery: scanned up to log sequence number 28393459582104
InnoDB: 5 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1358 row operations to undo
InnoDB: Trx id counter is A8021700
130106  8:05:51  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 91229964, file name /myhome/vdc/globalaction/node3/binlog/binlog.053841
130106  8:06:39  InnoDB: Rolling back trx with id A8021586, 3 rows to undo
InnoDB: Dropping table with id 0 384 in recovery if it exists
130106  8:06:40  InnoDB: error: space object of table'yearbooks_search_v2/#sql2-3f98-e4ca01',
InnoDB: space id 368 did not exist in memory. Retrying an open.
130106  8:06:40  InnoDB: Assertion failure in thread 47710574677872 in file fil/fil0fil.c line 2981
InnoDB: Failing assertion: flags != DICT_TF_COMPACT
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
130106  8:06:40 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=2000
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1572993 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
/opt/mysql/5.1.47/bin/mysqld(my_print_stacktrace+0x2e)[0x8b23be]
/opt/mysql/5.1.47/bin/mysqld(handle_segfault+0x322)[0x5e2942]
/lib64/libpthread.so.0[0x318ee0eb10]
/lib64/libc.so.6(gsignal+0x35)[0x318e630265]
/lib64/libc.so.6(abort+0x110)[0x318e631d10]
/opt/mysql/5.1.47/lib/plugin/ha_innodb_plugin.so[0x2aaaabb3fc15]
/opt/mysql/5.1.47/lib/plugin/ha_innodb_plugin.so[0x2aaaabb33a60]
/opt/mysql/5.1.47/lib/plugin/ha_innodb_plugin.so[0x2aaaabb34017]
/opt/mysql/5.1.47/lib/plugin/ha_innodb_plugin.so[0x2aaaabbddda0]
/opt/mysql/5.1.47/lib/plugin/ha_innodb_plugin.so[0x2aaaabbcedfd]
/opt/mysql/5.1.47/lib/plugin/ha_innodb_plugin.so[0x2aaaabb5a9d8]
/opt/mysql/5.1.47/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x31)[0x6d35d1]
/opt/mysql/5.1.47/bin/mysqld[0x759b2e]
/opt/mysql/5.1.47/bin/mysqld(_Z11plugin_initPiPPci+0x875)[0x75c745]
/opt/mysql/5.1.47/bin/mysqld[0x5e3115]
/opt/mysql/5.1.47/bin/mysqld(main+0x1c1)[0x5e7651]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x318e61d994]
/opt/mysql/5.1.47/bin/mysqld(fmod+0x62)[0x51499a]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
130106 08:06:41 mysqld_safe mysqld from pid file /myhome/vdc/globalaction/node3/admin/var/run/mysqld/mysqld.pid ended
130106 08:08:50 mysqld_safe Starting mysqld daemon with databases from /myhome/vdc/globalaction/node3/data
130106  8:08:50 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
130106  8:08:55  InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 28390316553953
130106  8:08:56  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 28390321796608
InnoDB: Doing recovery: scanned up to log sequence number 28390327039488
[16 Jan 2013 4:20] Praneeth Reddy Kuchavaram
Can someone check this issue and suggest ?
[22 Jan 2013 19:48] Sveta Smirnova
Thank you for the report.

Unfortunately this 600 seconds limit is hardcoded, so you can do nothing with it.

This is known issue and this report is duplicate of bug #48050.