Jun 13 11:11:12 database1 mysqld[777]: InnoDB: ###### Diagnostic info printed to the standard error stream
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1525676976 has waited at btr0sea.c line 916 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: S-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1465908144 has waited at btr0sea.c line 916 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: S-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1513094064 has waited at row0sel.c line 4162 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: Mutex at 0xb68db168 created file srv0srv.c line 872, lock var 1
Jun 13 11:11:14 database1 mysqld[777]: waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1486879664 has waited at trx0trx.c line 371 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: Mutex at 0xb68db168 created file srv0srv.c line 872, lock var 1
Jun 13 11:11:14 database1 mysqld[777]: waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1557134256 has waited at btr0sea.c line 489 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: X-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1528822704 has waited at btr0sea.c line 1236 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: X-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1480588208 has waited at btr0sea.c line 1127 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: S-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1516239792 has waited at btr0sea.c line 1127 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: S-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1490025392 has waited at btr0sea.c line 1127 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: S-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1500445616 has waited at trx0trx.c line 1609 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: Mutex at 0xb68db168 created file srv0srv.c line 872, lock var 1
Jun 13 11:11:14 database1 mysqld[777]: waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1897311152 has waited at srv0srv.c line 1952 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: Mutex at 0xb68db168 created file srv0srv.c line 872, lock var 1
Jun 13 11:11:14 database1 mysqld[777]: waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1487928240 has waited at btr0sea.c line 1127 for 786.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: S-lock on RW-latch at 0xb68dcb68 created in file btr0sea.c line 139
Jun 13 11:11:14 database1 mysqld[777]: a writer (thread id 1557134256) has reserved it in mode  wait exclusive
Jun 13 11:11:14 database1 mysqld[777]: number of readers 1, waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: Last time read locked in file btr0sea.c line 746
Jun 13 11:11:14 database1 mysqld[777]: Last time write locked in file btr0sea.c line 489
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Warning: a long semaphore wait:
Jun 13 11:11:14 database1 mysqld[777]: --Thread 1880533936 has waited at row0mysql.c line 2242 for 785.00 seconds the semaphore:
Jun 13 11:11:14 database1 mysqld[777]: Mutex at 0xb68db168 created file srv0srv.c line 872, lock var 1
Jun 13 11:11:14 database1 mysqld[777]: waiters flag 1
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
Jun 13 11:11:14 database1 mysqld[777]: InnoDB: Pending preads 0, pwrites 0
Jun 13 11:11:17 database1 snmpd[22701]: Connection from UDP: [83.145.232.84]:58164
Jun 13 11:11:29 database1 snmpd[22701]: Connection from UDP: [83.145.232.84]:58166
Jun 13 11:11:42 database1 snmpd[22701]: Connection from UDP: [83.145.232.84]:58167
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: ###### Diagnostic info printed to the standard error stream
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: Error: semaphore wait has lasted > 600 seconds
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: We intentionally crash the server, because it appears to be hung.
Jun 13 11:11:44 database1 mysqld[777]: 070613 11:11:44InnoDB: Assertion failure in thread 1888922544 in file srv0srv.c line 2093
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: We intentionally generate a memory trap.
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: If you get repeated assertion failures or crashes, even
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: immediately after the mysqld startup, there may be
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: corruption in the InnoDB tablespace. Please refer to
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
Jun 13 11:11:44 database1 mysqld[777]: InnoDB: about forcing recovery.
Jun 13 11:11:44 database1 mysqld[777]: mysqld got signal 11;
Jun 13 11:11:44 database1 mysqld[777]: This could be because you hit a bug. It is also possible that this binary
Jun 13 11:11:44 database1 mysqld[777]: or one of the libraries it was linked against is corrupt, improperly built,
Jun 13 11:11:44 database1 mysqld[777]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jun 13 11:11:44 database1 mysqld[777]: We will try our best to scrape up some info that will hopefully help diagnose
Jun 13 11:11:44 database1 mysqld[777]: the problem, but since we have already crashed, something is definitely wrong
Jun 13 11:11:44 database1 mysqld[777]: and this may fail.
Jun 13 11:11:44 database1 mysqld[777]: 
Jun 13 11:11:44 database1 mysqld[777]: key_buffer_size=268435456
Jun 13 11:11:44 database1 mysqld[777]: read_buffer_size=131072
Jun 13 11:11:44 database1 mysqld[777]: max_used_connections=101
Jun 13 11:11:44 database1 mysqld[777]: max_connections=100
Jun 13 11:11:44 database1 mysqld[777]: threads_connected=48
Jun 13 11:11:44 database1 mysqld[777]: It is possible that mysqld could use up to 
Jun 13 11:11:44 database1 mysqld[777]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 479743 K
Jun 13 11:11:44 database1 mysqld[777]: bytes of memory
Jun 13 11:11:44 database1 mysqld[777]: Hope that's ok; if not, decrease some variables in the equation.
Jun 13 11:11:44 database1 mysqld[777]: 
Jun 13 11:11:44 database1 mysqld[777]: thd=(nil)
Jun 13 11:11:44 database1 mysqld[777]: Attempting backtrace. You can use the following information to find out
Jun 13 11:11:44 database1 mysqld[777]: where mysqld died. If you see no messages after this, something went
Jun 13 11:11:44 database1 mysqld[777]: terribly wrong...
Jun 13 11:11:44 database1 mysqld[777]: Cannot determine thread, fp=0x7096a118, backtrace may not be correct.
Jun 13 11:11:44 database1 mysqld[777]: Stack range sanity check OK, backtrace follows:
Jun 13 11:11:44 database1 mysqld[777]: 0x81c0649
Jun 13 11:11:44 database1 mysqld[777]: 0x82ee280
Jun 13 11:11:44 database1 mysqld[777]: 0xb7ef10bd
Jun 13 11:11:44 database1 mysqld[777]: 0xb7d2b93e
Jun 13 11:11:44 database1 mysqld[777]: New value of fp=(nil) failed sanity check, terminating stack trace!
Jun 13 11:11:44 database1 mysqld[777]: Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
Jun 13 11:11:44 database1 mysqld[777]: stack trace is much more helpful in diagnosing the problem, so please do 
Jun 13 11:11:44 database1 mysqld[777]: resolve it
Jun 13 11:11:44 database1 mysqld[777]: The manual page at http://www.mysql.com/doc/en/Crashing.html contains
Jun 13 11:11:44 database1 mysqld[777]: information that should help you find out what is causing the crash.