070228 22:53:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 22:54:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 22:55:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 22:56:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 22:57:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 22:58:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 22:59:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:00:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:01:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:02:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:03:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:04:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:05:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:06:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070228 23:07:30 [ERROR] Failed to open log (file './db2-prod3-bin.000193', errno 2) 070301 23:15:09 [ERROR] Failed to open log (file './db2-prod3-bin.000001', errno 2) 070301 23:15:09 [ERROR] Failed to open log (file './db2-prod3-bin.000001', errno 2) 070301 23:16:09 [ERROR] Failed to open log (file './db2-prod3-bin.000001', errno 2) 070301 23:17:09 [ERROR] Failed to open log (file './db2-prod3-bin.000001', errno 2) 070301 23:18:09 [ERROR] Failed to open log (file './db2-prod3-bin.000001', errno 2) 070303 2:49:50 InnoDB: Error: trying to declare trx to enter InnoDB, but InnoDB: it already is declared. TRANSACTION 0 427515204, ACTIVE 0 sec, process no 3489, OS thread id 1291577696, thread declared inside InnoDB 0 mysql tables in use 1, locked 1 1 lock struct(s), heap size 368 MySQL thread id 3375, query id 1397020056 app4.prod3.offermatica.com 10.3.3.40 web update insert into pii_event (client_id, event_time, pc_id, property_name, property_value, type_id) values (255, '2007-03-03 02:49:50', '1172906374203-916055', '9105.33472.branch', '2147483647', 4) 070315 2:05:36 [Warning] Could not remove tmp table: '/tmp/#sqlda1_d51_0', error: 2 070315 2:05:37 [Warning] Could not remove tmp table: '/tmp/#sqlda1_d93_0', error: 2 070322 6:34:37InnoDB: Assertion failure in thread 1258297696 in file ../include/buf0buf.ic line 626 InnoDB: Failing assertion: block->buf_fix_count > 0 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Thread 1201588576 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1230608736 stopped in file ../include/sync0sync.ic line 111 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=2093056 max_used_connections=547 max_connections=1000 threads_connected=542 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2557016 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x3090492440 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x4b001118, backtrace may not be correct. Bogus stack limit or frame pointer, fp=0x4b001118, stack_bottom=0x4b000000, thread_stack=262144, aborting backtrace. Trying to get some variables. Some pointers may be invalid and cause the dump to abort... InnoDB: Thread 1209309536 stopped in file ha_innodb.cc line 978 InnoDB: Thread 1192536416 stopped in file ../include/sync0sync.ic line 111 thd->query at 0x2cd9c1e0 = select SUM(visitevent0_.sum_elapsed_time) as col_0_0_ from ana_event_summary_by_time visitevent0_ where visitevent0_.environment_id=34583 and (visitevent0_.campaign_step_id in (62319)) and (visitevent0_.summary_time between '2007-01-22 00:00:00' and '2007-03-22 04:00:00') thd->thread_id=4715 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 0 070322 06:34:40 mysqld restarted 070322 6:34:40 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=db2-prod3-bin' to avoid this problem. 070322 6:34:48 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 070322 6:34:48 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 567 1934359442. InnoDB: Doing recovery: scanned up to log sequence number 567 1939601920 InnoDB: Doing recovery: scanned up to log sequence number 567 1944844800 InnoDB: Doing recovery: scanned up to log sequence number 567 1950087680 InnoDB: Doing recovery: scanned up to log sequence number 567 1955330560 InnoDB: Doing recovery: scanned up to log sequence number 567 1960573440 InnoDB: Doing recovery: scanned up to log sequence number 567 1965816320 InnoDB: Doing recovery: scanned up to log sequence number 567 1971059200 InnoDB: Doing recovery: scanned up to log sequence number 567 1976302080 InnoDB: Doing recovery: scanned up to log sequence number 567 1981544960 InnoDB: Doing recovery: scanned up to log sequence number 567 1986787840 InnoDB: Doing recovery: scanned up to log sequence number 567 1992030720 InnoDB: Doing recovery: scanned up to log sequence number 567 1997273600 InnoDB: Doing recovery: scanned up to log sequence number 567 2002516480 InnoDB: Doing recovery: scanned up to log sequence number 567 2007759360 InnoDB: Doing recovery: scanned up to log sequence number 567 2013002240 InnoDB: Doing recovery: scanned up to log sequence number 567 2018245120 InnoDB: Doing recovery: scanned up to log sequence number 567 2023488000 InnoDB: Doing recovery: scanned up to log sequence number 567 2028730880 InnoDB: Doing recovery: scanned up to log sequence number 567 2033973760 InnoDB: Doing recovery: scanned up to log sequence number 567 2039216640 InnoDB: Doing recovery: scanned up to log sequence number 567 2044459520 InnoDB: Doing recovery: scanned up to log sequence number 567 2049702400 InnoDB: Doing recovery: scanned up to log sequence number 567 2054945280 InnoDB: Doing recovery: scanned up to log sequence number 567 2060188160 InnoDB: Doing recovery: scanned up to log sequence number 567 2065431040 InnoDB: Doing recovery: scanned up to log sequence number 567 2070673920 InnoDB: Doing recovery: scanned up to log sequence number 567 2075916800 InnoDB: Doing recovery: scanned up to log sequence number 567 2081159680 InnoDB: Doing recovery: scanned up to log sequence number 567 2086402560 InnoDB: Doing recovery: scanned up to log sequence number 567 2091645440 InnoDB: Doing recovery: scanned up to log sequence number 567 2096888320 InnoDB: Doing recovery: scanned up to log sequence number 567 2102131200 InnoDB: Doing recovery: scanned up to log sequence number 567 2107374080 InnoDB: Doing recovery: scanned up to log sequence number 567 2112616960 InnoDB: Doing recovery: scanned up to log sequence number 567 2117859840 InnoDB: Doing recovery: scanned up to log sequence number 567 2123102720 InnoDB: Doing recovery: scanned up to log sequence number 567 2128345600 InnoDB: Doing recovery: scanned up to log sequence number 567 2133588480 InnoDB: Doing recovery: scanned up to log sequence number 567 2138831360 InnoDB: Doing recovery: scanned up to log sequence number 567 2143551522 InnoDB: 6 transaction(s) which must be rolled back or cleaned up InnoDB: in total 1521 row operations to undo InnoDB: Trx id counter is 0 625110528 070322 6:35:13 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 756071355, file name db1-prod3-bin.000271 InnoDB: Last MySQL binlog file position 0 208693501, file name ./db2-prod3-bin.000509 InnoDB: Starting in background the rollback of uncommitted transactions 070322 6:38:06 InnoDB: Rolling back trx with id 0 625109712, 255 rows to undo 070322 6:38:06 InnoDB: Started; log sequence number 567 2143551522 070322 6:38:06 [Note] Recovering after a crash using db2-prod3-bin InnoDB: Rolling back of trx id 0 625109712 completed 070322 6:38:06 InnoDB: Rolling back trx with id 0 625109711, 262 rows to undo InnoDB: Rolling back of trx id 0 625109711 completed 070322 6:38:06 InnoDB: Rolling back trx with id 0 625109710, 238 rows to undo InnoDB: Rolling back of trx id 0 625109710 completed 070322 6:38:06 InnoDB: Rolling back trx with id 0 625109709, 276 rows to undo InnoDB: Rolling back of trx id 0 625109709 completed 070322 6:38:06 InnoDB: Rolling back trx with id 0 625109708, 242 rows to undo InnoDB: Rolling back of trx id 0 625109708 completed 070322 6:38:06 InnoDB: Rolling back trx with id 0 625109707, 248 rows to undo InnoDB: Rolling back of trx id 0 625109707 completed 070322 6:38:06 InnoDB: Rollback of non-prepared transactions completed 070322 6:38:07 [Note] Starting crash recovery... 070322 6:38:07 [Note] Crash recovery finished. 070322 6:38:07 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=db2-prod3-relay-bin' to avoid this problem. 070322 6:38:07 [Note] /home/m2/work/m2/scripts/system/mysql/bin/mysqld: ready for connections. Version: '5.0.32-enterprise-gpl-log' socket: '/usr/local/mysql/current/data/mysql.sock' port: 3306 MySQL Enterprise Server (GPL)