080806 23:28:07 mysqld started 080806 23:28:11 InnoDB: Started; log sequence number 66 2325356978 080806 23:28:11 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.45-community-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition (GPL) 080806 23:55:10 [Note] Slave SQL thread initialized, starting replication in log 'db-main-bin.000207' at position 1042422471, relay log '/database/cst/db/db-cst-relay.000598' position: 1042422610 080806 23:55:10 [Note] Slave I/O thread: connected to master 'repl@x.x.x.x:3306', replication started in log 'db-main-bin.000207' at position 1042422471 080916 8:20:35 [ERROR] /usr/sbin/mysqld: Sort aborted 080916 12:45:22InnoDB: Assertion failure in thread 1168841056 in file ./../include/buf0buf.ic line 626 InnoDB: Failing assertion: block->buf_fix_count > 0 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Thread 1169639776 stopped in file ./../include/sync0sync.ic line 111 InnoDB: Thread 1190406496 stopped in file row0sel.c line 2088 InnoDB: Thread 1198659936 stopped in file ./../include/sync0sync.ic line 111 InnoDB: Thread 1253239136 stopped in file sync0arr.c line 315 080916 12:45:22 - mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=520192 max_used_connections=477 max_connections=1500 threads_connected=139 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1538180 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2ba2177510 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x45ab0ff8, backtrace may not be correct. Bogus stack limit or frame pointer, fp=0x45ab0ff8, stack_bottom=0x45ab0000, thread_stack=262144, aborting backtrace. Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x2bcc1a52a0 is invalid pointer thd->thread_id=140734 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 0 080916 12:45:22 mysqld restarted 080916 12:45:24 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 080916 12:45:30 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 233 1704927053. InnoDB: Doing recovery: scanned up to log sequence number 233 1710169600 InnoDB: Doing recovery: scanned up to log sequence number 233 1715412480 InnoDB: Doing recovery: scanned up to log sequence number 233 1720655360 InnoDB: Doing recovery: scanned up to log sequence number 233 1725898240 InnoDB: Doing recovery: scanned up to log sequence number 233 1731141120 InnoDB: Doing recovery: scanned up to log sequence number 233 1736384000 InnoDB: Doing recovery: scanned up to log sequence number 233 1741626880 InnoDB: Doing recovery: scanned up to log sequence number 233 1746869760 InnoDB: Doing recovery: scanned up to log sequence number 233 1752112640 InnoDB: Doing recovery: scanned up to log sequence number 233 1757355520 InnoDB: Doing recovery: scanned up to log sequence number 233 1762598400 InnoDB: Doing recovery: scanned up to log sequence number 233 1767841280 InnoDB: Doing recovery: scanned up to log sequence number 233 1773084160 InnoDB: Doing recovery: scanned up to log sequence number 233 1778327040 InnoDB: Doing recovery: scanned up to log sequence number 233 1783569920 InnoDB: Doing recovery: scanned up to log sequence number 233 1788812800 InnoDB: Doing recovery: scanned up to log sequence number 233 1794055680 InnoDB: Doing recovery: scanned up to log sequence number 233 1799298560 InnoDB: Doing recovery: scanned up to log sequence number 233 1804541440 InnoDB: Doing recovery: scanned up to log sequence number 233 1809784320 InnoDB: Doing recovery: scanned up to log sequence number 233 1815027200 InnoDB: Doing recovery: scanned up to log sequence number 233 1820270080 InnoDB: Doing recovery: scanned up to log sequence number 233 1825512960 InnoDB: Doing recovery: scanned up to log sequence number 233 1830755840 InnoDB: Doing recovery: scanned up to log sequence number 233 1835998720 InnoDB: Doing recovery: scanned up to log sequence number 233 1841241600 InnoDB: Doing recovery: scanned up to log sequence number 233 1846484480 InnoDB: Doing recovery: scanned up to log sequence number 233 1851727360 InnoDB: Doing recovery: scanned up to log sequence number 233 1856970240 InnoDB: Doing recovery: scanned up to log sequence number 233 1862213120 InnoDB: Doing recovery: scanned up to log sequence number 233 1867456000 InnoDB: Doing recovery: scanned up to log sequence number 233 1872698880 InnoDB: Doing recovery: scanned up to log sequence number 233 1877941760 InnoDB: Doing recovery: scanned up to log sequence number 233 1883184640 InnoDB: Doing recovery: scanned up to log sequence number 233 1888427520 InnoDB: Doing recovery: scanned up to log sequence number 233 1893670400 InnoDB: Doing recovery: scanned up to log sequence number 233 1898913280 InnoDB: Doing recovery: scanned up to log sequence number 233 1904156160 InnoDB: Doing recovery: scanned up to log sequence number 233 1909399040 InnoDB: Doing recovery: scanned up to log sequence number 233 1914641920 InnoDB: Doing recovery: scanned up to log sequence number 233 1919884800 InnoDB: Doing recovery: scanned up to log sequence number 233 1925127680 InnoDB: Doing recovery: scanned up to log sequence number 233 1930370560 InnoDB: Doing recovery: scanned up to log sequence number 233 1935613440 InnoDB: Doing recovery: scanned up to log sequence number 233 1940856320 InnoDB: Doing recovery: scanned up to log sequence number 233 1946099200 InnoDB: Doing recovery: scanned up to log sequence number 233 1951342080 InnoDB: Doing recovery: scanned up to log sequence number 233 1956584960 InnoDB: Doing recovery: scanned up to log sequence number 233 1961827840 InnoDB: Doing recovery: scanned up to log sequence number 233 1967070720 InnoDB: Doing recovery: scanned up to log sequence number 233 1972313600 InnoDB: Doing recovery: scanned up to log sequence number 233 1977556480 InnoDB: Doing recovery: scanned up to log sequence number 233 1982799360 InnoDB: Doing recovery: scanned up to log sequence number 233 1988042240 InnoDB: Doing recovery: scanned up to log sequence number 233 1993285120 InnoDB: Doing recovery: scanned up to log sequence number 233 1998528000 InnoDB: Doing recovery: scanned up to log sequence number 233 2003770880 InnoDB: Doing recovery: scanned up to log sequence number 233 2009013760 InnoDB: Doing recovery: scanned up to log sequence number 233 2014256640 InnoDB: Doing recovery: scanned up to log sequence number 233 2019499520 InnoDB: Doing recovery: scanned up to log sequence number 233 2024742400 InnoDB: Doing recovery: scanned up to log sequence number 233 2029985280 InnoDB: Doing recovery: scanned up to log sequence number 233 2035228160 InnoDB: Doing recovery: scanned up to log sequence number 233 2040471040 InnoDB: Doing recovery: scanned up to log sequence number 233 2045713920 InnoDB: Doing recovery: scanned up to log sequence number 233 2050956800 InnoDB: Doing recovery: scanned up to log sequence number 233 2056199680 InnoDB: Doing recovery: scanned up to log sequence number 233 2061442560 InnoDB: Doing recovery: scanned up to log sequence number 233 2066685440 InnoDB: Doing recovery: scanned up to log sequence number 233 2071928320 InnoDB: Doing recovery: scanned up to log sequence number 233 2077171200 InnoDB: Doing recovery: scanned up to log sequence number 233 2082414080 InnoDB: Doing recovery: scanned up to log sequence number 233 2087656960 InnoDB: Doing recovery: scanned up to log sequence number 233 2092899840 InnoDB: Doing recovery: scanned up to log sequence number 233 2098142720 InnoDB: Doing recovery: scanned up to log sequence number 233 2103385600 InnoDB: Doing recovery: scanned up to log sequence number 233 2108628480 InnoDB: Doing recovery: scanned up to log sequence number 233 2113871360 InnoDB: Doing recovery: scanned up to log sequence number 233 2119114240 InnoDB: Doing recovery: scanned up to log sequence number 233 2124357120 InnoDB: Doing recovery: scanned up to log sequence number 233 2129600000 InnoDB: Doing recovery: scanned up to log sequence number 233 2134842880 InnoDB: Doing recovery: scanned up to log sequence number 233 2140085760 InnoDB: Doing recovery: scanned up to log sequence number 233 2145328640 InnoDB: Doing recovery: scanned up to log sequence number 233 2150571520 InnoDB: Doing recovery: scanned up to log sequence number 233 2155814400 InnoDB: Doing recovery: scanned up to log sequence number 233 2161057280 InnoDB: Doing recovery: scanned up to log sequence number 233 2166300160 InnoDB: Doing recovery: scanned up to log sequence number 233 2171543040 InnoDB: Doing recovery: scanned up to log sequence number 233 2176785920 InnoDB: Doing recovery: scanned up to log sequence number 233 2182028800 InnoDB: Doing recovery: scanned up to log sequence number 233 2187271680 InnoDB: Doing recovery: scanned up to log sequence number 233 2192514560 InnoDB: Doing recovery: scanned up to log sequence number 233 2197757440 InnoDB: Doing recovery: scanned up to log sequence number 233 2203000320 InnoDB: Doing recovery: scanned up to log sequence number 233 2208243200 InnoDB: Doing recovery: scanned up to log sequence number 233 2213486080 InnoDB: Doing recovery: scanned up to log sequence number 233 2218728960 InnoDB: Doing recovery: scanned up to log sequence number 233 2223971840 InnoDB: Doing recovery: scanned up to log sequence number 233 2229214720 InnoDB: Doing recovery: scanned up to log sequence number 233 2234457600 InnoDB: Doing recovery: scanned up to log sequence number 233 2237510725 080916 12:46:43 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 126, file name db-main-bin.000391 InnoDB: Last MySQL binlog file position 0 399877681, file name /database/cst/bin-logs/db-cst-bin.000672 080916 12:50:46 InnoDB: Started; log sequence number 233 2237510725 080916 12:50:46 [Note] Recovering after a crash using /database/cst/bin-logs/db-cst-bin 080916 12:50:53 [Note] Starting crash recovery... 080916 12:50:53 [Note] Crash recovery finished. 080916 12:50:53 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.45-community-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition (GPL) 080916 13:02:50 [Note] Slave SQL thread initialized, starting replication in log 'db-main-bin.000391' at position 254039164, relay log '/database/cst/db/db-cst-relay.001205' position: 254039303 080916 13:02:50 [Note] Slave I/O thread: connected to master 'repl@x.x.x.x:3306', replication started in log 'db-main-bin.000391' at position 254039164