Number of processes running now: 0 031117 12:31:55 mysqld restarted 031117 12:31:57 InnoDB: Database was not shut down normally. InnoDB: Starting recovery from log files... InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 6 3585027559 InnoDB: Doing recovery: scanned up to log sequence number 6 3590269952 InnoDB: Doing recovery: scanned up to log sequence number 6 3590447957 InnoDB: 8 transaction(s) which must be rolled back or cleaned up InnoDB: in total 10 row operations to undo InnoDB: Trx id counter is 0 19055360 031117 12:31:57 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Starting rollback of uncommitted transactions InnoDB: Rolling back trx with id 0 19054931, 2 rows to undo InnoDB: Rolling back of trx id 0 19054931 completed InnoDB: Rolling back trx with id 0 19054899, 2 rows to undo InnoDB: Rolling back of trx id 0 19054899 completed InnoDB: Rolling back trx with id 0 19047330, 1 rows to undo InnoDB: Rolling back of trx id 0 19047330 completed InnoDB: Rolling back trx with id 0 19046553, 1 rows to undo InnoDB: Rolling back of trx id 0 19046553 completed InnoDB: Rolling back trx with id 0 18951796, 1 rows to undo InnoDB: Rolling back of trx id 0 18951796 completed InnoDB: Rolling back trx with id 0 17091365, 1 rows to undo InnoDB: Rolling back of trx id 0 17091365 completed InnoDB: Rolling back trx with id 0 16773745, 1 rows to undo InnoDB: Rolling back of trx id 0 16773745 completed InnoDB: Rolling back trx with id 0 16713917, 1 rows to undo InnoDB: Rolling back of trx id 0 16713917 completed InnoDB: Rollback of uncommitted transactions completed InnoDB: Last MySQL binlog file position 0 398660759, file name /data4/mysql/grays-bin.008 031117 12:31:59 InnoDB: Flushing modified pages from the buffer pool... 031117 12:32:00 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.12-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388600 read_buffer_size=131072 sort_buffer_size=2097144 max_used_connections=14 max_connections=100 threads_connected=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x4fc22568 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfd4fc78, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x80741ea 0x8286058 (nil) New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x87c8cd0 = INSERT INTO fn_mon_temp.temp_C5E24D898DAFEDAA505FD0FA25433CDE_20031117123210 SELECT NULL, o.OrderID, o.InitialDispatchTimeHost, IF(st.ArrivalTimeHost IS NOT NULL AND st.WindowStartTimeHost IS NOT NULL AND st.WindowEndTimeHost IS NOT NULL AND o.State='EMPTY', CASE WHEN (st.ArrivalTimeHostst.WindowEndTimeHost) THEN (-(UNIX_TIMESTAMP(st.WindowEndTimeHost)-UNIX_TIMESTAMP(st.ArrivalTimeHost))/3600) WHEN (st.ArrivalTimeHost>=st.WindowStartTimeHost AND st.ArrivalTimeHost<=st.WindowEndTimeHost) THEN 0 ELSE NULL END, NULL), CASE WHEN (o.State='DISP') THEN 'Dispatched' WHEN (o.State='CAN') THEN 'Canceled' ELSE 'Complete' END, o.ShipperCustID,o.BillToCustID,s1.CityName,s1.RegionCode,s2.CityName,s2.RegionCode FROM orders o, stop s1, stop s2 LEFT OUTER JOIN dispatch_order AS do ON do.ClientID=o.ClientID AND do.OrderID=o.OrderID LEFT OUTER JOIN dispatch AS d ON d thd->thread_id=35 Successfully dumped variables, if you ran with --log, take a look at the details of what thread 35 did to cause the crash. In some cases of really bad corruption, the values shown above may be invalid. The manual page at http://www.mysql.com/doc/C/r/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 0 031117 12:32:12 mysqld restarted 031117 12:32:12 Can't start server: Bind on TCP/IP port: Address already in use 031117 12:32:12 Do you already have another mysqld server running on port: 3306 ? 031117 12:32:12 Aborting 031117 12:32:12 /usr/sbin/mysqld: Shutdown Complete 031117 12:32:12 mysqld ended 031117 12:37:21 mysqld started 031117 12:37:22 InnoDB: Database was not shut down normally. InnoDB: Starting recovery from log files... InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 6 3590449776 InnoDB: Doing recovery: scanned up to log sequence number 6 3590478145 InnoDB: 6 transaction(s) which must be rolled back or cleaned up InnoDB: in total 9 row operations to undo InnoDB: Trx id counter is 0 19055872 031117 12:37:22 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Starting rollback of uncommitted transactions InnoDB: Rolling back trx with id 0 19055502, 1 rows to undo InnoDB: Rolling back of trx id 0 19055502 completed InnoDB: Rolling back trx with id 0 19055481, 3 rows to undo InnoDB: Rolling back of trx id 0 19055481 completed InnoDB: Rolling back trx with id 0 19055425, 2 rows to undo InnoDB: Rolling back of trx id 0 19055425 completed InnoDB: Rolling back trx with id 0 19055403, 1 rows to undo InnoDB: Rolling back of trx id 0 19055403 completed InnoDB: Rolling back trx with id 0 19055399, 1 rows to undo InnoDB: Rolling back of trx id 0 19055399 completed InnoDB: Rolling back trx with id 0 19055388, 1 rows to undo InnoDB: Rolling back of trx id 0 19055388 completed InnoDB: Rollback of uncommitted transactions completed InnoDB: Last MySQL binlog file position 0 5943, file name /data4/mysql/grays-bin.009 031117 12:37:23 InnoDB: Flushing modified pages from the buffer pool... 031117 12:37:23 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.12-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388600 read_buffer_size=131072 sort_buffer_size=2097144 max_used_connections=11 max_connections=100 threads_connected=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x87ab288 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfdcfc78, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x80741ea 0x8286058 (nil) New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x87d4f38 = INSERT INTO fn_mon_temp.temp_3A8D9B0E4F29DF7B005531420FD31D6B_20031117124323 SELECT NULL, o.OrderID, o.InitialDispatchTimeHost, IF(st.ArrivalTimeHost IS NOT NULL AND st.WindowStartTimeHost IS NOT NULL AND st.WindowEndTimeHost IS NOT NULL AND o.State='EMPTY', CASE WHEN (st.ArrivalTimeHostst.WindowEndTimeHost) THEN (-(UNIX_TIMESTAMP(st.WindowEndTimeHost)-UNIX_TIMESTAMP(st.ArrivalTimeHost))/3600) WHEN (st.ArrivalTimeHost>=st.WindowStartTimeHost AND st.ArrivalTimeHost<=st.WindowEndTimeHost) THEN 0 ELSE NULL END, NULL), CASE WHEN (o.State='DISP') THEN 'Dispatched' WHEN (o.State='CAN') THEN 'Canceled' ELSE 'Complete' END, o.ShipperCustID,o.BillToCustID,s1.CityName,s1.RegionCode,s2.CityName,s2.RegionCode FROM orders o, stop s1, stop s2 LEFT OUTER JOIN dispatch_order AS do ON do.ClientID=o.ClientID AND do.OrderID=o.OrderID LEFT OUTER JOIN dispatch AS d ON d thd->thread_id=101 Successfully dumped variables, if you ran with --log, take a look at the details of what thread 101 did to cause the crash. In some cases of really bad corruption, the values shown above may be invalid. The manual page at http://www.mysql.com/doc/C/r/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 0 031117 12:43:24 mysqld restarted 031117 12:43:25 InnoDB: Database was not shut down normally. InnoDB: Starting recovery from log files... InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 6 3591118352 InnoDB: Doing recovery: scanned up to log sequence number 6 3594943351 InnoDB: 5 transaction(s) which must be rolled back or cleaned up InnoDB: in total 14 row operations to undo InnoDB: Trx id counter is 0 19056640 031117 12:43:25 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Starting rollback of uncommitted transactions InnoDB: Rolling back trx with id 0 19056247, 1 rows to undo InnoDB: Rolling back of trx id 0 19056247 completed InnoDB: Rolling back trx with id 0 19056224, 10 rows to undo InnoDB: Rolling back of trx id 0 19056224 completed InnoDB: Rolling back trx with id 0 19056167, 1 rows to undo InnoDB: Rolling back of trx id 0 19056167 completed InnoDB: Rolling back trx with id 0 19055956, 1 rows to undo InnoDB: Rolling back of trx id 0 19055956 completed InnoDB: Rolling back trx with id 0 19055942, 1 rows to undo InnoDB: Rolling back of trx id 0 19055942 completed InnoDB: Rollback of uncommitted transactions completed InnoDB: Last MySQL binlog file position 0 4852624, file name /data4/mysql/grays-bin.010 031117 12:43:27 InnoDB: Flushing modified pages from the buffer pool... 031117 12:43:29 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.12-log' socket: '/var/lib/mysql/mysql.sock' port: 3306