| Bug #5414 | innodb assertion failure in 4.1.4 | ||
|---|---|---|---|
| Submitted: | 4 Sep 2004 20:28 | Modified: | 18 Oct 2004 16:37 |
| Reporter: | Donny Simonton | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S1 (Critical) |
| Version: | 4.1.4 | OS: | Linux (Linux) |
| Assigned to: | Heikki Tuuri | CPU Architecture: | Any |
[4 Sep 2004 20:28]
Donny Simonton
[5 Sep 2004 0:07]
MySQL Verification Team
Not enough information was provided for us to be able to handle this bug. Please re-read the instructions at http://bugs.mysql.com/how-to-report.php If you can provide more information, feel free to add it to this bug and change the status back to 'Open'. Thank you for your interest in MySQL.
[5 Sep 2004 21:46]
Marko Mäkelä
The assertion failure occurs in the function sync_array_reserve_cell(). Apparently, InnoDB is trying to create more locking objects (mutexes or R/W-locks) than has been allocated. The limit is 1000 or 10000, depending on the configuration. Did you obtain the stack trace from an official pre-compiled binary made by MySQL Ab? If so, which one? If not, could you please try to resolve the stack trace (using gdb or similar tools)? Have you found any common denominator for the crashes in the query log?
[5 Sep 2004 23:38]
Donny Simonton
We ran into the problem once again a few hours ago. I'll see if I can give a little more history on it first. Last Monday, Tuesday, and Wednesday we had Peter from mysql do a 3 day onsite consulting. He recommended that we migrate some of our tables to innodb because of some problems that he saw. The box is a quad xeon, with 6 gigs of memory, but now we know that anything more than 4 gigs is kind of worthless on a 32 bit machine. On Tuesday we plan on upgrading to a quad opteron with 32 gigs of memory so this problem may go away completely when that happens. This is a master box which replicates to 2 slaves. But 4 webservers connect to it and do selects and inserts. It averages about 3000 queries per second. We are running the 4.1.4 RPM from MySQL. We are running 2.6.8.1 linux kernel. Below are the settings that Peter setup for us on the server on Tuesday. # Uncomment the following if you are using InnoDB tables # PZ: This box has everything on one set of drives so there is no # reason to put logs in separate place # Innodb_file_per_table is not used as it is not well tested yet # innodb_flush_logs_at_trx_commit=2 give MyISAM like redundancy #innodb_data_home_dir = /www/mysql/innodb/ innodb_data_file_path = ibdata1:1G:autoextend #innodb_log_group_home_dir = /tmp/innodb-logs/ #innodb_log_arch_dir = /tmp/innodb-logs/ # You can set .._buffer_pool_size up to 50 - 80 % # of RAM but beware of setting memory usage too high innodb_buffer_pool_size = 512M innodb_additional_mem_pool_size = 16M # Set .._log_file_size to 25 % of buffer pool size innodb_log_file_size = 256M innodb_log_buffer_size = 8M innodb_flush_log_at_trx_commit =2 #innodb_lock_wait_timeout = 50 #innodb_file_per_table innodb_flush_method=O_DIRECT innodb_thread_concurrency=16 On Thursday we did drop innodb_buffer_pool_size from 768 to 512 because we could not get more than 2000 connections to the box. In most cases we never have more than 100-200 at a time. But for some reason sometimes everything seems to want to stall out and that's where the extra connections come in more than anything. Also Peter turned this on as well, low_priority_updates since we have so many selects and inserts going on at the same time. This is what we got today we MySQL crashed. 040905 13:29:39InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:39InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:39InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:39InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:39InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:40InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:40InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:40InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:40InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:41InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:41InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:41InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:44InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:44InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:44InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:44InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:44InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:45InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:45InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:50InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:50InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:50InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:50InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:51InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:51InnoDB: Warning: cannot find a free slot for an undo log. Do you have too InnoDB: many active transactions running concurrently?040905 13:29:51InnoDB: Assertion failure in thread 3375066508 in file sync0arr.c line 384 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. See section 6.1 of InnoDB: http://www.innodb.com/ibman.php about forcing recovery. mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=536870912 read_buffer_size=258048 max_used_connections=2497 max_connections=4100 threads_connected=2497 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 3656655 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. You seem to be running 32-bit Linux and have 2497 concurrent connections. If you have not changed STACK_SIZE in LinuxThreads and built the binary yourself, LinuxThreads is quite likely to steal a part of the global heap for the thread stack. Please read http://www.mysql.com/doc/en/Linux.html thd=0xa91491e8 InnoDB: Thread 3375275815 stopped in file btr0pcur.c line 205 InnoDB: Thread 3376232300 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 3376282324 stopped in file sync0arr.c line 339 InnoDB: Thread 3375912669 stopped in file sync0arr.c line 339 InnoDB: Thread 3374914823 stopped in file sync0arr.c line 339 InnoDB: Thread 3376299072 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 3371394499 stopped in file sync0arr.c line 339 InnoDB: Thread 3365818871 stopped in file ha_innodb.cc line 631 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xa4e3e6a8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x808c817 0x82de458 0x8263d50 0x8267b77 0x8265d7f 0x81b0440 0x817e467 0x818b8b4 0x818c6b5 0x818c8b2 0x8174eab 0x81005e0 0x80d6580 0x809d65c 0x80a05f9 0x809aaff 0x809a484 0x8099b97 0x82dbc0c 0x830519a New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0xbedaee8 = UPDATE DomainHit SET hit = NOW() WHERE domain = 'chinaexhibition.org'/* /index.php */ thd->thread_id=7547226 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. InnoDB: Thread 3373819962 stopped in file btr0pcur.c line 205 InnoDB: Thread 3372492550 stopped in file btr0pcur.c line 205 InnoDB: Thread 3376257742 stopped in file sync0arr.c line 339 InnoDB: Thread 3375937312 stopped in file btr0pcur.c line 205 InnoDB: Thread 3376097216 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 3375136205 stopped in file trx0trx.c line 1488 InnoDB: Thread 3371951387 stopped in file trx0trx.c line 1488 InnoDB: Thread 3368408231 stopped in file sync0arr.c line 339 InnoDB: Thread 3375546899 stopped in file btr0pcur.c line 205 InnoDB: Thread 3375534554 stopped in file btr0pcur.c line 205 InnoDB: Thread 3376323929 stopped in file sync0arr.c line 339 Number of processes running now: 0 When I run a stack trace on the above information, I get: [root@parking-master tmp]# resolve_stack_dump -s /usr/lib/mysql/mysqld.sym -n /tmp/mysql.stack 0x808c817 handle_segfault + 423 0x82de458 pthread_sighandler + 184 0x8263d50 sync_array_reserve_cell + 736 0x8267b77 mutex_spin_wait + 615 0x8265d7f rw_lock_s_lock_spin + 591 0x81b0440 btr_cur_search_to_nth_level + 1568 0x817e467 row_search_index_entry + 71 0x818b8b4 row_upd_sec_index_entry + 292 0x818c6b5 row_upd + 197 0x818c8b2 row_upd_step + 306 0x8174eab row_update_for_mysql + 667 0x81005e0 update_row__11ha_innobasePCcPc + 288 0x80d6580 mysql_update__FP3THDP13st_table_listRt4List1Z4ItemT2P4ItemUiP8st_orderUl15enum_duplicates + 2432 0x809d65c mysql_execute_command__FP3THD + 7340 0x80a05f9 mysql_parse__FP3THDPcUi + 169 0x809aaff dispatch_command__F19enum_server_commandP3THDPcUi + 1647 0x809a484 do_command__FP3THD + 196 0x8099b97 handle_one_connection + 615 0x82dbc0c pthread_start_thread + 220 0x830519a thread_start + 4 I'm not sure what other information would be needed if anything. Please let me know. Donny
[6 Sep 2004 4:21]
Donny Simonton
Just crashed again. 040905 21:06:50InnoDB: Assertion failure in thread 785320151 in file sync0arr.c line 384 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. See section 6.1 of InnoDB: http://www.innodb.com/ibman.php about forcing recovery. mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=536870912 read_buffer_size=258048 max_used_connections=3589 max_connections=4100 threads_connected=2248 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 3656655 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. You seem to be running 32-bit Linux and have 2248 concurrent connections. If you have not changed STACK_SIZE in LinuxThreads and built the binary yourself, LinuxThreads is quite likely to steal a part of the global heap for the thread stack. Please read http://www.mysql.com/doc/en/Linux.html thd=0xa6f09688 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xae4dddb8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x808c817 0x82de458 0x8263d50 0x8267b77 InnoDB: Thread 16389 stopped in file os0sync.c line 501 0x81e864c 0x81d5b95 0x81b4697 0x81b5312 0x818bd58 0x818c5c0 0x818c65a 0x818c8b2 0x8174eab 0x81005e0 0x80d6580 0x809d65c 0x80a05f9 0x809aaff 0x809a484 0x8099b97 0x82dbc0c 0x830519a New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0xee058e0 = UPDATE DomainHit SET hit = NOW() WHERE domain = 'zinfommx.com'/* /index.php */ thd->thread_id=2100535 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. InnoDB: Thread 785324248 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 785332442 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 785328345 stopped in file sync0arr.c line 339
[6 Sep 2004 6:28]
Peter Zaitsev
I've repeated this problem quite easily by running a lot of (3000) concurrent connections in sysbench OTPL test. Sysbench is available in "sysbench" Internal BitKeeper tree but I would guess any large number of concurrent transactions will lead to same behavior. I used Dynamic (NPTL) build to get such many connections but Static RPM also allows them. 040905 21:21:03 [ERROR] Found an entry in the 'db' table with empty database name; Skipped 040905 21:21:03 [WARNING] Can't open and lock time zone table: Table 'mysql.time_zone_leap_second' doesn't exist trying to live without them /home/pz/bk/mysql-4.1/sql/mysqld: ready for connections. Version: '4.1.5-gamma-log' socket: '/tmp/mysql.sock' port: 3306 Source distribution 040905 21:22:41InnoDB: Assertion failure in thread 1716661168 in file sync0arr.c line 384 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. See section 6.1 of InnoDB: http://www.innodb.com/ibman.php about forcing recovery. InnoDB: Thread 1699601328 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1443904432 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1655446448 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1565330352 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1464777648 stopped in file sync0arr.c line 339 InnoDB: Thread 1703013296 stopped in file sync0arr.c line 339 InnoDB: Thread 1653439408 stopped in file sync0arr.c line 339 InnoDB: Thread 1648421808 stopped in file sync0arr.c line 339 InnoDB: Thread 1565531056 stopped in file sync0arr.c line 339 InnoDB: Thread 1519770544 stopped in file sync0arr.c line 339 InnoDB: Thread 1446914992 stopped in file sync0arr.c line 339 InnoDB: Thread 1821027248 stopped in file sync0arr.c line 339 InnoDB: Thread 1531009968 stopped in file sync0arr.c line 339 InnoDB: Thread 1631562672 stopped in file sync0arr.c line 339 InnoDB: Thread 1650428848 stopped in file sync0arr.c line 339 InnoDB: Thread 1629154224 stopped in file sync0arr.c line 339 InnoDB: Thread 1567337392 stopped in file sync0arr.c line 339 InnoDB: Thread 1585400752 stopped in file sync0arr.c line 339 InnoDB: Thread 1552284592 stopped in file sync0arr.c line 339 InnoDB: Thread 1558105008 stopped in file sync0arr.c line 339 InnoDB: Thread 1613499312 stopped in file sync0arr.c line 339 InnoDB: Thread 1485851568 stopped in file sync0arr.c line 339 InnoDB: Thread 1587407792 stopped in file sync0arr.c line 339 InnoDB: Thread 1586805680 stopped in file sync0arr.c line 339 InnoDB: Thread 1562721200 stopped in file sync0arr.c line 339 InnoDB: Thread 1597643696 stopped in file sync0arr.c line 339 InnoDB: Thread 1624337328 stopped in file sync0arr.c line 339 InnoDB: Thread 1771453360 stopped in file sync0arr.c line 339 InnoDB: Thread 1706425264 stopped in file sync0arr.c line 339 InnoDB: Thread 1562319792 stopped in file sync0arr.c line 339 InnoDB: Thread 1576369072 stopped in file sync0arr.c line 339 InnoDB: Thread 1395133360 stopped in file sync0arr.c line 339 InnoDB: Thread 1587809200 stopped in file sync0arr.c line 339 InnoDB: Thread 1536228272 stopped in file sync0arr.c line 339 InnoDB: Thread 1653840816 stopped in file sync0arr.c line 339 InnoDB: Thread 1610689456 stopped in file sync0arr.c line 339 InnoDB: Thread 1539038128 stopped in file sync0arr.c line 339 InnoDB: Thread 1578576816 stopped in file sync0arr.c line 339 InnoDB: Thread 1634171824 stopped in file sync0arr.c line 339 InnoDB: Thread 1583995824 stopped in file sync0arr.c line 339 InnoDB: Thread 1782291376 stopped in file sync0arr.c line 339 InnoDB: Thread 1659661232 stopped in file sync0arr.c line 339 InnoDB: Thread 1654844336 stopped in file sync0arr.c line 339 InnoDB: Thread 1664076720 stopped in file sync0arr.c line 339 InnoDB: Thread 1688562608 stopped in file sync0arr.c line 339 InnoDB: Thread 1637383088 stopped in file sync0arr.c line 339 InnoDB: Thread 1566133168 stopped in file sync0arr.c line 339 InnoDB: Thread 1511943088 stopped in file sync0arr.c line 339 InnoDB: Thread 1652636592 stopped in file sync0arr.c line 339 InnoDB: Thread 1835477936 stopped in file sync0arr.c line 339 InnoDB: Thread 1481034672 stopped in file sync0arr.c line 339 InnoDB: Thread 1783094192 stopped in file sync0arr.c line 339 InnoDB: Thread 1809587120 stopped in file sync0arr.c line 339 InnoDB: Thread 1810189232 stopped in file sync0arr.c line 339 InnoDB: Thread 1594432432 stopped in file sync0arr.c line 339 InnoDB: Thread 1666886576 stopped in file sync0arr.c line 339 InnoDB: Thread 1600252848 stopped in file sync0arr.c line 339 InnoDB: Thread 1381284784 stopped in file sync0arr.c line 339 InnoDB: Thread 1817615280 stopped in file sync0arr.c line 339 InnoDB: Thread 1824037808 stopped in file sync0arr.c line 339 InnoDB: Thread 1434471344 stopped in file sync0arr.c line 339 InnoDB: Thread 1566534576 stopped in file sync0arr.c line 339 InnoDB: Thread 1824439216 stopped in file sync0arr.c line 339 InnoDB: Thread 1463372720 stopped in file sync0arr.c line 339 InnoDB: Thread 1397742512 stopped in file sync0arr.c line 339 InnoDB: Thread 1492876208 stopped in file sync0arr.c line 339 InnoDB: Thread 1496488880 stopped in file sync0arr.c line 339 InnoDB: Thread 1376668592 stopped in file sync0arr.c line 339 InnoDB: Thread 1414400944 stopped in file sync0arr.c line 339 InnoDB: Thread 1636981680 stopped in file sync0arr.c line 339 InnoDB: Thread 1720675248 stopped in file sync0arr.c line 339 InnoDB: Thread 1627147184 stopped in file sync0arr.c line 339 InnoDB: Thread 1648622512 stopped in file sync0arr.c line 339 InnoDB: Thread 1801157552 stopped in file sync0arr.c line 339 InnoDB: Thread 1497492400 stopped in file sync0arr.c line 339 InnoDB: Thread 1424034736 stopped in file sync0arr.c line 339 InnoDB: Thread 1410988976 stopped in file sync0arr.c line 339 InnoDB: Thread 1840094128 stopped in file sync0arr.c line 339 InnoDB: Thread 1393527728 stopped in file sync0arr.c line 339 InnoDB: Thread 1515355056 stopped in file sync0arr.c line 339 InnoDB: Thread 1832868784 stopped in file sync0arr.c line 339 InnoDB: Thread 1436879792 stopped in file sync0arr.c line 339 InnoDB: Thread 1518365616 stopped in file sync0arr.c line 339 InnoDB: Thread 1402960816 stopped in file sync0arr.c line 339 InnoDB: Thread 1504316336 stopped in file sync0arr.c line 339 InnoDB: Thread 1812196272 stopped in file sync0arr.c line 339 InnoDB: Thread 1382890416 stopped in file sync0arr.c line 339 InnoDB: Thread 1377872816 stopped in file sync0arr.c line 339 InnoDB: Thread 1366834096 stopped in file sync0arr.c line 339 InnoDB: Thread 1490267056 stopped in file sync0arr.c line 339 InnoDB: Thread 1468189616 stopped in file sync0arr.c line 339 InnoDB: Thread 1478425520 stopped in file sync0arr.c line 339 InnoDB: Thread 1500101552 stopped in file sync0arr.c line 339 InnoDB: Thread 1822030768 stopped in file sync0arr.c line 339 InnoDB: Thread 1572555696 stopped in file sync0arr.c line 339 InnoDB: Thread 1453939632 stopped in file sync0arr.c line 339 InnoDB: Thread 1765432240 stopped in file sync0arr.c line 339 InnoDB: Thread 1671703472 stopped in file sync0arr.c line 339 InnoDB: Thread 1652837296 stopped in file sync0arr.c line 339 InnoDB: Thread 1509333936 stopped in file sync0arr.c line 339 InnoDB: Thread 1467988912 stopped in file sync0arr.c line 339 InnoDB: Thread 1389915056 stopped in file sync0arr.c line 339 InnoDB: Thread 1837886384 stopped in file sync0arr.c line 339 InnoDB: Thread 1372253104 stopped in file sync0arr.c line 339 InnoDB: Thread 1441094576 stopped in file sync0arr.c line 339 InnoDB: Thread 1454943152 stopped in file sync0arr.c line 339 InnoDB: Thread 1478224816 stopped in file sync0arr.c line 339 InnoDB: Thread 1405369264 stopped in file sync0arr.c line 339 InnoDB: Thread 1831263152 stopped in file sync0arr.c line 339 InnoDB: Thread 1834675120 stopped in file sync0arr.c line 339 InnoDB: Thread 1822632880 stopped in file sync0arr.c line 339 InnoDB: Thread 1825442736 stopped in file sync0arr.c line 339 InnoDB: Thread 1400953776 stopped in file sync0arr.c line 339 InnoDB: Thread 1404165040 stopped in file sync0arr.c line 339 InnoDB: Thread 1454140336 stopped in file sync0arr.c line 339 InnoDB: Thread 1823034288 stopped in file sync0arr.c line 339 InnoDB: Thread 1395534768 stopped in file sync0arr.c line 339 InnoDB: Thread 1829456816 stopped in file sync0arr.c line 339 InnoDB: Thread 1396337584 stopped in file sync0arr.c line 339 InnoDB: Thread 1828654000 stopped in file sync0arr.c line 339 InnoDB: Thread 1829256112 stopped in file sync0arr.c line 339 InnoDB: Thread 1831062448 stopped in file sync0arr.c line 339 InnoDB: Thread 1412996016 stopped in file sync0arr.c line 339 InnoDB: Thread 1519369136 stopped in file sync0arr.c line 339 InnoDB: Thread 1481636784 stopped in file sync0arr.c line 339 InnoDB: Thread 1380080560 stopped in file sync0arr.c line 339 InnoDB: Thread 1495886768 stopped in file sync0arr.c line 339 InnoDB: Thread 1384897456 stopped in file sync0arr.c line 339 InnoDB: Thread 1832668080 stopped in file sync0arr.c line 339 InnoDB: Thread 1840495536 stopped in file sync0arr.c line 339 InnoDB: Thread 1364425648 stopped in file sync0arr.c line 339 InnoDB: Thread 1832065968 stopped in file sync0arr.c line 339 InnoDB: Thread 1369443248 stopped in file sync0arr.c line 339 InnoDB: Thread 1405971376 stopped in file sync0arr.c line 339 InnoDB: Thread 1480633264 stopped in file sync0arr.c line 339 InnoDB: Thread 1673911216 stopped in file sync0arr.c line 339 InnoDB: Thread 1783495600 stopped in file sync0arr.c line 339 InnoDB: Thread 1782893488 stopped in file sync0arr.c line 339 InnoDB: Thread 1793129392 stopped in file sync0arr.c line 339 InnoDB: Thread 1788111792 stopped in file sync0arr.c line 339 InnoDB: Thread 1773661104 stopped in file sync0arr.c line 339 InnoDB: Thread 1769847728 stopped in file sync0arr.c line 339 InnoDB: Thread 1760013232 stopped in file sync0arr.c line 339 InnoDB: Thread 1815608240 stopped in file sync0arr.c line 339 InnoDB: Thread 1742551984 stopped in file sync0arr.c line 339 InnoDB: Thread 1779682224 stopped in file sync0arr.c line 339 InnoDB: Thread 1739942832 stopped in file sync0arr.c line 339 InnoDB: Thread 1773861808 stopped in file sync0arr.c line 339 InnoDB: Thread 1780083632 stopped in file sync0arr.c line 339 InnoDB: Thread 1733921712 stopped in file sync0arr.c line 339 InnoDB: Thread 1744358320 stopped in file sync0arr.c line 339 InnoDB: Thread 1812999088 stopped in file sync0arr.c line 339 InnoDB: Thread 1495083952 stopped in file sync0arr.c line 339 InnoDB: Thread 1489464240 stopped in file sync0arr.c line 339 InnoDB: Thread 1409784752 stopped in file sync0arr.c line 339 InnoDB: Thread 1672907696 stopped in file sync0arr.c line 339 InnoDB: Thread 1400552368 stopped in file sync0arr.c line 339 InnoDB: Thread 1398746032 stopped in file sync0arr.c line 339 InnoDB: Thread 1834073008 stopped in file sync0arr.c line 339 InnoDB: Thread 1460562864 stopped in file sync0arr.c line 339 InnoDB: Thread 1479228336 stopped in file sync0arr.c line 339 InnoDB: Thread 1421425584 stopped in file sync0arr.c line 339 InnoDB: Thread 1674111920 stopped in file sync0arr.c line 339 InnoDB: Thread 1518967728 stopped in file sync0arr.c line 339 InnoDB: Thread 1836280752 stopped in file sync0arr.c line 339 InnoDB: Thread 1801358256 stopped in file sync0arr.c line 339 InnoDB: Thread 1483041712 stopped in file sync0arr.c line 339 InnoDB: Thread 1445911472 stopped in file sync0arr.c line 339 InnoDB: Thread 1517362096 stopped in file sync0arr.c line 339 InnoDB: Thread 1418214320 stopped in file sync0arr.c line 339 InnoDB: Thread 1819622320 stopped in file sync0arr.c line 339 InnoDB: Thread 1818016688 stopped in file sync0arr.c line 339 InnoDB: Thread 1495686064 stopped in file sync0arr.c line 339 InnoDB: Thread 1382489008 stopped in file sync0arr.c line 339 InnoDB: Thread 1465580464 stopped in file sync0arr.c line 339 InnoDB: Thread 1446714288 stopped in file sync0arr.c line 339 InnoDB: Thread 1768041392 stopped in file sync0arr.c line 339 InnoDB: Thread 1423633328 stopped in file sync0arr.c line 339 InnoDB: Thread 1371249584 stopped in file sync0arr.c line 339 InnoDB: Thread 1474612144 stopped in file sync0arr.c line 339 InnoDB: Thread 1821830064 stopped in file sync0arr.c line 339 InnoDB: Thread 1379478448 stopped in file sync0arr.c line 339 InnoDB: Thread 1425439664 stopped in file sync0arr.c line 339 InnoDB: Thread 1407978416 stopped in file sync0arr.c line 339 InnoDB: Thread 1499098032 stopped in file sync0arr.c line 339 InnoDB: Thread 1512344496 stopped in file sync0arr.c line 339 InnoDB: Thread 1673509808 stopped in file sync0arr.c line 339 InnoDB: Thread 1491872688 stopped in file sync0arr.c line 339 InnoDB: Thread 1456749488 stopped in file sync0arr.c line 339 InnoDB: Thread 1830058928 stopped in file sync0arr.c line 339 InnoDB: Thread 1510337456 stopped in file sync0arr.c line 339 InnoDB: Thread 1513749424 stopped in file sync0arr.c line 339 InnoDB: Thread 1481235376 stopped in file sync0arr.c line 339 InnoDB: Thread 1839893424 stopped in file sync0arr.c line 339 InnoDB: Thread 1514752944 stopped in file sync0arr.c line 339 InnoDB: Thread 1839291312 stopped in file sync0arr.c line 339 InnoDB: Thread 1391119280 stopped in file sync0arr.c line 339 InnoDB: Thread 1368038320 stopped in file sync0arr.c line 339 InnoDB: Thread 1830259632 stopped in file sync0arr.c line 339 InnoDB: Thread 1499298736 stopped in file sync0arr.c line 339 InnoDB: Thread 1453337520 stopped in file sync0arr.c line 339 InnoDB: Thread 1478626224 stopped in file sync0arr.c line 339 InnoDB: Thread 1271901104 stopped in file sync0arr.c line 339 InnoDB: Thread 1474010032 stopped in file sync0arr.c line 339 InnoDB: Thread 1411591088 stopped in file sync0arr.c line 339 InnoDB: Thread 1412193200 stopped in file sync0arr.c line 339 InnoDB: Thread 1830861744 stopped in file sync0arr.c line 339 InnoDB: Thread 1674312624 stopped in file sync0arr.c line 339 InnoDB: Thread 1378474928 stopped in file sync0arr.c line 339 InnoDB: Thread 1498897328 stopped in file sync0arr.c line 339 InnoDB: Thread 1672706992 stopped in file sync0arr.c line 339 InnoDB: Thread 1436478384 stopped in file sync0arr.c line 339 InnoDB: Thread 1454742448 stopped in file sync0arr.c line 339 InnoDB: Thread 1466784688 stopped in file sync0arr.c line 339 InnoDB: Thread 1425238960 stopped in file sync0arr.c line 339 InnoDB: Thread 1365429168 stopped in file sync0arr.c line 339 InnoDB: Thread 1459559344 stopped in file sync0arr.c line 339 InnoDB: Thread 1399548848 stopped in file sync0arr.c line 339 InnoDB: Thread 1404365744 stopped in file sync0arr.c line 339 InnoDB: Thread 1368439728 stopped in file sync0arr.c line 339 InnoDB: Thread 1817414576 stopped in file sync0arr.c line 339 InnoDB: Thread 1464978352 stopped in file sync0arr.c line 339 InnoDB: Thread 1510939568 stopped in file sync0arr.c line 339 InnoDB: Thread 1476819888 stopped in file sync0arr.c line 339 InnoDB: Thread 1366633392 stopped in file sync0arr.c line 339 InnoDB: Thread 1493879728 stopped in file sync0arr.c line 339 InnoDB: Thread 1366031280 stopped in file sync0arr.c line 339 InnoDB: Thread 1376467888 stopped in file sync0arr.c line 339 InnoDB: Thread 1478024112 stopped in file sync0arr.c line 339 InnoDB: Thread 1381084080 stopped in file sync0arr.c line 339 InnoDB: Thread 1397541808 stopped in file sync0arr.c line 339 InnoDB: Thread 1479830448 stopped in file sync0arr.c line 339 InnoDB: Thread 1431059376 stopped in file sync0arr.c line 339 InnoDB: Thread 1816210352 stopped in file sync0arr.c line 339 InnoDB: Thread 1831664560 stopped in file sync0arr.c line 339 InnoDB: Thread 1479629744 stopped in file sync0arr.c line 339 InnoDB: Thread 1496689584 stopped in file sync0arr.c line 339 InnoDB: Thread 1413196720 stopped in file sync0arr.c line 339 InnoDB: Thread 1513348016 stopped in file sync0arr.c line 339 InnoDB: Thread 1387305904 stopped in file sync0arr.c line 339 InnoDB: Thread 1832266672 stopped in file sync0arr.c line 339 InnoDB: Thread 1385098160 stopped in file sync0arr.c line 339 InnoDB: Thread 1393327024 stopped in file sync0arr.c line 339 InnoDB: Thread 1403763632 stopped in file sync0arr.c line 339 InnoDB: Thread 1402158000 stopped in file sync0arr.c line 339 InnoDB: Thread 1823435696 stopped in file sync0arr.c line 339 InnoDB: Thread 1820625840 stopped in file sync0arr.c line 339 InnoDB: Thread 1375263664 stopped in file sync0arr.c line 339 InnoDB: Thread 1448119216 stopped in file sync0arr.c line 339 InnoDB: Thread 1821428656 stopped in file sync0arr.c line 339 InnoDB: Thread 1426041776 stopped in file sync0arr.c line 339 InnoDB: Thread 1468390320 stopped in file sync0arr.c line 339 InnoDB: Thread 1436679088 stopped in file sync0arr.c line 339 InnoDB: Thread 1373658032 stopped in file sync0arr.c line 339 InnoDB: Thread 1827048368 stopped in file sync0arr.c line 339 InnoDB: Thread 1495485360 stopped in file sync0arr.c line 339 InnoDB: Thread 1421024176 stopped in file sync0arr.c line 339 InnoDB: Thread 1825844144 stopped in file sync0arr.c line 339 InnoDB: Thread 1408379824 stopped in file sync0arr.c line 339 InnoDB: Thread 1547266992 stopped in file sync0arr.c line 339 InnoDB: Thread 1828051888 stopped in file sync0arr.c line 339 InnoDB: Thread 1714854832 stopped in file sync0arr.c line 339 InnoDB: Thread 1400753072 stopped in file sync0arr.c line 339 InnoDB: Thread 1807379376 stopped in file sync0arr.c line 339 InnoDB: Thread 1439087536 stopped in file sync0arr.c line 339 InnoDB: Thread 1369844656 stopped in file sync0arr.c line 339 InnoDB: Thread 1824840624 stopped in file sync0arr.c line 339 InnoDB: Thread 1674914736 stopped in file sync0arr.c line 339 InnoDB: Thread 1833069488 stopped in file sync0arr.c line 339 InnoDB: Thread 1822833584 stopped in file sync0arr.c line 339 InnoDB: Thread 1506323376 stopped in file sync0arr.c line 339 InnoDB: Thread 1367235504 stopped in file sync0arr.c line 339 InnoDB: Thread 1817815984 stopped in file sync0arr.c line 339 InnoDB: Thread 1433066416 stopped in file sync0arr.c line 339 InnoDB: Thread 1383492528 stopped in file sync0arr.c line 339 InnoDB: Thread 1834273712 stopped in file sync0arr.c line 339 InnoDB: Thread 1480031152 stopped in file sync0arr.c line 339 InnoDB: Thread 1452936112 stopped in file sync0arr.c line 339 InnoDB: Thread 1593629616 stopped in file sync0arr.c line 339 InnoDB: Thread 1811594160 stopped in file sync0arr.c line 339 InnoDB: Thread 1663273904 stopped in file sync0arr.c line 339 InnoDB: Thread 1826646960 stopped in file sync0arr.c line 339 InnoDB: Thread 1415003056 stopped in file sync0arr.c line 339 InnoDB: Thread 1424436144 stopped in file sync0arr.c line 339 InnoDB: Thread 1422429104 stopped in file sync0arr.c line 339 InnoDB: Thread 1373858736 stopped in file sync0arr.c line 339 InnoDB: Thread 1492675504 stopped in file sync0arr.c line 339 InnoDB: Thread 1692375984 stopped in file sync0arr.c line 339 InnoDB: Thread 1826044848 stopped in file sync0arr.c line 339 InnoDB: Thread 1517562800 stopped in file sync0arr.c line 339 InnoDB: Thread 1501907888 stopped in file sync0arr.c line 339 InnoDB: Thread 1575365552 stopped in file sync0arr.c line 339 InnoDB: Thread 1708030896 stopped in file sync0arr.c line 339 InnoDB: Thread 1430457264 stopped in file sync0arr.c line 339 InnoDB: Thread 1630759856 stopped in file sync0arr.c line 339 InnoDB: Thread 1636780976 stopped in file sync0arr.c line 339 InnoDB: Thread 1823234992 stopped in file sync0arr.c line 339 InnoDB: Thread 1367636912 stopped in file sync0arr.c line 339 InnoDB: Thread 1778478000 stopped in file sync0arr.c line 339 InnoDB: Thread 1442499504 stopped in file sync0arr.c line 339 InnoDB: Thread 1814404016 stopped in file sync0arr.c line 339 InnoDB: Thread 1838689200 stopped in file sync0arr.c line 339 InnoDB: Thread 1664478128 stopped in file sync0arr.c line 339 InnoDB: Thread 1797143472 stopped in file sync0arr.c line 339 InnoDB: Thread 1377471408 stopped in file sync0arr.c line 339 InnoDB: Thread 1818217392 stopped in file sync0arr.c line 339 InnoDB: Thread 1487858608 stopped in file sync0arr.c line 339 InnoDB: Thread 1725692848 stopped in file sync0arr.c line 339 InnoDB: Thread 1482238896 stopped in file sync0arr.c line 339 InnoDB: Thread 1655647152 stopped in file sync0arr.c line 339 InnoDB: Thread 1480833968 stopped in file sync0arr.c line 339 InnoDB: Thread 1698798512 stopped in file sync0arr.c line 339 InnoDB: Thread 1403964336 stopped in file sync0arr.c line 339 InnoDB: Thread 1816411056 stopped in file sync0arr.c line 339 InnoDB: Thread 1741347760 stopped in file sync0arr.c line 339 InnoDB: Thread 1826245552 stopped in file sync0arr.c line 339 InnoDB: Thread 1790520240 stopped in file sync0arr.c line 339 InnoDB: Thread 1505119152 stopped in file sync0arr.c line 339 InnoDB: Thread 1771252656 stopped in file sync0arr.c line 339 InnoDB: Thread 1368640432 stopped in file sync0arr.c line 339 InnoDB: Thread 1632967600 stopped in file sync0arr.c line 339 InnoDB: Thread 1689566128 stopped in file sync0arr.c line 339 InnoDB: Thread 1693580208 stopped in file sync0arr.c line 339 InnoDB: Thread 1804569520 stopped in file sync0arr.c line 339 InnoDB: Thread 1607879600 stopped in file sync0arr.c line 339 InnoDB: Thread 1702009776 stopped in file sync0arr.c line 339 InnoDB: Thread 1741147056 stopped in file sync0arr.c line 339 InnoDB: Thread 1690168240 stopped in file sync0arr.c line 339 InnoDB: Thread 1628351408 stopped in file sync0arr.c line 339 InnoDB: Thread 1484446640 stopped in file sync0arr.c line 339 InnoDB: Thread 1487055792 stopped in file sync0arr.c line 339 InnoDB: Thread 1634773936 stopped in file sync0arr.c line 339 InnoDB: Thread 1394731952 stopped in file sync0arr.c line 339 InnoDB: Thread 1387506608 stopped in file sync0arr.c line 339 InnoDB: Thread 1372453808 stopped in file sync0arr.c line 339 InnoDB: Thread 1625541552 stopped in file sync0arr.c line 339 InnoDB: Thread 1670097840 stopped in file sync0arr.c line 339 InnoDB: Thread 1709034416 stopped in file sync0arr.c line 339 InnoDB: Thread 1640795056 stopped in file sync0arr.c line 339 InnoDB: Thread 1417812912 stopped in file sync0arr.c line 339 InnoDB: Thread 1406172080 stopped in file sync0arr.c line 339 InnoDB: Thread 1415605168 stopped in file sync0arr.c line 339 InnoDB: Thread 1386503088 stopped in file sync0arr.c line 339 InnoDB: Thread 1682541488 stopped in file sync0arr.c line 339 InnoDB: Thread 1707228080 stopped in file sync0arr.c line 339 InnoDB: Thread 1663876016 stopped in file sync0arr.c line 339 InnoDB: Thread 1811192752 stopped in file sync0arr.c line 339 InnoDB: Thread 1479027632 stopped in file sync0arr.c line 339 InnoDB: Thread 1752587184 stopped in file sync0arr.c line 339 InnoDB: Thread 1775266736 stopped in file sync0arr.c line 339 InnoDB: Thread 1438686128 stopped in file sync0arr.c line 339 InnoDB: Thread 1569745840 stopped in file sync0arr.c line 339 InnoDB: Thread 1728502704 stopped in file sync0arr.c line 339 InnoDB: Thread 1665080240 stopped in file sync0arr.c line 339 InnoDB: Thread 1465179056 stopped in file sync0arr.c line 339 InnoDB: Thread 1701809072 stopped in file sync0arr.c line 339 InnoDB: Thread 1819020208 stopped in file sync0arr.c line 339 InnoDB: Thread 1441295280 stopped in file sync0arr.c line 339 InnoDB: Thread 1510136752 stopped in file sync0arr.c line 339 InnoDB: Thread 1686956976 stopped in file sync0arr.c line 339 InnoDB: Thread 1633369008 stopped in file sync0arr.c line 339 InnoDB: Thread 1617714096 stopped in file sync0arr.c line 339 InnoDB: Thread 1636580272 stopped in file sync0arr.c line 339 InnoDB: Thread 1627548592 stopped in file sync0arr.c line 339 InnoDB: Thread 1833872304 stopped in file sync0arr.c line 339 InnoDB: Thread 1660062640 stopped in file sync0arr.c line 339 InnoDB: Thread 1586002864 stopped in file sync0arr.c line 339 InnoDB: Thread 1380281264 stopped in file sync0arr.c line 339 InnoDB: Thread 1683344304 stopped in file sync0arr.c line 339 InnoDB: Thread 1490467760 stopped in file sync0arr.c line 339 InnoDB: Thread 1818418096 stopped in file sync0arr.c line 339 InnoDB: Thread 1635576752 stopped in file sync0arr.c line 339 InnoDB: Thread 1431460784 stopped in file sync0arr.c line 339 InnoDB: Thread 1426242480 stopped in file sync0arr.c line 339 InnoDB: Thread 1661066160 stopped in file sync0arr.c line 339 InnoDB: Thread 1438485424 stopped in file sync0arr.c line 339 InnoDB: Thread 1789516720 stopped in file sync0arr.c line 339 InnoDB: Thread 1804368816 stopped in file sync0arr.c line 339 InnoDB: Thread 1793731504 stopped in file sync0arr.c line 339 InnoDB: Thread 1820826544 stopped in file sync0arr.c line 339 InnoDB: Thread 1462168496 stopped in file sync0arr.c line 339 InnoDB: Thread 1627347888 stopped in file sync0arr.c line 339 InnoDB: Thread 1833470896 stopped in file sync0arr.c line 339 InnoDB: Thread 1402559408 stopped in file sync0arr.c line 339 InnoDB: Thread 1442700208 stopped in file sync0arr.c line 339 InnoDB: Thread 1371851696 stopped in file sync0arr.c line 339 InnoDB: Thread 1472404400 stopped in file sync0arr.c line 339 InnoDB: Thread 1579379632 stopped in file sync0arr.c line 339 InnoDB: Thread 1450527664 stopped in file sync0arr.c line 339 InnoDB: Thread 1768242096 stopped in file sync0arr.c line 339 InnoDB: Thread 1407376304 stopped in file sync0arr.c line 339 InnoDB: Thread 1367034800 stopped in file sync0arr.c line 339 InnoDB: Thread 1451330480 stopped in file sync0arr.c line 339 InnoDB: Thread 1407577008 stopped in file sync0arr.c line 339 InnoDB: Thread 1439288240 stopped in file sync0arr.c line 339 InnoDB: Thread 1777675184 stopped in file sync0arr.c line 339 InnoDB: Thread 1811794864 stopped in file sync0arr.c line 339 InnoDB: Thread 1803365296 stopped in file sync0arr.c line 339 InnoDB: Thread 1466182576 stopped in file sync0arr.c line 339 InnoDB: Thread 1471400880 stopped in file sync0arr.c line 339 InnoDB: Thread 1835076528 stopped in file sync0arr.c line 339 InnoDB: Thread 1368239024 stopped in file sync0arr.c line 339 InnoDB: Thread 1421626288 stopped in file sync0arr.c line 339 InnoDB: Thread 1391520688 stopped in file sync0arr.c line 339 InnoDB: Thread 1445108656 stopped in file sync0arr.c line 339 InnoDB: Thread 1445510064 stopped in file sync0arr.c line 339 InnoDB: Thread 1571752880 stopped in file sync0arr.c line 339 InnoDB: Thread 1827449776 stopped in file sync0arr.c line 339 InnoDB: Thread 1480231856 stopped in file sync0arr.c line 339 InnoDB: Thread 1488460720 stopped in file sync0arr.c line 339 InnoDB: Thread 1365228464 stopped in file sync0arr.c line 339 InnoDB: Thread 1388108720 stopped in file sync0arr.c line 339 InnoDB: Thread 1822432176 stopped in file sync0arr.c line 339 InnoDB: Thread 1816611760 stopped in file sync0arr.c line 339 InnoDB: Thread 1362619312 stopped in file sync0arr.c line 339 InnoDB: Thread 1518566320 stopped in file sync0arr.c line 339 InnoDB: Thread 1727900592 stopped in file sync0arr.c line 339 InnoDB: Thread 1790319536 stopped in file sync0arr.c line 339 InnoDB: Thread 1408580528 stopped in file sync0arr.c line 339 InnoDB: Thread 1704418224 stopped in file sync0arr.c line 339 InnoDB: Thread 1522178992 stopped in file sync0arr.c line 339 InnoDB: Thread 1547467696 stopped in file sync0arr.c line 339 InnoDB: Thread 1815808944 stopped in file sync0arr.c line 339 InnoDB: Thread 1487256496 stopped in file sync0arr.c line 339 InnoDB: Thread 1840696240 stopped in file sync0arr.c line 339 InnoDB: Thread 1506724784 stopped in file sync0arr.c line 339 InnoDB: Thread 1516559280 stopped in file sync0arr.c line 339 InnoDB: Thread 1691974576 stopped in file sync0arr.c line 339 InnoDB: Thread 1645611952 stopped in file sync0arr.c line 339 InnoDB: Thread 1363221424 stopped in file sync0arr.c line 339 InnoDB: Thread 1378073520 stopped in file sync0arr.c line 339 InnoDB: Thread 1602862000 stopped in file sync0arr.c line 339 InnoDB: Thread 1584798640 stopped in file sync0arr.c line 339 InnoDB: Thread 1558707120 stopped in file sync0arr.c line 339 InnoDB: Thread 1533217712 stopped in file sync0arr.c line 339 InnoDB: Thread 1637182384 stopped in file sync0arr.c line 339 InnoDB: Thread 1622330288 stopped in file sync0arr.c line 339 InnoDB: Thread 1586604976 stopped in file sync0arr.c line 339 InnoDB: Thread 1591823280 stopped in file sync0arr.c line 339 InnoDB: Thread 1590619056 stopped in file sync0arr.c line 339 InnoDB: Thread 1611492272 stopped in file sync0arr.c line 339 InnoDB: Thread 1625942960 stopped in file sync0arr.c line 339 InnoDB: Thread 1588009904 stopped in file sync0arr.c line 339 InnoDB: Thread 1588210608 stopped in file sync0arr.c line 339 InnoDB: Thread 1618918320 stopped in file sync0arr.c line 339 InnoDB: Thread 1624738736 stopped in file sync0arr.c line 339 InnoDB: Thread 1621527472 stopped in file sync0arr.c line 339 InnoDB: Thread 1617111984 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1786706864 stopped in file sync0arr.c line 339 InnoDB: Thread 1453538224 stopped in file sync0arr.c line 339 InnoDB: Thread 1370647472 stopped in file sync0arr.c line 339 InnoDB: Thread 1740143536 stopped in file sync0arr.c line 339 InnoDB: Thread 1781087152 stopped in file sync0arr.c line 339 InnoDB: Thread 1779882928 stopped in file sync0arr.c line 339 InnoDB: Thread 1687157680 stopped in file sync0arr.c line 339 InnoDB: Thread 1806977968 stopped in file sync0arr.c line 339 InnoDB: Thread 1711643568 stopped in file sync0arr.c line 339 InnoDB: Thread 1489263536 stopped in file sync0arr.c line 339 InnoDB: Thread 1746164656 stopped in file sync0arr.c line 339 InnoDB: Thread 1418816432 stopped in file sync0arr.c line 339 InnoDB: Thread 1455946672 stopped in file sync0arr.c line 339 InnoDB: Thread 1718266800 stopped in file sync0arr.c line 339 InnoDB: Thread 1749978032 stopped in file sync0arr.c line 339 InnoDB: Thread 1374661552 stopped in file sync0arr.c line 339 InnoDB: Thread 1605069744 stopped in file sync0arr.c line 339 InnoDB: Thread 1792527280 stopped in file sync0arr.c line 339 InnoDB: Thread 1692978096 stopped in file sync0arr.c line 339 InnoDB: Thread 1651633072 stopped in file sync0arr.c line 339 InnoDB: Thread 1634573232 stopped in file sync0arr.c line 339 InnoDB: Thread 1626946480 stopped in file sync0arr.c line 339 InnoDB: Thread 1322077104 stopped in file sync0arr.c line 339 InnoDB: Thread 1638988720 stopped in file sync0arr.c line 339 InnoDB: Thread 1568943024 stopped in file sync0arr.c line 339 InnoDB: Thread 1679932336 stopped in file sync0arr.c line 339 InnoDB: Thread 1676520368 stopped in file sync0arr.c line 339 InnoDB: Thread 1669295024 stopped in file sync0arr.c line 339 InnoDB: Thread 1825643440 stopped in file sync0arr.c line 339 InnoDB: Thread 1697594288 stopped in file sync0arr.c line 339 InnoDB: Thread 1703816112 stopped in file sync0arr.c line 339 InnoDB: Thread 1743154096 stopped in file sync0arr.c line 339 InnoDB: Thread 1752988592 stopped in file sync0arr.c line 339 InnoDB: Thread 1443101616 stopped in file sync0arr.c line 339 InnoDB: Thread 1707830192 stopped in file sync0arr.c line 339 InnoDB: Thread 1695787952 stopped in file sync0arr.c line 339 InnoDB: Thread 1685351344 stopped in file sync0arr.c line 339 InnoDB: Thread 1436277680 stopped in file sync0arr.c line 339 InnoDB: Thread 1456950192 stopped in file sync0arr.c line 339 InnoDB: Thread 1664277424 stopped in file sync0arr.c line 339 InnoDB: Thread 1418013616 stopped in file sync0arr.c line 339 InnoDB: Thread 1471601584 stopped in file sync0arr.c line 339 InnoDB: Thread 1685752752 stopped in file sync0arr.c line 339 InnoDB: Thread 1829657520 stopped in file sync0arr.c line 339 InnoDB: Thread 1668492208 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1689365424 stopped in file sync0arr.c line 339 InnoDB: Thread 1747168176 stopped in file sync0arr.c line 339 InnoDB: Thread 1664879536 stopped in file sync0arr.c line 339 InnoDB: Thread 1761618864 stopped in file sync0arr.c line 339 InnoDB: Thread 1686756272 stopped in file sync0arr.c line 339 InnoDB: Thread 1391922096 stopped in file sync0arr.c line 339 InnoDB: Thread 1695386544 stopped in file sync0arr.c line 339 InnoDB: Thread 1507929008 stopped in file sync0arr.c line 339 InnoDB: Thread 1683545008 stopped in file sync0arr.c line 339 InnoDB: Thread 1714252720 stopped in file sync0arr.c line 339 InnoDB: Thread 1813199792 stopped in file sync0arr.c line 339 InnoDB: Thread 1523182512 stopped in file sync0arr.c line 339 InnoDB: Thread 1390115760 stopped in file sync0arr.c line 339 InnoDB: Thread 1791323056 stopped in file sync0arr.c line 339 InnoDB: Thread 1435876272 stopped in file sync0arr.c line 339 InnoDB: Thread 1434069936 stopped in file sync0arr.c line 339 InnoDB: Thread 1549876144 stopped in file sync0arr.c line 339 InnoDB: Thread 1635777456 stopped in file sync0arr.c line 339 InnoDB: Thread 1716259760 stopped in file sync0arr.c line 339 InnoDB: Thread 1632365488 stopped in file sync0arr.c line 339 InnoDB: Thread 1390316464 stopped in file sync0arr.c line 339 InnoDB: Thread 1670900656 stopped in file sync0arr.c line 339 InnoDB: Thread 1679731632 stopped in file sync0arr.c line 339 InnoDB: Thread 1780886448 stopped in file sync0arr.c line 339 InnoDB: Thread 1570147248 stopped in file sync0arr.c line 339 InnoDB: Thread 1698597808 stopped in file sync0arr.c line 339 InnoDB: Thread 1661467568 stopped in file sync0arr.c line 339 InnoDB: Thread 1447316400 stopped in file sync0arr.c line 339 InnoDB: Thread 1394330544 stopped in file sync0arr.c line 339 InnoDB: Thread 1680534448 stopped in file sync0arr.c line 339 InnoDB: Thread 1675717552 stopped in file sync0arr.c line 339 InnoDB: Thread 1394129840 stopped in file sync0arr.c line 339 InnoDB: Thread 1631361968 stopped in file sync0arr.c line 339 InnoDB: Thread 1684147120 stopped in file sync0arr.c line 339 InnoDB: Thread 1750580144 stopped in file sync0arr.c line 339 InnoDB: Thread 1775467440 stopped in file sync0arr.c line 339 InnoDB: Thread 1643404208 stopped in file sync0arr.c line 339 InnoDB: Thread 1641196464 stopped in file sync0arr.c line 339 InnoDB: Thread 1461566384 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1560112048 stopped in file sync0arr.c line 339 InnoDB: Thread 1684949936 stopped in file sync0arr.c line 339 InnoDB: Thread 1816812464 stopped in file sync0arr.c line 339 InnoDB: Thread 1555094448 stopped in file sync0arr.c line 339 InnoDB: Thread 1661868976 stopped in file sync0arr.c line 339 InnoDB: Thread 1651432368 stopped in file sync0arr.c line 339 InnoDB: Thread 1820023728 stopped in file sync0arr.c line 339 InnoDB: Thread 1696390064 stopped in file sync0arr.c line 339 InnoDB: Thread 1548471216 stopped in file sync0arr.c line 339 InnoDB: Thread 1658256304 stopped in file sync0arr.c line 339 InnoDB: Thread 1707428784 stopped in file sync0arr.c line 339 InnoDB: Thread 1684548528 stopped in file sync0arr.c line 339 InnoDB: Thread 1444305840 stopped in file sync0arr.c line 339 InnoDB: Thread 1667087280 stopped in file sync0arr.c line 339 InnoDB: Thread 1794333616 stopped in file sync0arr.c line 339 InnoDB: Thread 1561918384 stopped in file sync0arr.c line 339 InnoDB: Thread 1693981616 stopped in file sync0arr.c line 339 InnoDB: Thread 1701006256 stopped in file sync0arr.c line 339 InnoDB: Thread 1693379504 stopped in file sync0arr.c line 339 InnoDB: Thread 1586404272 stopped in file sync0arr.c line 339 InnoDB: Thread 1696791472 stopped in file sync0arr.c line 339 InnoDB: Thread 1794735024 stopped in file sync0arr.c line 339 InnoDB: Thread 1423231920 stopped in file sync0arr.c line 339 InnoDB: Thread 1680735152 stopped in file sync0arr.c line 339 InnoDB: Thread 1429654448 stopped in file sync0arr.c line 339 InnoDB: Thread 1654242224 stopped in file sync0arr.c line 339 InnoDB: Thread 1410788272 stopped in file sync0arr.c line 339 InnoDB: Thread 1662872496 stopped in file sync0arr.c line 339 InnoDB: Thread 1645411248 stopped in file sync0arr.c line 339 InnoDB: Thread 1686154160 stopped in file sync0arr.c line 339 InnoDB: Thread 1675918256 stopped in file sync0arr.c line 339 InnoDB: Thread 1663474608 stopped in file sync0arr.c line 339 InnoDB: Thread 1481436080 stopped in file sync0arr.c line 339 InnoDB: Thread 1657453488 stopped in file sync0arr.c line 339 InnoDB: Thread 1706224560 stopped in file sync0arr.c line 339 InnoDB: Thread 1721277360 stopped in file sync0arr.c line 339 InnoDB: Thread 1794534320 stopped in file sync0arr.c line 339 InnoDB: Thread 1656048560 stopped in file sync0arr.c line 339 InnoDB: Thread 1425640368 stopped in file sync0arr.c line 339 InnoDB: Thread 1672506288 stopped in file sync0arr.c line 339 InnoDB: Thread 1748171696 stopped in file sync0arr.c line 339 InnoDB: Thread 1787308976 stopped in file sync0arr.c line 339 InnoDB: Thread 1485249456 stopped in file sync0arr.c line 339 InnoDB: Thread 1721879472 stopped in file sync0arr.c line 339 InnoDB: Thread 1659861936 stopped in file sync0arr.c line 339 InnoDB: Thread 1446112176 stopped in file sync0arr.c line 339 InnoDB: Thread 1712245680 stopped in file sync0arr.c line 339 InnoDB: Thread 1643002800 stopped in file sync0arr.c line 339 InnoDB: Thread 1667287984 stopped in file sync0arr.c line 339 InnoDB: Thread 1827851184 stopped in file sync0arr.c line 339 InnoDB: Thread 1722481584 stopped in file sync0arr.c line 339 InnoDB: Thread 1504717744 stopped in file sync0arr.c line 339 InnoDB: Thread 1708833712 stopped in file sync0arr.c line 339 InnoDB: Thread 1648221104 stopped in file sync0arr.c line 339 InnoDB: Thread 1699400624 stopped in file sync0arr.c line 339 InnoDB: Thread 1714654128 stopped in file sync0arr.c line 339 InnoDB: Thread 1712647088 stopped in file sync0arr.c line 339 InnoDB: Thread 1748974512 stopped in file sync0arr.c line 339 InnoDB: Thread 1694985136 stopped in file sync0arr.c line 339 InnoDB: Thread 1365830576 stopped in file sync0arr.c line 339 InnoDB: Thread 1720474544 stopped in file sync0arr.c line 339 InnoDB: Thread 1683143600 stopped in file sync0arr.c line 339 InnoDB: Thread 1662671792 stopped in file sync0arr.c line 339 InnoDB: Thread 1520171952 stopped in file sync0arr.c line 339 InnoDB: Thread 1392724912 stopped in file sync0arr.c line 339 InnoDB: Thread 1657654192 stopped in file sync0arr.c line 339 InnoDB: Thread 1628552112 stopped in file sync0arr.c line 339 InnoDB: Thread 1704016816 stopped in file sync0arr.c line 339 InnoDB: Thread 1668090800 stopped in file sync0arr.c line 339 InnoDB: Thread 1726495664 stopped in file sync0arr.c line 339 InnoDB: Thread 1791724464 stopped in file sync0arr.c line 339 InnoDB: Thread 1389513648 stopped in file sync0arr.c line 339 InnoDB: Thread 1638386608 stopped in file sync0arr.c line 339 InnoDB: Thread 1686555568 stopped in file sync0arr.c line 339 InnoDB: Thread 1639791536 stopped in file sync0arr.c line 339 InnoDB: Thread 1459358640 stopped in file sync0arr.c line 339 InnoDB: Thread 1743555504 stopped in file sync0arr.c line 339 InnoDB: Thread 1822231472 stopped in file sync0arr.c line 339 InnoDB: Thread 1448520624 stopped in file sync0arr.c line 339 InnoDB: Thread 1738939312 stopped in file sync0arr.c line 339 InnoDB: Thread 1695988656 stopped in file sync0arr.c line 339 InnoDB: Thread 1514953648 stopped in file sync0arr.c line 339 InnoDB: Thread 1689967536 stopped in file sync0arr.c line 339 InnoDB: Thread 1755597744 stopped in file sync0arr.c line 339 InnoDB: Thread 1760213936 stopped in file sync0arr.c line 339 InnoDB: Thread 1701407664 stopped in file sync0arr.c line 339 InnoDB: Thread 1492274096 stopped in file sync0arr.c line 339 InnoDB: Thread 1692576688 stopped in file sync0arr.c line 339 InnoDB: Thread 1704618928 stopped in file sync0arr.c line 339 InnoDB: Thread 1694383024 stopped in file sync0arr.c line 339 InnoDB: Thread 1730309040 stopped in file sync0arr.c line 339 InnoDB: Thread 1719671728 stopped in file sync0arr.c line 339 InnoDB: Thread 1704819632 stopped in file sync0arr.c line 339 InnoDB: Thread 1693178800 stopped in file sync0arr.c line 339 InnoDB: Thread 1471802288 stopped in file sync0arr.c line 339 InnoDB: Thread 1709235120 stopped in file sync0arr.c line 339 InnoDB: Thread 1493076912 stopped in file sync0arr.c line 339 InnoDB: Thread 1740946352 stopped in file sync0arr.c line 339 InnoDB: Thread 1833270192 stopped in file sync0arr.c line 339 InnoDB: Thread 1733319600 stopped in file sync0arr.c line 339 InnoDB: Thread 1669094320 stopped in file sync0arr.c line 339 InnoDB: Thread 1759812528 stopped in file sync0arr.c line 339 InnoDB: Thread 1755196336 stopped in file sync0arr.c line 339 InnoDB: Thread 1769245616 stopped in file sync0arr.c line 339 InnoDB: Thread 1437481904 stopped in file sync0arr.c line 339 InnoDB: Thread 1739541424 stopped in file sync0arr.c line 339 InnoDB: Thread 1430858672 stopped in file sync0arr.c line 339 InnoDB: Thread 1732516784 stopped in file sync0arr.c line 339 InnoDB: Thread 1773058992 stopped in file sync0arr.c line 339 InnoDB: Thread 1737534384 stopped in file sync0arr.c line 339 InnoDB: Thread 1463573424 stopped in file sync0arr.c line 339 InnoDB: Thread 1752787888 stopped in file sync0arr.c line 339 InnoDB: Thread 1750780848 stopped in file sync0arr.c line 339 InnoDB: Thread 1769044912 stopped in file sync0arr.c line 339 InnoDB: Thread 1792928688 stopped in file sync0arr.c line 339 InnoDB: Thread 1461767088 stopped in file sync0arr.c line 339 InnoDB: Thread 1605270448 stopped in file sync0arr.c line 339 InnoDB: Thread 1769647024 stopped in file sync0arr.c line 339 InnoDB: Thread 1821629360 stopped in file sync0arr.c line 339 InnoDB: Thread 1457351600 stopped in file sync0arr.c line 339 InnoDB: Thread 1755798448 stopped in file sync0arr.c line 339 InnoDB: Thread 1384094640 stopped in file sync0arr.c line 339 InnoDB: Thread 1762421680 stopped in file sync0arr.c line 339 InnoDB: Thread 1640594352 stopped in file sync0arr.c line 339 InnoDB: Thread 1753791408 stopped in file sync0arr.c line 339 InnoDB: Thread 1754594224 stopped in file sync0arr.c line 339 InnoDB: Thread 1736932272 stopped in file sync0arr.c line 339 InnoDB: Thread 1388510128 stopped in file sync0arr.c line 339 InnoDB: Thread 1738537904 stopped in file sync0arr.c line 339 InnoDB: Thread 1732115376 stopped in file sync0arr.c line 339 InnoDB: Thread 1761418160 stopped in file sync0arr.c line 339 InnoDB: Thread 1771854768 stopped in file sync0arr.c line 339 InnoDB: Thread 1754995632 stopped in file sync0arr.c line 339 InnoDB: Thread 1713249200 stopped in file sync0arr.c line 339 InnoDB: Thread 1709636528 stopped in file sync0arr.c line 339 InnoDB: Thread 1726897072 stopped in file sync0arr.c line 339 InnoDB: Thread 1724890032 stopped in file sync0arr.c line 339 InnoDB: Thread 1372654512 stopped in file sync0arr.c line 339 InnoDB: Thread 1702611888 stopped in file sync0arr.c line 339 InnoDB: Thread 1721478064 stopped in file sync0arr.c line 339 InnoDB: Thread 1715657648 stopped in file sync0arr.c line 339 InnoDB: Thread 1462369200 stopped in file sync0arr.c line 339 InnoDB: Thread 1715256240 stopped in file sync0arr.c line 339 InnoDB: Thread 1458555824 stopped in file sync0arr.c line 339 InnoDB: Thread 1711442864 stopped in file sync0arr.c line 339 InnoDB: Thread 1453738928 stopped in file sync0arr.c line 339 InnoDB: Thread 1437281200 stopped in file sync0arr.c line 339 InnoDB: Thread 1718467504 stopped in file sync0arr.c line 339 InnoDB: Thread 1711844272 stopped in file sync0arr.c line 339 InnoDB: Thread 1699802032 stopped in file sync0arr.c line 339 InnoDB: Thread 1695185840 stopped in file sync0arr.c line 339 InnoDB: Thread 1676118960 stopped in file sync0arr.c line 339 InnoDB: Thread 1723083696 stopped in file sync0arr.c line 339 InnoDB: Thread 1671302064 stopped in file sync0arr.c line 339 InnoDB: Thread 1701206960 stopped in file sync0arr.c line 339 InnoDB: Thread 1759210416 stopped in file sync0arr.c line 339 InnoDB: Thread 1695587248 stopped in file ../include/sync0sync.ic line 111 InnoDB: Thread 1498295216 stopped in file sync0arr.c line 339 InnoDB: Thread 1757604784 stopped in file sync0arr.c line 339 InnoDB: Thread 1617312688 stopped in file sync0arr.c line 339 InnoDB: Thread 1578175408 stopped in file sync0arr.c line 339 InnoDB: Thread 1724488624 stopped in file sync0arr.c line 339 InnoDB: Thread 1487457200 stopped in file sync0arr.c line 339 InnoDB: Thread 1509534640 stopped in file sync0arr.c line 339 InnoDB: Thread 1834474416 stopped in file sync0arr.c line 339 InnoDB: Thread 1485048752 stopped in file sync0arr.c line 339 InnoDB: Thread 1473608624 stopped in file sync0arr.c line 339 InnoDB: Thread 1838287792 stopped in file sync0arr.c line 339 InnoDB: Thread 1389714352 stopped in file sync0arr.c line 339 InnoDB: Thread 1684749232 stopped in file sync0arr.c line 339 InnoDB: Thread 1477823408 stopped in file sync0arr.c line 339 InnoDB: Thread 1836682160 stopped in file sync0arr.c line 339 InnoDB: Thread 1707629488 stopped in file sync0arr.c line 339 InnoDB: Thread 1477020592 stopped in file sync0arr.c line 339 InnoDB: Thread 1771654064 stopped in file sync0arr.c line 339 InnoDB: Thread 1833671600 stopped in file sync0arr.c line 339 InnoDB: Thread 1712847792 stopped in file sync0arr.c line 339 InnoDB: Thread 1562921904 stopped in file sync0arr.c line 339 InnoDB: Thread 1562119088 stopped in file sync0arr.c line 339 InnoDB: Thread 1773460400 stopped in file sync0arr.c line 339 InnoDB: Thread 1225513904 stopped in file sync0arr.c line 339 InnoDB: Thread 1836882864 stopped in file sync0arr.c line 339 InnoDB: Thread 1655847856 stopped in file os0sync.c line 501 InnoDB: Thread 1225780144 stopped in file ../include/sync0sync.ic line 111 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388600 read_buffer_size=98304 max_used_connections=3001 max_connections=10000 threads_connected=3001 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1944754 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. You seem to be running 32-bit Linux and have 3001 concurrent connections. If you have not changed STACK_SIZE in LinuxThreads and built the binary yourself, LinuxThreads is quite likely to steal a part of the global heap for the thread stack. Please read http://www.mysql.com/doc/en/Linux.html thd=0xfbe3e88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x66520dcc, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x81630bd 0xffffe420 0x82f6138 0x82f6138 0x82c7b8d 0x82970d6 0x8205b51 0x82057d5 0x81a35c5 0x81ad4ed 0x81afac9 0x81b2bea 0x81b6b17 0x817d63d 0x8181f3a 0x8183e02 0x81853aa 0x4004f9dd 0x4017b64a New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0xfbed8e8 = SELECT c from sbtest where id=3737751 thd->thread_id=2383 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash.
[6 Sep 2004 12:24]
Heikki Tuuri
Hi!
I found the bug. In MySQL-4.1, InnoDB initially has srv_pool_size in units of 1 kB, while in 4.0 it was in unit of one byte. I has forgotten to adjust the code below! The end result was that in 4.1 InnoDB only allowed a concurrency of 1000 threads at the maximum.
I changed this now so that if the buffer pool in Unix is > 8 MB, 10 000 threads are allowed, and if the buffer pool is > 1000 MB, then 50 000 threads are allowed.
The fix will be in 4.1.5. Also the other warning that you got, running out of undo slots for transactions, may be alleviated by this fix, since in fact, this makes innodb_thread_concurrency to work again (it overflowed at 1000 threads), and you have it set to 16. If you still get those undo log warnings, then we have to look at this again.
Thank you,
Heikki
if (srv_pool_size >= 8 * 1024 * 1024) {
/* Here we still have srv_pool_size counted
in bytes, srv_boot converts the value to
pages; if buffer pool is less than 8 MB,
assume fewer threads. */
srv_max_n_threads = 10000;
} else {
srv_max_n_threads = 1000; /* saves several MB of memory,
especially in 64-bit
computers */
}
[6 Sep 2004 16:17]
Donny Simonton
Heikki and Peter, I thank both of you. Donny
