| Bug #55143 | Failing assertion: copied_len < local_len + extern_len | ||
|---|---|---|---|
| Submitted: | 10 Jul 2010 0:14 | Modified: | 25 Apr 2012 13:26 |
| Reporter: | Vince Cox | Email Updates: | |
| Status: | No Feedback | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S1 (Critical) |
| Version: | 5.0.72,5.1.40 | OS: | Linux (CentOS 5.4 64 bit) |
| Assigned to: | CPU Architecture: | Any | |
[10 Jul 2010 0:14]
Vince Cox
[10 Jul 2010 0:17]
Vince Cox
Initial breakdown of core dump using gdb
Attachment: linux_gdb_from core_dump.txt (text/plain), 3.82 KiB.
[10 Jul 2010 0:18]
Vince Cox
Backtrace from Initial Core Dump Text File
Attachment: Linux_gdb_core_dump_full_backtrace.txt (text/plain), 29.70 KiB.
[10 Jul 2010 0:20]
Vince Cox
Mysql error file after crash
Attachment: Mysql_server_log_file_after_crash.txt (text/plain), 74.68 KiB.
[10 Jul 2010 6:01]
Valeriy Kravchuk
Can this be related to some hardware failure? Please, check OS logs. We had a couple of similar bug reports in the past, but nobody ever came up with a repeatable test case.
[11 Jul 2010 15:58]
Vince Cox
Valeriy, This issue has been produced on various hardware platforms running linux. It's been on physical hosts, vm's as well as blade servers in a chassis. And I've already cleared the hardware platform it's running on now. We also ran memtest, it ran for several hours and came back clean. -Vince
[12 Jul 2010 6:20]
Sveta Smirnova
Thank you for the feedback. Could you please send us dump of problem table of output of SHOW CREATE TABLE and SHOW TABLE STATUS for problem table? Please send us your configuration file also.
[12 Jul 2010 19:36]
Vince Cox
Show create table and show table status output
Attachment: show_table _and _show_create_table.txt (text/plain), 54.33 KiB.
[12 Jul 2010 19:37]
Vince Cox
New attachment added. The output from the 2 commands are in the textfile.
[13 Jul 2010 7:27]
Sveta Smirnova
Thank you for the feedback. I can not repeat described behavior. > We had 2 server crashes,one on 7/2/2010 and one on 7/6/2010. Since then the Master server had the problem listed below and the slave had to be promoted. Probably table is corrupted, although InnoDB should fix this at startup anyway. What CHECK TABLE Applications prints? > This issue has been produced on various hardware platforms running linux. How do you produce this issue? Can you create repeatable test case?
[13 Jul 2010 15:54]
Vince Cox
I can cause this failure be running either select * from Application; or by running mysqloptimize against the table. I will get you that other data that you requested.
[13 Jul 2010 17:24]
Vince Cox
mysql> use prodCEMIC Database changed Here is the output of the check table command that you requested. -Vince mysql> check table Application; +-----------------------+-------+----------+----------+ | Table | Op | Msg_type | Msg_text | +-----------------------+-------+----------+----------+ | prodCEMIC.Application | check | status | OK | +-----------------------+-------+----------+----------+ 1 row in set (0.41 sec) mysql>
[13 Jul 2010 19:38]
Sveta Smirnova
Thank you for the feedback. > I can cause this failure be running either select * from Application; > or > by running mysqloptimize against the table. Problem is I can not repeat crash with test data. So something different should be in your setup if you are able to recreate the problem on different machines. How do you move table to another machine? Do you use mysqldump or moving binary files? Please also try current version 5.1.48: probably problem solved already by fix of another bug.
[14 Jul 2010 0:39]
Vince Cox
We move data between the systems using mysqldump. I can use a working table (Applications)from another server. Drop table on broken server. Add table and import data and it works. If I then restore Broken DB from backup it 's in the back in the broken state. We are trying to figure out what keeps causing this. That's the reason I've sent all of the stack/backtraces. Thanks. -Vince
[21 Jul 2010 19:05]
Vince Cox
Are you able to get any information from the backtraces that were sent?
[21 Jul 2010 20:25]
MySQL Verification Team
Vince, mysqldump/import is not enough to be sure all corruption is gone. You need to start with a totally new innodb tablespace (ibdata* and ib_log*). This means mysqldump/reload all innodb tables... The error log contained one "*** glibc detected" which usually means some memory was corrupted elsewhere, which might be a sign of the problem. I wrote a better random data generator that does insert/select/update/delete on this table, but it still doesn't repeat a problem here.. Can you upload a my.cnf?
[21 Jul 2010 20:28]
MySQL Verification Team
my testcase to try repeat it. only run on a test server (it uses real table name!)
Attachment: bug55143.php (application/octet-stream, text), 7.54 KiB.
[30 Jul 2010 21:10]
Vince Cox
Below is the my.cnf file that you requested. # The following options will be passed to all MySQL clients [client] port = 3306 socket = /var/lib/mysql/mysql.sock user = root # The MySQL server [mysqld] port = 3306 basedir = /opt/mysql datadir = /var/lib/mysql tmpdir = /var/tmp socket = /var/lib/mysql/mysql.sock user = mysql skip-federated back_log = 50 max_connections = 500 max_connect_errors = 10 table_cache = 2048 max_allowed_packet = 32M binlog_cache_size = 1M max_heap_table_size = 64M sort_buffer_size = 16M join_buffer_size = 16M thread_cache_size = 8 # Try number of CPU's*2 for thread_concurrency thread_concurrency = 8 query_cache_size = 64M query_cache_limit = 2M ft_min_word_len = 4 # See SF case #8103 # memlock default_table_type = INNODB thread_stack = 192K transaction_isolation = READ-COMMITTED tmp_table_size = 128M # Enable the full query log. Every query (even ones # with incorrect syntax) that the server receives will # be logged. This is useful for debugging, it is # usually disabled in production use. ## log log_warnings slow_query_log slow_query_log_file = mysql-slow.log long-query-time = 5 log_long_format log-error = mysql.err pid-file = mysql.pid # Replication Setup ######################## log-bin = mysql-bin log-bin-index = mysql-bin.index binlog_format = mixed expire_logs_days = 14 # server-id needs to be unique on each replication server server-id = 3 # relay-log statements apply only to slave servers relay-log = mysql-relay-bin relay-log-index = mysql-relay-index relay-log-info-file = mysql-relay-log.info # only SUPER and # read-only slave_load_tmpdir = /tmp tmpdir = /tmp innodb_additional_mem_pool_size = 4M innodb_buffer_pool_size = 4G innodb_file_per_table = 1 # Where to put the ibdata (default: datadir) # innodb_data_home_dir # How to create the ibdata innodb_data_file_path = "ibdata1:100M:autoextend" # ibdata growth step innodb_autoextend_increment = 100M innodb_thread_concurrency = 8 innodb_flush_log_at_trx_commit = 1 innodb_flush_method = O_DIRECT # Global buffer for transaction# assembly before commit innodb_log_buffer_size = 4M # Size of one ib_logfile innodb_log_file_size = 256M # Number of ib_logfile? innodb_log_files_in_group = 3 # Percentage of buffer pool pages that # may be dirty before forcing a checkpoint innodb_max_dirty_pages_pct = 90 innodb_lock_wait_timeout = 120 ## Added for Innovation kw. interactive_timeout = 432000 wait_timeout = 432000 [mysqldump] quick max_allowed_packet = 32M [mysql] no-auto-rehash [mysqlhotcopy] interactive-timeout [mysqld_safe] open-files-limit = 8192
[5 Aug 2010 18:50]
Vince Cox
We have not had any crashes recently, but we really would like to know what the cause if this is. Is has occurred in several environments, and they all stop abruptly. And we end up promoting the slave. We can't keep going on like this. Any advice on what we need to do at this point? Thanks. -Vince
[9 Aug 2010 20:57]
Sveta Smirnova
Thank you for the feedback. I still can not repeat the problem using Shane's test case and options provided. Please check your RAM and if it is OK please try current version 5.1.49: we continuously fix bugs and this can be fixed already.
[9 Sep 2010 23:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[8 Oct 2010 15:32]
MySQL Verification Team
i got this with random blob testing on 5.0.72. 101008 17:25:54InnoDB: Assertion failure in thread 1804 in file .\btr\btr0cur.c line 3788 InnoDB: Failing assertion: copied_len < local_len + extern_len ntdll.dll!RtlUnicodeToMultiByteSize() mysqld-debug.exe!_free_base()[free.c:103] mysqld-debug.exe!_free_dbg_lk()[dbgheap.c:1207] mysqld-debug.exe!_free_dbg()[dbgheap.c:1070] mysqld-debug.exe!free()[dbgheap.c:1025] mysqld-debug.exe!ut_free()[ut0mem.c:236] mysqld-debug.exe!mem_area_free()[mem0pool.c:464] mysqld-debug.exe!mem_heap_block_free()[mem0mem.c:317] mysqld-debug.exe!mem_heap_free_func()[mem0mem.ic:472] mysqld-debug.exe!row_vers_impl_x_locked_off_kernel()[row0vers.c:249] mysqld-debug.exe!lock_sec_rec_some_has_impl_off_kernel()[lock0lock.c:1696] mysqld-debug.exe!lock_rec_convert_impl_to_expl()[lock0lock.c:4892] mysqld-debug.exe!lock_sec_rec_read_check_and_lock()[lock0lock.c:5073] mysqld-debug.exe!sel_set_rec_lock()[row0sel.c:785] mysqld-debug.exe!row_search_for_mysql()[row0sel.c:3784] mysqld-debug.exe!ha_innobase::index_read()[ha_innodb.cc:3879] mysqld-debug.exe!ha_innobase::index_first()[ha_innodb.cc:4130] mysqld-debug.exe!handler::read_range_first()[handler.cc:2664] mysqld-debug.exe!handler::read_multi_range_first()[handler.cc:2542] mysqld-debug.exe!QUICK_RANGE_SELECT::get_next()[opt_range.cc:6991] mysqld-debug.exe!rr_quick()[records.cc:261] mysqld-debug.exe!mysql_update()[sql_update.cc:460] mysqld-debug.exe!mysql_execute_command()[sql_parse.cc:3589] mysqld-debug.exe!mysql_parse()[sql_parse.cc:6266] mysqld-debug.exe!dispatch_command()[sql_parse.cc:1938] mysqld-debug.exe!do_command()[sql_parse.cc:1628] mysqld-debug.exe!handle_one_connection()[sql_parse.cc:1234] mysqld-debug.exe!pthread_start()[my_winthread.c:85] mysqld-debug.exe!_threadstart()[thread.c:196] kernel32.dll!FlsSetValue()
[10 May 2011 13:03]
Karsten Brusch
Hello Everyone, we are facing a similar problem with a DB which restarts every ~20 minutes while under load. There is one massive LONGBLOB which seems to cause the crash. It stores Slavic text mainly. May 10 06:42:22 <hostname> mysqld[7916]: 110510 6:42:22InnoDB: Assertion failure in thread 729971632 in file btr0cur.c line 3781 May 10 06:42:22 <hostname> mysqld[7916]: InnoDB: Failing assertion: copied_len == local_len + extern_len When running a select on the table the DB crashes and restarts. Unfortunately I cannot send the content of that dataset because the server even crashes when performing mysqldump and the ibdata is >20GB. "Check table" returns OK for me as well. CREATE TABLE `tablename` ( `id` int(11) NOT NULL, `column2` varchar(250) character set latin1 collate latin1_bin default NULL, `column3` longblob, `column4` int(11) default NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 We found that truncating the table made the error go away once, but it came back later (after App filled it with new data). Affected version is: 5.0.32 on "Linux 2.6.18-6-686-bigmem i686 GNU/Linux" We're going to run some tests with 5.5.11 and see if it shows up again. I'll keep you posted. Vince, were there any useful findings in your setup? Cheers K.
[25 Mar 2012 13:26]
Valeriy Kravchuk
I wonder if this assertion still happens to somebody on recent server versions (5.1.62 or 5.5.22).
[26 Apr 2012 1:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
