Description:
InnoDB server crashes with Assertion failure in thread 140721784878848 in file rem0rec.ic while running concurrent memcached testcses with DML. This is a sporadic issue , it does not happen very frequently.this happens on trunk.
it might due to some concurrent issue with update from both SQL side and Memcached side.
Core Dump:
----------
2015-12-03T05:42:15.160644Z 8 [Note] Aborted connection 8 to db: 'mysql' user: 'jettestuser' host: 'localhost' (Got an error reading communication packets)
2015-12-03 06:42:24 0x7ffc57fff700 InnoDB: Assertion failure in thread 140721784878848 in file rem0rec.ic line 616
InnoDB: Failing assertion: rec_info_bits_valid(val)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
#0 0x00007ffdd8bb5771 in pthread_kill () from /lib64/libpthread.so.0
#1 0x000000000177f354 in my_write_core (sig=6) at /home/vigudipa/GIT-MySQL/mysql/mysys/stacktrace.c:251
#2 0x0000000000fd9fe3 in handle_fatal_signal (sig=6) at /home/vigudipa/GIT-MySQL/mysql/sql/signal_handler.cc:221
#3 <signal handler called>
#4 0x00007ffdd75af5d7 in raise () from /lib64/libc.so.6
#5 0x00007ffdd75b0cc8 in abort () from /lib64/libc.so.6
#6 0x0000000001ace57f in ut_dbg_assertion_failed (expr=0x20bd4f5 "rec_info_bits_valid(val)",
file=0x20bd480 "/home/vigudipa/GIT-MySQL/mysql/storage/innobase/include/rem0rec.ic", line=616)
at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/ut/ut0dbg.cc:67
#7 0x0000000001ad58d9 in rec_get_info_bits (rec=0x7ffd613913b5 "", comp=1)
at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/include/rem0rec.ic:616
#8 0x0000000001ad6a18 in ib_read_tuple (rec=0x7ffd613913b5 "", page_format=1, tuple=0x7ffc64006d40, cmp_tuple=0x0, mode=0,
rec_buf=0x7ffc640c7a80, len=0x7ffc640c7a88, used_len=0x7ffc640c7a90) at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/api/api0api.cc:314
#9 0x0000000001ad90ab in ib_cursor_read_row (ib_crsr=0x7ffc64016050, ib_tpl=0x7ffc64006d40, cmp_tpl=0x0, mode=0, row_buf=0x7ffc640c7a80,
row_len=0x7ffc640c7a88, used_len=0x7ffc640c7a90) at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/api/api0api.cc:1800
#10 0x00007ffd041eaac1 in innodb_api_search (cursor_data=0x7ffc640c7a30, crsr=0x7ffc57ffe240, key=0x7ffc64004b28 "F18Florida append", len=3,
item=0x7ffc57ffe250, r_tpl=0x7ffc57ffe248, sel_only=0 '\000', range_key=0x0)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/innodb_memcache/src/innodb_api.c:832
#11 0x00007ffd041ec866 in innodb_api_store (engine=0x7ffcdc071680, cursor_data=0x7ffc640c7a30, key=0x7ffc64004b28 "F18Florida append", len=3,
val_len=14, exp=0, cas=0x7ffc640470e0, input_cas=0, flags=32, op=OPERATION_APPEND)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/innodb_memcache/src/innodb_api.c:1776
#12 0x00007ffd041e8c34 in innodb_store (handle=0x7ffcdc071680, cookie=0x7ffc64046e90, item=0x7ffc64004af0, cas=0x7ffc640470e0,
op=OPERATION_APPEND, vbucket=0) at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/innodb_memcache/src/innodb_engine.c:2293
#13 0x00007ffd1c011edf in complete_update_ascii (c=0x7ffc64046e90)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:1115
#14 0x00007ffd1c019834 in complete_nread_ascii (c=0x7ffc64046e90)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:3511
#15 0x00007ffd1c0198c0 in complete_nread (c=0x7ffc64046e90)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:3521
#16 0x00007ffd1c01ffea in conn_nread (c=0x7ffc64046e90)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:5409
#17 0x00007ffd1c021213 in event_handler (fd=199, which=2, arg=0x7ffc64046e90)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:5715
#18 0x00007ffd1c029c4a in event_process_active (base=0x7ffcdc264600) at /home/vigudipa/GIT-MySQL/mysql/libevent/event.c:392
#19 0x00007ffd1c029fd8 in event_base_loop (base=0x7ffcdc264600, flags=0) at /home/vigudipa/GIT-MySQL/mysql/libevent/event.c:544
#20 0x00007ffd1c0262e5 in worker_libevent (arg=0x7ffcdc089538)
at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/thread.c:306
#21 0x00007ffdd8bb0df5 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
How to repeat:
1) pull the JET testcases from GIT
2) copy the following testcase and 2 jar files and xml files
commons-pool-1.5.6.jar (copy this in extlib location)
java_memcached-release_2.6.6.jar (copy this in extlib location)
TestMemcacheGetRange.java ( copy this file in mysql-test-extra/mysql-test/jet/src/com/sun/mysql/jet/connectorj/v5/testcases/concurrency)
TestMemcacheGetRange.xml (mysql-test-extra/mysql-test/jet/src/com/sun/mysql/jet/tests/concurrency
* The above xml should have the following config
<TestSetup
class="com.sun.mysql.jet.setups.SiteSetup"
comment="Set up MySQL with default configuration">
<Binding key="mysqld.cmdline.max_connections" value="50000" />
<Binding key="mysqld.cmdline.innodb_buffer_pool_size" value="2024M"
comment="Set innodb_buffer_pool_size to 2024M" />
<Binding key="mysqld.cmdline.innodb_buffer_pool_instances" value="40"
comment="Set innodb_buffer_pool_instances to 40" />
<Binding key="mysqld.cmdline.innodb_api_trx_level" value="1"
comment="Set innodb_api_trx_level to 1" />
<TestSuite>
<TestCase
class="com.sun.mysql.jet.connectorj.v5.testcases.concurrency.TestMemcacheGetRange"
method="testConcurrency"
comment="Concurrency tests for Server Version Token" >
<Binding key="jet.versiontoken.concurrentusers" value="10" />
<Binding key="jet.versiontoken.concurrentduration" value="60" />
</TestCase>
<TestCase
class="com.sun.mysql.jet.connectorj.v5.testcases.concurrency.TestMemcacheGetRange"
method="testConcurrency"
comment="Concurrency tests for Server Version Token" >
<Binding key="jet.versiontoken.concurrentusers" value="20" />
<Binding key="jet.versiontoken.concurrentduration" value="60" />
</TestCase>
</TestSuite>
</TestSetup>
3) Build.xml should have the following entry with rest to new jar files
<!-- External Jars -->
<property name="memcachedclientjar" value="${extlib}/java_memcached-release_2.6.6.jar"/>
<property name=" <!-- External Jars -->" value="${extlib}/commons-pool-1.5.6.jar"/>
<property name="classpath_build" value="${classpath_prefix}${build}:${jetjar}:${jagopsjar}:${jetloadjar}:${sshjar}:${jsonjar}:${gcsjar}:${jetbatchjar}:${memcachedclientjar}:${sockiopooljar}"/>
<property name="classpath_runjet" value="${classpath_prefix}${jetjar}:${jagopsjar}:${jetloadjar}:${sshjar}:${jsonjar}:${jagjar}:${mysqltestsjar}:${protobufjar}:${mysqlconnectorjar}:${memcachedclientjar}:${sockiopooljar}"/>
<property name="classpath_run" value="${classpath_prefix}${jetbatchjar}:${jagjar}:${jagopsjar}:${mysqltestsjar}:${memcachedclientjar}:${sockiopooljar}"/>
Run command :
3) ant clean all
4) ant runjet -Dxmlfile=/home/vigudipa/GIT-MySQL/mysql-test-extra/mysql-test/jet/src/com/sun/mysql/jet/tests/concurrency/TestMemcacheGetRange.xml -Dpropfile=properties/memcache.property
Note : Core file and java files has been attached with this bug
Suggested fix:
No crash
