Bug #79500 InnoDB: Assertion failure in thread 140721784878848 in file rem0rec.ic

Description:
InnoDB server crashes with Assertion failure in thread 140721784878848 in file rem0rec.ic while running concurrent memcached testcses with DML. This is a sporadic issue , it does not happen very frequently.this happens on trunk.
it might due to some concurrent issue with update from both SQL side and Memcached side.

Core Dump:
----------
2015-12-03T05:42:15.160644Z 8 [Note] Aborted connection 8 to db: 'mysql' user: 'jettestuser' host: 'localhost' (Got an error reading communication packets)
2015-12-03 06:42:24 0x7ffc57fff700  InnoDB: Assertion failure in thread 140721784878848 in file rem0rec.ic line 616
InnoDB: Failing assertion: rec_info_bits_valid(val)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
#0  0x00007ffdd8bb5771 in pthread_kill () from /lib64/libpthread.so.0
#1  0x000000000177f354 in my_write_core (sig=6) at /home/vigudipa/GIT-MySQL/mysql/mysys/stacktrace.c:251
#2  0x0000000000fd9fe3 in handle_fatal_signal (sig=6) at /home/vigudipa/GIT-MySQL/mysql/sql/signal_handler.cc:221
#3  <signal handler called>
#4  0x00007ffdd75af5d7 in raise () from /lib64/libc.so.6
#5  0x00007ffdd75b0cc8 in abort () from /lib64/libc.so.6
#6  0x0000000001ace57f in ut_dbg_assertion_failed (expr=0x20bd4f5 "rec_info_bits_valid(val)", 
    file=0x20bd480 "/home/vigudipa/GIT-MySQL/mysql/storage/innobase/include/rem0rec.ic", line=616)
    at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/ut/ut0dbg.cc:67
#7  0x0000000001ad58d9 in rec_get_info_bits (rec=0x7ffd613913b5 "", comp=1)
    at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/include/rem0rec.ic:616
#8  0x0000000001ad6a18 in ib_read_tuple (rec=0x7ffd613913b5 "", page_format=1, tuple=0x7ffc64006d40, cmp_tuple=0x0, mode=0, 
    rec_buf=0x7ffc640c7a80, len=0x7ffc640c7a88, used_len=0x7ffc640c7a90) at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/api/api0api.cc:314
#9  0x0000000001ad90ab in ib_cursor_read_row (ib_crsr=0x7ffc64016050, ib_tpl=0x7ffc64006d40, cmp_tpl=0x0, mode=0, row_buf=0x7ffc640c7a80, 
    row_len=0x7ffc640c7a88, used_len=0x7ffc640c7a90) at /home/vigudipa/GIT-MySQL/mysql/storage/innobase/api/api0api.cc:1800
#10 0x00007ffd041eaac1 in innodb_api_search (cursor_data=0x7ffc640c7a30, crsr=0x7ffc57ffe240, key=0x7ffc64004b28 "F18Florida append", len=3, 
    item=0x7ffc57ffe250, r_tpl=0x7ffc57ffe248, sel_only=0 '\000', range_key=0x0)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/innodb_memcache/src/innodb_api.c:832
#11 0x00007ffd041ec866 in innodb_api_store (engine=0x7ffcdc071680, cursor_data=0x7ffc640c7a30, key=0x7ffc64004b28 "F18Florida append", len=3, 
    val_len=14, exp=0, cas=0x7ffc640470e0, input_cas=0, flags=32, op=OPERATION_APPEND)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/innodb_memcache/src/innodb_api.c:1776
#12 0x00007ffd041e8c34 in innodb_store (handle=0x7ffcdc071680, cookie=0x7ffc64046e90, item=0x7ffc64004af0, cas=0x7ffc640470e0, 
    op=OPERATION_APPEND, vbucket=0) at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/innodb_memcache/src/innodb_engine.c:2293
#13 0x00007ffd1c011edf in complete_update_ascii (c=0x7ffc64046e90)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:1115
#14 0x00007ffd1c019834 in complete_nread_ascii (c=0x7ffc64046e90)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:3511
#15 0x00007ffd1c0198c0 in complete_nread (c=0x7ffc64046e90)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:3521
#16 0x00007ffd1c01ffea in conn_nread (c=0x7ffc64046e90)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:5409
#17 0x00007ffd1c021213 in event_handler (fd=199, which=2, arg=0x7ffc64046e90)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/memcached.c:5715
#18 0x00007ffd1c029c4a in event_process_active (base=0x7ffcdc264600) at /home/vigudipa/GIT-MySQL/mysql/libevent/event.c:392
#19 0x00007ffd1c029fd8 in event_base_loop (base=0x7ffcdc264600, flags=0) at /home/vigudipa/GIT-MySQL/mysql/libevent/event.c:544
#20 0x00007ffd1c0262e5 in worker_libevent (arg=0x7ffcdc089538)
    at /home/vigudipa/GIT-MySQL/mysql/plugin/innodb_memcached/daemon_memcached/daemon/thread.c:306
#21 0x00007ffdd8bb0df5 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---

How to repeat:
1) pull the JET testcases  from GIT
2) copy the following testcase and 2 jar files and xml files

commons-pool-1.5.6.jar (copy this in extlib location) 
java_memcached-release_2.6.6.jar (copy this in extlib location) 
TestMemcacheGetRange.java ( copy this file in mysql-test-extra/mysql-test/jet/src/com/sun/mysql/jet/connectorj/v5/testcases/concurrency)
TestMemcacheGetRange.xml (mysql-test-extra/mysql-test/jet/src/com/sun/mysql/jet/tests/concurrency

* The above xml should have the following config
    <TestSetup
    class="com.sun.mysql.jet.setups.SiteSetup"
    comment="Set up MySQL with default configuration">
        <Binding key="mysqld.cmdline.max_connections" value="50000" />
        <Binding key="mysqld.cmdline.innodb_buffer_pool_size" value="2024M"
          comment="Set innodb_buffer_pool_size to 2024M" />
        <Binding key="mysqld.cmdline.innodb_buffer_pool_instances" value="40"
          comment="Set innodb_buffer_pool_instances to 40" />
       <Binding key="mysqld.cmdline.innodb_api_trx_level" value="1"
          comment="Set innodb_api_trx_level to 1" />
        <TestSuite>
            <TestCase
                class="com.sun.mysql.jet.connectorj.v5.testcases.concurrency.TestMemcacheGetRange"
                method="testConcurrency"
                comment="Concurrency tests for Server Version Token" >
                <Binding key="jet.versiontoken.concurrentusers" value="10" />
                <Binding key="jet.versiontoken.concurrentduration" value="60" />
            </TestCase>
            <TestCase
                class="com.sun.mysql.jet.connectorj.v5.testcases.concurrency.TestMemcacheGetRange"
                method="testConcurrency"
                comment="Concurrency tests for Server Version Token" >
                <Binding key="jet.versiontoken.concurrentusers" value="20" />
                <Binding key="jet.versiontoken.concurrentduration" value="60" />
            </TestCase>
        </TestSuite>
    </TestSetup>

3) Build.xml should have the following entry with rest to new jar files

 <!-- External Jars -->
  <property name="memcachedclientjar" value="${extlib}/java_memcached-release_2.6.6.jar"/>
  <property name="  <!-- External Jars -->" value="${extlib}/commons-pool-1.5.6.jar"/>

  <property name="classpath_build"           value="${classpath_prefix}${build}:${jetjar}:${jagopsjar}:${jetloadjar}:${sshjar}:${jsonjar}:${gcsjar}:${jetbatchjar}:${memcachedclientjar}:${sockiopooljar}"/>
  <property name="classpath_runjet"          value="${classpath_prefix}${jetjar}:${jagopsjar}:${jetloadjar}:${sshjar}:${jsonjar}:${jagjar}:${mysqltestsjar}:${protobufjar}:${mysqlconnectorjar}:${memcachedclientjar}:${sockiopooljar}"/>
  <property name="classpath_run"             value="${classpath_prefix}${jetbatchjar}:${jagjar}:${jagopsjar}:${mysqltestsjar}:${memcachedclientjar}:${sockiopooljar}"/>

Run command :
3) ant clean all
4) ant runjet -Dxmlfile=/home/vigudipa/GIT-MySQL/mysql-test-extra/mysql-test/jet/src/com/sun/mysql/jet/tests/concurrency/TestMemcacheGetRange.xml -Dpropfile=properties/memcache.property

Note : Core file and java files has been attached with this bug 

Suggested fix:
No crash

Posted by developer:
 
The core file could not attach as it is 3GB after zip/tar , it is available with reported person

Posted by developer:
 
Fixed as of the upcoming 5.8.0  release, and here's the changelog entry:

Reallocation of memcached-referenced memory raised an assertion.

Moderated - already published in changelogs, making public.

Posted by developer:
 
Reported version value updated to reflect release name change from 5.8 to 8.0