Bug #52360 Innodb assertion wait_lock in lock/lock0lock.c on UPDATE

• Submitted: 25 Mar 2010 12:38
• Modified: 19 Jun 2010 0:19
• Reporter: Victor Kirkebo
• Status: Closed
• Category: MySQL Server: InnoDB Plugin storage engine
• Severity: S3 (Non-critical)
• Version: 5.1
• OS: Any
• Assigned to: Marko Mäkelä
• CPU Architecture: Any

[25 Mar 2010 12:38] Victor Kirkebo

Description:
When running a concurrent UPDATE workload on an Innodb table against 5.1.45 or earlier, innodb asserts. I have only seen this on Solaris x86 - e.g. with server mysql-advanced-5.1.45-solaris10-x86_64. It also seems necessary to set the parameter innodb-thread-concurrency to something different than the default value in order for this to happen.

100325 15:25:28 InnoDB: Assertion failure in thread 33 in file lock/lock0lock.c line 3342
InnoDB: Failing assertion: wait_lock
InnoDB: We intentionally generate a memory trap.

/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:my_print_stacktrace+0x20
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:handle_segfault+0x1dc
/lib/amd64/libc.so.1:0xd5386
/lib/amd64/libc.so.1:0xc9c82
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x3f08cd [ Signal 11 (SEGV)]
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x3f0a50
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x3f078e
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x3e9770
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x3ea601
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x3ea877
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x3f5569
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:0x5e4843
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:__1cJha_innodbKindex_read6MpCpkCInQha_rkey_function__i_+0x1e7
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/lib/plugin/ha_innodb_plugin.so.0.0.0:__1cJha_innodbLindex_first6MpC_i_+0x38
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cHhandlerQread_range_first6MpknMst_key_range_3bb_i_+0xed
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cHhandlerWread_multi_range_first6MppnSst_key_multi_range_2IbpnRst_handler_buffer__i_+0xe3
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cSQdDUICK_RANGE_SELECTIget_next6M_i_+0x3e6
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:0x414d5e
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cMmysql_update6FpnDTHD_pnKTABLE_LIST_rnEList4nEItem___5pn0C_IpnIst_order_XnPenum_duplicates_b_i_+0x1451
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cVmysql_execute_command6FpnDTHD__i_+0x50ef
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cLmysql_parse6FpnDTHD_pkcIp3_v_+0x130
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_+0x998
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:__1cKdo_command6FpnDTHD__b_+0xe2
/export/home/tmp/mysql-advanced-5.1.45-solaris10-x86_64/bin/mysqld:handle_one_connection+0xe1
/lib/amd64/libc.so.1:0xd504b
/lib/amd64/libc.so.1:0xd5280

How to repeat:
ZZ file:

$tables = {
        partitions => [ undef ]
};

$fields = {
        types => [ 'int'],
        indexes => [undef],
};

$data = {
        numbers => [ 'digit', 'null', undef ],
}

YY file:

query:
        UPDATE _table SET _field_no_pk = _digit WHERE _field_key < _digit LIMIT 3 ;

RQG command line:

$ perl runall.pl \
  --engine=Innodb \
  --reporters=ErrorLog,Backtrace \
  --rows=10 \
  --queries=100000000 \
  --duration=600 \
  --basedir=/path/to/mysql-advanced-5.1.45-solaris10-x86_64 \
  --vardir=/path/to/vardir \
  --gendata=/path/to/file.zz \
  --grammar=/path/to/file.yy \
  --mysqld=--ignore-builtin-innodb \
  --mysqld=--plugin-load=innodb=ha_innodb_plugin.so \
  --mysqld=--innodb-thread-concurrency=20 \
  --threads=20 \
  --mysqld=--innodb_lock_wait_timeout=40

The Random Query Generator can be obtained from
http://launchpad.net/randgen/2.0/2.1.4/+download/randgen-2.1.4.tgz

[26 Mar 2010 18:57] Sveta Smirnova

Thank you for the report.

Verified as described.

[6 Apr 2010 7:58] Bugs System

Pushed into 5.1.46 (revid:[e-mail]-20100405111026-7kz1p8qlzglqgfmu) (version source revid:[e-mail]-20100401151005-c6re90vdvutln15d) (merge vers: 5.1.46) (pib:16)

[7 Apr 2010 6:30] Marko Mäkelä

This bug cannot be repeated with the current InnoDB Plugin source tree (r6900, upcoming version 1.0.7). I was unable to repeat this with the previous storage/innodb_plugin snapshot (version 1.0.6) on a different system. Version 1.0.6 corresponds to InnoDB svn revision 6210, plus c6348 (removing the print_arbitrary_thd functions, merging the Bug #38883 fix from MySQL) and some changes to the build and test files.

If this bug is in the InnoDB Plugin, it should have been fixed in -r6210:6900. The main suspects I could find are these:

r6526: Fix Bug #47622 by adding an index translation table
r6790: Fix Bug #51356 (Valgrind warnings for too short string allocation in diagnostic messages)

Especially r6526 could affect the timing so much that this bug goes away. This bug can still exist; the root cause of the crashes with the old InnoDB Plugin tree was not found.

[8 Apr 2010 10:34] Marko Mäkelä

This bug was fixed in r6713 of the InnoDB Plugin source tree, which will be part of the upcoming InnoDB Plugin 1.0.7 release:

branches/zip: Change the bit fields back to ulint. Bit fields were causing
problems with concurrency on SMP systems because of word packing issues.
The number of trx_t's in a system is not sufficient enough to require that
we try and save a few bytes in the data structure.

A wild guess is that r6713 could also have fixed Bug #52626 (failing assertion !lock->recursive). The InnoDB transaction object is closely involved with locking.

[8 Apr 2010 17:33] Paul DuBois

Noted in 5.1.46 changelog (that's the version where InnoDB Plugin 1.0.7 first appears).

For InnoDB Plugin, bit fields were causing problems with concurrency
on SMP systems because of word-packing issues.

[5 May 2010 15:24] Bugs System

Pushed into 5.1.47 (revid:[e-mail]-20100505145753-ivlt4hclbrjy8eye) (version source revid:[e-mail]-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

[6 May 2010 17:05] Paul DuBois

Push resulted from incorporation of InnoDB tree. No changes pertinent to this bug.
Re-closing.

[28 May 2010 5:58] Bugs System

Pushed into mysql-next-mr (revid:[e-mail]-20100524190136-egaq7e8zgkwb9aqi) (version source revid:[e-mail]-20100331130613-8ja7n0vh36a80457) (pib:16)

[28 May 2010 6:27] Bugs System

Pushed into 6.0.14-alpha (revid:[e-mail]-20100524190941-nuudpx60if25wsvx) (version source revid:[e-mail]-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

[28 May 2010 6:55] Bugs System

Pushed into 5.5.5-m3 (revid:[e-mail]-20100524185725-c8k5q7v60i5nix3t) (version source revid:[e-mail]-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

[30 May 2010 0:18] Paul DuBois

Push resulted from incorporation of InnoDB tree. No changes pertinent to this bug.
Re-closing.

[15 Jun 2010 8:21] Bugs System

Pushed into 5.5.5-m3 (revid:[e-mail]-20100615080459-smuswd9ooeywcxuc) (version source revid:[e-mail]-20100415070122-1nxji8ym4mao13ao) (merge vers: 5.1.47) (pib:16)

[15 Jun 2010 8:39] Bugs System

Pushed into mysql-next-mr (revid:[e-mail]-20100615080558-cw01bzdqr1bdmmec) (version source revid:[e-mail]-20100415070122-1nxji8ym4mao13ao) (pib:16)

[17 Jun 2010 12:01] Bugs System

Pushed into 5.1.47-ndb-7.0.16 (revid:[e-mail]-20100617114014-bva0dy24yyd67697) (version source revid:[e-mail]-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

[17 Jun 2010 12:41] Bugs System

Pushed into 5.1.47-ndb-6.2.19 (revid:[e-mail]-20100617115448-idrbic6gbki37h1c) (version source revid:[e-mail]-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

[17 Jun 2010 13:28] Bugs System

Pushed into 5.1.47-ndb-6.3.35 (revid:[e-mail]-20100617114611-61aqbb52j752y116) (version source revid:[e-mail]-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)