Bug #80273 MySQL server crashes when used with memcached plugin under severe load
Submitted: 5 Feb 2016 4:59 Modified: 26 Nov 2018 13:01
Reporter: Sergey Skachkov Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.7.10 OS:Linux
Assigned to: CPU Architecture:Any

[5 Feb 2016 4:59] Sergey Skachkov
Description:
MySQL crashes pretty often when using memcached plugin under severe load (about 20 concurrent connections constantly writing and reading entries with payload around 65k size), we use MEDIUMTEXT column type for payload, if it matters
here is the stack trace:

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/opt/mysql/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0xe7324c]
/opt/mysql/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x451)[0x77e1c1]
/lib64/libpthread.so.0(+0xf100)[0x7fe3583ee100]
/lib64/libc.so.6(gsignal+0x37)[0x7fe3571f25f7]
/lib64/libc.so.6(abort+0x148)[0x7fe3571f3ce8]
/opt/mysql/usr/local/mysql/bin/mysqld[0x74efc8]
/opt/mysql/usr/local/mysql/bin/mysqld(_Z20rec_get_offsets_funcPKhPK12dict_index_tPmmPP16mem_block_info_t+0x54)[0xfa76b4]
/opt/mysql/usr/local/mysql/bin/mysqld[0x1080a01]
/opt/mysql/usr/local/mysql/bin/mysqld(_Z18ib_cursor_read_rowP11ib_cursor_tP10ib_tuple_tPPvPm+0x33e)[0x108655e]
/opt/mysql/usr/local/mysql/lib/plugin//innodb_engine.so(innodb_api_search+0x274)[0x7fe0042fbf84]
/opt/mysql/usr/local/mysql/lib/plugin//innodb_engine.so(innodb_api_store+0x75)[0x7fe0042fdc95]
/opt/mysql/usr/local/mysql/lib/plugin//innodb_engine.so(+0xb4eb)[0x7fe0042f94eb]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(conn_nread+0x1355)[0x7fe00451e875]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(+0x9de9)[0x7fe004512de9]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(event_base_loop+0x592)[0x7fe004525892]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(+0x18982)[0x7fe004521982]
/lib64/libpthread.so.0(+0x7dc5)[0x7fe3583e6dc5]
/lib64/libc.so.6(clone+0x6d)[0x7fe3572b3bdd]

the same thing happens with version 5.6.28, stack trace is slightly different:

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/opt/mysql/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x8eb515]
/opt/mysql/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x41b)[0x653ffb]
/lib64/libpthread.so.0(+0xf100)[0x7fb871e35100]
/lib64/libc.so.6(gsignal+0x37)[0x7fb870c3a5f7]
/lib64/libc.so.6(abort+0x148)[0x7fb870c3bce8]
/opt/mysql/usr/local/mysql/bin/mysqld[0x9b61cd]
/opt/mysql/usr/local/mysql/bin/mysqld[0xa254d1]
/opt/mysql/usr/local/mysql/bin/mysqld[0xa25927]
/opt/mysql/usr/local/mysql/lib/plugin//innodb_engine.so(innodb_api_search+0x18e)[0x7fb1b5ba053e]
/opt/mysql/usr/local/mysql/lib/plugin//innodb_engine.so(innodb_api_store+0x70)[0x7fb1b5ba0b60]
/opt/mysql/usr/local/mysql/lib/plugin//innodb_engine.so(+0xa09f)[0x7fb1b5b9d09f]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(+0x13cba)[0x7fb852e29cba]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(conn_nread+0x211)[0x7fb852e2a6b1]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(+0x95bc)[0x7fb852e1f5bc]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(event_base_loop+0x286)[0x7fb852e31aa6]
/opt/mysql/usr/local/mysql/lib/plugin/libmemcached.so(+0x19582)[0x7fb852e2f582]
/lib64/libpthread.so.0(+0x7dc5)[0x7fb871e2ddc5]
/lib64/libc.so.6(clone+0x6d)[0x7fb870cfbbdd]

How to repeat:
create table with mediumtext column as a payload, map it to memcached plugin, connect with about 20 memcached clients and start performing reads and writes
[23 Oct 2018 13:19] MySQL Verification Team
Hi,

You are using a very old release of 5.7.

First you have to try and see whether you can repeat the crash with our 5.7.24 production binary.

Next, you have to send the exact configuration, queries to run, number of threads and all details necessary to repeat the crash with 5.7.24.
[24 Nov 2018 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".