Bug #76221 Assertion `1 == num_erased' failed in remove_global_thread | sql/mysqld.cc:874
Submitted: 9 Mar 2015 9:52 Modified: 16 Mar 2015 6:18
Reporter: Ramesh Sivaraman Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.6.23 OS:Any
Assigned to: CPU Architecture:Any
Tags: debug

[9 Mar 2015 9:52] Ramesh Sivaraman
Description:
#0  0x00007ff52c9d7771 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000a980a6 in my_write_core (sig=6) at /sda/mysql-server-5.6/mysys/stacktrace.c:422
#2  0x00000000007277f0 in handle_fatal_signal (sig=6) at /sda/mysql-server-5.6/sql/signal_handler.cc:230
#3  <signal handler called>
#4  0x00007ff52b7e35c9 in raise () from /lib64/libc.so.6
#5  0x00007ff52b7e4cd8 in abort () from /lib64/libc.so.6
#6  0x00007ff52b7dc536 in __assert_fail_base () from /lib64/libc.so.6
#7  0x00007ff52b7dc5e2 in __assert_fail () from /lib64/libc.so.6
#8  0x000000000061b261 in remove_global_thread (thd=0x7ff505c45020) at /sda/mysql-server-5.6/sql/mysqld.cc:874
#9  0x00000000007bd1b9 in Delayed_insert::~Delayed_insert (this=0x7ff505c45000, __in_chrg=<optimized out>) at /sda/mysql-server-5.6/sql/sql_insert.cc:2245
#10 0x00000000007bd368 in Delayed_insert::~Delayed_insert (this=0x7ff505c45000, __in_chrg=<optimized out>) at /sda/mysql-server-5.6/sql/sql_insert.cc:2255
#11 0x00000000007b616a in delayed_get_table (thd=0x7ff51bf16000, grl_protection_request=0x7ff515d27840, table_list=0x7ff505c1f100) at /sda/mysql-server-5.6/sql/sql_insert.cc:2424
#12 0x00000000007b22c5 in open_and_lock_for_insert_delayed (thd=0x7ff51bf16000, table_list=0x7ff505c1f100) at /sda/mysql-server-5.6/sql/sql_insert.cc:563
#13 0x00000000007b2749 in mysql_insert (thd=0x7ff51bf16000, table_list=0x7ff505c1f100, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /sda/mysql-server-5.6/sql/sql_insert.cc:731
#14 0x00000000007d5704 in mysql_execute_command (thd=0x7ff51bf16000) at /sda/mysql-server-5.6/sql/sql_parse.cc:3443
#15 0x00000000007dd54a in mysql_parse (thd=0x7ff51bf16000, rawbuf=0x7ff505c1f010 "INSERT DELAYED t1 VALUES (7)", length=28, parser_state=0x7ff515d28e70) at /sda/mysql-server-5.6/sql/sql_parse.cc:6357
#16 0x00000000007d06bc in dispatch_command (command=COM_QUERY, thd=0x7ff51bf16000, packet=0x7ff51bf1a001 "INSERT DELAYED t1 VALUES (7); ;", packet_length=31) at /sda/mysql-server-5.6/sql/sql_parse.cc:1332
#17 0x00000000007cf7ab in do_command (thd=0x7ff51bf16000) at /sda/mysql-server-5.6/sql/sql_parse.cc:1034
#18 0x0000000000797b3d in do_handle_one_connection (thd_arg=0x7ff51bf16000) at /sda/mysql-server-5.6/sql/sql_connect.cc:982
#19 0x0000000000797626 in handle_one_connection (arg=0x7ff51bf16000) at /sda/mysql-server-5.6/sql/sql_connect.cc:898
#20 0x0000000000b5d7f0 in pfs_spawn_thread (arg=0x7ff52633e740) at /sda/mysql-server-5.6/storage/perfschema/pfs.cc:1860
#21 0x00007ff52c9d2df3 in start_thread () from /lib64/libpthread.so.0
#22 0x00007ff52b8a41ad in clone () from /lib64/libc.so.6

How to repeat:
DROP DATABASE test;CREATE DATABASE test;USE test;
CREATE TABLE t1 AS SELECT TIMESTAMP('2000-01-01 00:00:00.123');
INSERT DELAYED t1 VALUES (7); ;

To reproduce the issue we need to run multi thread pquery binary. The attached tarball gives the testcase as an exact match of our system, including some handy utilities
$ vi {epoch}_mybase # Update base path in this file (the only change
required!)
$ ./{epoch}_init # Initializes the data dir
$ ./{epoch}_start # Starts mysqld
$ ./{epoch}_cl # To check mysqld is up
$ ./{epoch}_run_pquery # Run the testcase with pquery binary(produces
output)
$ vi /dev/shm/{epoch}/error.log.out # Verify the error log
$ ./{epoch}_gdb # Brings you to a gdb prompt attached to correct
mysqld
& generated core
$ ./{epoch}_parse_core # Create {epoch}_STD.gdb and {epoch}_FULL.gdb;
standard and full var gdb stack traces
etc.
[9 Mar 2015 9:54] Ramesh Sivaraman
Testcase bundle

Attachment: 1425600530_bug_bundle.tar.gz (application/gzip, text), 144.38 KiB.

[9 Mar 2015 9:54] Ramesh Sivaraman
added debug tag
[16 Mar 2015 5:53] MySQL Verification Team
Hello Ramesh,

Thank you for the report.
I cannot repeat the issue with 5.6.23/5.6.25 opt/debug builds.
Moreover, provided reproducible test case *run_pquery Segfaults most of the times.

(few samples collected)

[umshastr@hod03]/export/umesh/mysql-5.6.25: ./1425600530_run_pquery
Fri Mar 13 13:39:05 CET 2015
Executing testcase ./1425600530.sql against mysqld with socket /tmp/mysql_ushastry.sock using pquery...
Infile is ./1425600530.sql
Database is test
Starting with 30 threads
Query limit per thread is 400000
User is root
Socket is /tmp/mysql_ushastry.sock
MySQL Connection Info: Localhost via UNIX socket
MySQL Client Info: 5.6.25
MySQL Server Info: 5.6.25-enterprise-commercial-advanced
* Last 250 consecutive queries all failed. Likely crash/assert, user privileges drop, or similar. Ending run.
./1425600530_run_pquery: line 6: 27588 Segmentation fault      (core dumped) ./1425600530_pquery --infile=./1425600530.sql --database=test --threads=30 --queries=400000 --user=root --socket=/tmp/mysql_ushastry.sock
Fri Mar 13 13:40:31 CET 2015

[umshastr@hod03]/export/umesh/mysql-5.6.25: ./1425600530_run_pquery
Fri Mar 13 13:42:23 CET 2015
Executing testcase ./1425600530.sql against mysqld with socket /tmp/mysql_ushastry.sock using pquery...
Infile is ./1425600530.sql
Database is test
Starting with 30 threads
Query limit per thread is 400000
User is root
Socket is /tmp/mysql_ushastry.sock
MySQL Connection Info: Localhost via UNIX socket
MySQL Client Info: 5.6.25
MySQL Server Info: 5.6.25-enterprise-commercial-advanced-debug
* Last 250 consecutive queries all failed. Likely crash/assert, user privileges drop, or similar. Ending run.
./1425600530_run_pquery: line 6: 12238 Segmentation fault      (core dumped) ./1425600530_pquery --infile=./1425600530.sql --database=test --threads=30 --queries=400000 --user=root --socket=/tmp/mysql_ushastry.sock
Fri Mar 13 13:59:29 CET 2015

Please feel free to reopen the bug with exact reproducible test case.

Thanks,
Umesh
[16 Mar 2015 6:18] Ramesh Sivaraman
Could you please check the error log. pquery segfaults error is due to crash.
[17 Mar 2015 10:23] MySQL Verification Team
No, Ramesh. mysqld didn't assert - confirmed it from error log.