MySQL Bugs: #88555: Trigger causing MySQL crash?

Bug #88555	Trigger causing MySQL crash?
Submitted:	20 Nov 2017 11:06	Modified:	6 Jan 2018 14:10
Reporter:	Martyn Munday	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.7.20	OS:	Ubuntu (16 LTS)
Assigned to:		CPU Architecture:	Any

Description:
We've been seeing random crashes on one of our MySQL servers (5.7.20 on Ubuntu 16), but it does not seem to happen consistently and I've been able to reproduce myself. 
 
I believe from the back trace that it might be being caused by something in a trigger (fill_record_n_invoke_before_triggers). Is there anyway of confirming that, and ideally identifying the trigger being invoked / table, to debug further? I tried looking for the core dump file, but the only file I could find was blank.

Backtrace : 

10:11:57 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=67108864
read_buffer_size=8388608
max_used_connections=215
max_threads=214
thread_count=6
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2263575 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f01c00e9420
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f027046ee70 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xe8a93b]
/usr/sbin/mysqld(handle_fatal_signal+0x489)[0x786749]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f091cf4c390]
/usr/sbin/mysqld(_ZN14Arg_comparator15compare_decimalEv+0xad)[0x80ca4d]
/usr/sbin/mysqld(_ZN12Item_func_eq7val_intEv+0x2b)[0x7ff88b]
/usr/sbin/mysqld(_ZN4Item8val_boolEv+0xbc)[0x7e372c]
/usr/sbin/mysqld(_ZN12Item_func_if11val_decimalEP10my_decimal+0x26)[0x7ffe46]
/usr/sbin/mysqld(_ZN4Item19save_in_field_innerEP5Fieldb+0x268)[0x7e46f8]
/usr/sbin/mysqld(_ZN4Item13save_in_fieldEP5Fieldb+0x15)[0x7f45b5]
/usr/sbin/mysqld(_Z29update_generated_write_fieldsPK9st_bitmapP5TABLE+0x8f)[0xcf163f]
/usr/sbin/mysqld(_Z11fill_recordP3THDP5TABLER4ListI4ItemES6_P9st_bitmapS8_+0x162)[0xbf50e2]
/usr/sbin/mysqld(_Z36fill_record_n_invoke_before_triggersP3THDP9COPY_INFOR4ListI4ItemES6_P5TABLE23enum_trigger_event_typei+0x299)[0xbf5429]
/usr/sbin/mysqld(_ZN14Sql_cmd_insert12mysql_insertEP3THDP10TABLE_LIST+0x8d8)[0xdcb0f8]
/usr/sbin/mysqld(_ZN14Sql_cmd_insert7executeEP3THD+0xe2)[0xdcb8e2]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x24ac)[0xc4a96c]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3f5)[0xc4f295]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1093)[0xc50393]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1c7)[0xc51967]
/usr/sbin/mysqld(handle_connection+0x288)[0xd11df8]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xefc9c4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f091cf426ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f091c3d73dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f01c0150f80): is an invalid pointer
Connection ID (thread ID): 1815500
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

How to repeat:
Unfortunately, unsure on specific reproduction steps.

fill_record_n_invoke_before_triggers can be invoked by a normal DML operation. it doesn't mean trigger is involved.

Hi!

This can be a trigger that causes a crash, which is most likely in this case.

However, we must be able to reproduce this crash consistently on every run. In order to verify a bug we need from you a fully repeatable test case that will always lead to crash. 

We can not do much without it.

Hi!

In order to find a trigger that leads to crash, look for BEFORE INSERT triggers.

this is a crash on a table with generated column(s).

But still if there is  a trigger, then see:
https://bugs.mysql.com/bug.php?id=86637

Hi Mr. Munday,

It would be nice if you could discover the trigger that crashes our server, as in that case, we might have a candidate for the cause of the bug.

Thank you in advance.

unfortunately, our ubuntu binaries are stripped so we don't have chance to get proper line numbers for stack trace:

anon@anon-VirtualBox:~$ addr2line --demangle --pretty-print --exe=/usr/sbin/mysqld --basenames --functions --inlines --addresses 0xe8a93b 0x786749 0x80ca4d 0x7ff88b 0x7e372c 0x7ffe46 0x7e46f8 0x7f45b5 0xcf163f 0xbf50e2 0xbf5429 0xdcb0f8 0xdcb8e2 0xc4a96c 0xc4f295 0xc50393 0xc51967 0xd11df8 0xefc9c4
0x0000000000e8a93b: my_print_stacktrace at ??:?
0x0000000000786749: handle_fatal_signal at ??:?
0x000000000080ca4d: Arg_comparator::compare_decimal() at ??:?
0x00000000007ff88b: Item_func_eq::val_int() at ??:?
0x00000000007e372c: Item::val_bool() at ??:?
0x00000000007ffe46: Item_func_if::val_decimal(my_decimal*) at ??:?
0x00000000007e46f8: Item::save_in_field_inner(Field*, bool) at ??:?
0x00000000007f45b5: Item::save_in_field(Field*, bool) at ??:?
0x0000000000cf163f: update_generated_write_fields(st_bitmap const*, TABLE*) at ??:?
0x0000000000bf50e2: fill_record(THD*, TABLE*, List<Item>&, List<Item>&, st_bitmap*, st_bitmap*) at ??:?
0x0000000000bf5429: fill_record_n_invoke_before_triggers(THD*, COPY_INFO*, List<Item>&, List<Item>&, TABLE*, enum_trigger_event_type, int) at ??:?
0x0000000000dcb0f8: Sql_cmd_insert::mysql_insert(THD*, TABLE_LIST*) at ??:?
0x0000000000dcb8e2: Sql_cmd_insert::execute(THD*) at ??:?
0x0000000000c4a96c: mysql_execute_command(THD*, bool) at ??:?
0x0000000000c4f295: mysql_parse(THD*, Parser_state*) at ??:?
0x0000000000c50393: dispatch_command(THD*, COM_DATA const*, enum_server_command) at ??:?
0x0000000000c51967: do_command(THD*) at ??:?
0x0000000000d11df8: handle_connection at ??:?
0x0000000000efc9c4: pfs_spawn_thread at ??:?
anon@anon-VirtualBox:~$ file /usr/sbin/mysqld
/usr/sbin/mysqld: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=8f8dadec9d0670d05e7f4eab4ea6cab6e4f26660, stripped

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I have a similar problem.

Centos 6 (2.6.32-642.6.2.el6.x86_64）
MySQL 5.7.19

00:43:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=209715200
read_buffer_size=1048576
max_used_connections=10
max_threads=3000
thread_count=15
connection_count=4
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 27892831 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2abc4c0be8b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
 
stack_bottom = 2abc4803fe28 thread_stack 0x40000
/apps/svr/mysql57/bin/mysqld(my_print_stacktrace+0x35)[0xf45e05]
/apps/svr/mysql57/bin/mysqld(handle_fatal_signal+0x4a4)[0x7cd464]
/lib64/libpthread.so.0[0x382b00f7e0]
/lib64/libc.so.6(memcpy+0x3a1)[0x382ac89a51]
/apps/svr/mysql57/bin/mysqld(strmake_root+0x39)[0xf3e7f9]
/apps/svr/mysql57/bin/mysqld(_ZN17List_process_listclEP3THD+0x345)[0xd71db5]
/apps/svr/mysql57/bin/mysqld(_ZN18Global_THD_manager19do_for_all_thd_copyEP11Do_THD_Impl+0x25d)[0x7cbf0d]
/apps/svr/mysql57/bin/mysqld(_Z21mysqld_list_processesP3THDPKcb+0xfc8)[0xd69ef8]
/apps/svr/mysql57/bin/mysqld(_Z21mysql_execute_commandP3THDb+0xb29)[0xd13949]
/apps/svr/mysql57/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3a5)[0xd18245]
/apps/svr/mysql57/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x11af)[0xd1945f]
/apps/svr/mysql57/bin/mysqld(_Z10do_commandP3THD+0x194)[0xd1a324]
/apps/svr/mysql57/bin/mysqld(handle_connection+0x29c)[0xdea0fc]
/apps/svr/mysql57/bin/mysqld(pfs_spawn_thread+0x174)[0xfbdbf4]
/lib64/libpthread.so.0[0x382b007aa1]
/lib64/libc.so.6(clone+0x6d)[0x382ace8bcd]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2abc4c12e8a0): is an invalid pointer
Connection ID (thread ID): 12408003
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.