Bug #90869 Event scheduler (cant create worker thread) MySQL service restarted with bug.
Submitted: 15 May 2018 12:15 Modified: 17 May 2018 6:45
Reporter: Chaitanya Tondlekar Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: General Severity:S2 (Serious)
Version:5.7.19 OS:Linux (AWS RDS)
Assigned to: CPU Architecture:Other
Tags: MysQL Event Scheduler, Worker Thread Dies

[15 May 2018 12:15] Chaitanya Tondlekar
Description:
Hello,

I am running MySQL 5.7.19 on AWS RDS.

All were running fine, suddenly below error populated in MySQL error log

Error Details:

2018-05-15T08:04:45.600933-00:00 1 [ERROR] Event_scheduler::execute_top: Can not create event worker thread (errno=11). Stopping event scheduler

When we observed that event scheduler is off, then we tried to start the event scheduler it was giving following error

Error Details:

2018-05-15T10:11:14.778152-00:00 2243941 [ERROR] Event scheduler: Failed to start scheduler, Can not create thread for event scheduler (errno=11)

After some time, event scheduler thread started.

But after few minutes, MySQL service restarted with bug:

10:44:56 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=161
max_threads=10230
thread_count=71
connection_count=69
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 85512250 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2ad6078cb000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2ac3e1390e40 thread_stack 0x40000
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0xebdbfc]
/rdsdbbin/mysql/bin/mysqld(handle_fatal_signal+0x451)[0x7b0d71]
/lib64/libpthread.so.0(+0xf5b0)[0x2ac25efa15b0]
/rdsdbbin/mysql/bin/mysqld(_ZN13MYSQL_BIN_LOG6commitEP3THDb+0x700)[0xe5f0d0]
/rdsdbbin/mysql/bin/mysqld(_Z15ha_commit_transP3THDbb+0x1a9)[0x7fbc89]
/rdsdbbin/mysql/bin/mysqld(_Z17trans_commit_stmtP3THD+0x2e)[0xd3f9ee]
/rdsdbbin/mysql/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x3bd2)[0xc9c432]
/rdsdbbin/mysql/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3bd)[0xc9f4ed]
/rdsdbbin/mysql/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xa95)[0xc9fff5]
/rdsdbbin/mysql/bin/mysqld(_Z10do_commandP3THD+0x177)[0xca1967]
/rdsdbbin/mysql/bin/mysqld(handle_connection+0x270)[0xd5bc90]
/rdsdbbin/mysql/bin/mysqld(pfs_spawn_thread+0x18f)[0x121e55f]
/lib64/libpthread.so.0(+0x7f18)[0x2ac25ef99f18]
/lib64/libc.so.6(clone+0x6d)[0x2ac2604f4b2d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2ad60788e030): SELECT value FROM mysql.rds_heartbeat2
Connection ID (thread ID): 2243941
Status: NOT_KILLED
.
.
.
.
.

Can you help me on this?

How to repeat:
Didn't find any steps to repeat the error. 

Suggested fix:
If any thread dies, it should get recreated and MySQL service should not get impacted.
[15 May 2018 12:22] Miguel Solorzano
Thank you for the bug report. Please try latest released version and provide a repeatable test case too.
[17 May 2018 6:45] Chaitanya Tondlekar
This is a sudden behavior. We didn't find out the root cause why did the worker thread dies resulting into stopping event scheduler.

So how can we provide repeatable test case for the same?