Description:
Using a Python cmd-line client for reading binlog off a AWS RDS MySQL cluster. The tool allows specifying a log file & log pos as the starting point. I tried to read from position 154 and it worked. When I tried again I entered 153 for position and it immediately crashed the whole cluster with signal 6
The tool issues a COM_BINLOG_DUMP command for initiating the dump.
As a side not, mysql diagnoses the issue as running out of memory. For reference, the log file size is 128MB . As said above, the system works without any problems and without any noticeable impact on memory consumption as long as a correct position is specified.
How to repeat:
Issue a COM_BINLOG_DUMP command with valid parameters except for an invalid (one-off) position >4 as the binlog-pos parameter against a MySQL 5.7.12 server cluster on t2.mini with binlog set to ROW format (haven't tried any of the other formats): or other environments.
Relevant log entries (more available on request):
2020-02-07T21:21:49.953181Z 4 [Note] Start binlog_dump to master_thread_id(4) slave_server(101), pos(mysql-bin-changelog.000076, 153)
<jemalloc>: Error in mmap(): err: 12, msg: Cannot allocate memory
<jemalloc>: Error in malloc(): out of memory
<jemalloc>: System-wide: MemTotal: 2051524kb, MemFree: 111324kb, Buffers: 70492kb, Cached: 181440kb, Active: 978080kb, Dirty: 4724kb, Inactive: 122932kb, Mapped: 79800kb
<jemalloc>: terminating process due to out of resources
13:21:50
21:21:50 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the in
21:21:50 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=3
max_threads=45
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 40147 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2ab0815d5000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
