Bug #25776 Replication binlog dump crashing mysql
Submitted: 23 Jan 2007 11:21 Modified: 15 Feb 2009 12:07
Reporter: Stéphane CANEL Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Replication Severity:S1 (Critical)
Version:4.1.22 OS:Linux (RHEL 4 update 4)
Assigned to: Assigned Account CPU Architecture:Any
Tags: Replication binlog dump crashing (master)

[23 Jan 2007 11:21] Stéphane CANEL
Description:
"binlog dump" causes the crash of mysql after approximately one minute or at the time of operations modifying the bases on the master during the sending of data towards the slave, mysql restarded automatically. 

OS: RHEL4 update 4(x86_64)
Memory : 4Go
2 CPU: intel xeon dual core 1,60 

result of resolve_stackd_dump :
0x746f6f72 __stop___libc_freeres_ptrs + 1943527618
0x617571696c706572 __stop___libc_freeres_ptrs + 1809372866

myc.cnf:
[mysqld]
socket=/tmp/mysql.sock
user=mysql
skip-innodb
language=/usr/local/mysql/share/mysql/french
set-variable=key_buffer_size=256M
set-variable=table_cache=1024
set-variable=join_buffer_size=4M
set-variable=read_buffer_size=4M
set-variable=sort_buffer_size=4M
set-variable=thread_stack=262144
set-variable = ft_min_word_len=3
set-variable = tmp_table_size=64M
# replication maitre
log-bin
server-id = 2

[mysqld_safe]
socket=/tmp/mysql.sock
err-log=/var/log/mysql/mysqld.log
pid-file=/var/mysql/mysqld.pid

extract of file mysqld.log:
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=4190208
max_used_connections=2
max_connections=100
threads_connected=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1212015 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9e9540
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x2ae021242d28, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x617571696c706572
New value of fp=0x9e9540 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=2
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
070110 04:22:10  mysqld restarted
/usr/local/mysql/libexec/mysqld: Prêt pour des connections  Source distribution
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=4190208
max_used_connections=2
max_connections=100
threads_connected=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1212015 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9e9540
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x2ac7343d5d28, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x617571696c706572
New value of fp=0x9e9540 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=2
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
070110 04:26:14  mysqld restarted

How to repeat:
"binlog dump" causes the crash of mysql after approximately one minute or at the time of operations modifying the bases on the master during the sending of data towards the slave, mysql restarded automatically. 

OS: RHEL4 update 4(x86_64)
Memory : 4Go
2 CPU: intel xeon dual core 1,60

result of resolve_stackd_dump :
0x746f6f72 __stop___libc_freeres_ptrs + 1943527618
0x617571696c706572 __stop___libc_freeres_ptrs + 1809372866

myc.cnf:
[mysqld]
socket=/tmp/mysql.sock
user=mysql
skip-innodb
language=/usr/local/mysql/share/mysql/french
set-variable=key_buffer_size=256M
set-variable=table_cache=1024
set-variable=join_buffer_size=4M
set-variable=read_buffer_size=4M
set-variable=sort_buffer_size=4M
set-variable=thread_stack=262144
set-variable = ft_min_word_len=3
set-variable = tmp_table_size=64M
# replication maitre
log-bin
server-id = 2

[mysqld_safe]
socket=/tmp/mysql.sock
err-log=/var/log/mysql/mysqld.log
pid-file=/var/mysql/mysqld.pid

extract of file mysqld.log:
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=4190208
max_used_connections=2
max_connections=100
threads_connected=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1212015 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9e9540
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x2ae021242d28, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x617571696c706572
New value of fp=0x9e9540 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=2
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
070110 04:22:10  mysqld restarted
/usr/local/mysql/libexec/mysqld: Prêt pour des connections  Source distribution
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=4190208
max_used_connections=2
max_connections=100
threads_connected=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1212015 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9e9540
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x2ac7343d5d28, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x617571696c706572
New value of fp=0x9e9540 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=2
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
070110 04:26:14  mysqld restarted
[23 Jan 2007 11:41] Valeriy Kravchuk
Thank you for a problem report. Do you have a corresponding core dump for this crash? Is it possible to upload the appropriate binlog(s) to our ftp server?
[23 Jan 2007 13:59] Stéphane CANEL
my.cnf

Attachment: mycnf.txt (text/plain), 530 bytes.

[24 Feb 2007 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[6 Mar 2007 20:28] Sveta Smirnova
Thank you for the feedback.

But attached file is mysql error log file, not core dump.

Also, please, upload binary log files which cause corruption.
[7 Mar 2007 9:49] Stéphane CANEL
binlog after crash (2)

Attachment: Eliot-bdoc-bin.000965 (application/octet-stream, text), 1.27 KiB.

[23 May 2007 9:59] Valeriy Kravchuk
Please, check if there are any "core*" files created as a result (they should/may be in the data directory). Please, send the results of:

getconf GNU_LIBC_VERSION
getconf GNU_LIBPTHREAD_VERSION
free
ulimit -a
[23 May 2007 10:10] Stéphane CANEL
# getconf GNU_LIBC_VERSION
glibc 2.3.4

# getconf GNU_LIBPTHREAD_VERSION
NPTL 2.3.4

# free
             total       used       free     shared    buffers     cached
Mem:       4039308    3729348     309960          0     191684    1841972
-/+ buffers/cache:    1695692    2343616

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 38912
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 38912
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[7 Sep 2007 8:17] Sveta Smirnova
Thank you for the feedback.

File Eliot-bdoc-bin.000963 is corrupted. Could you please provide full Eliot-bdoc-bin.000963 file instead of last 50 lines? You can upload it on our FTP server as described in "Files" tab.

If you can't there is query "UPDATE bagheera_eliot.destinataire_diffusion_droit_photo SET etat_traitement = 3 WHERE id_diffusion = '101570' AND id_util = '421' AND id_photo = " only we can find in the Eliot-bdoc-bin.000963. Please provide output of SHOW CREATE TABLE and SHOW TABLE STATUS for table bagheera_eliot.destinataire_diffusion_droit_photo
[7 Oct 2007 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[15 Jan 2009 12:07] Susanne Ebrecht
MySQL 4.1.22 is a very old version. All our tests showed that this won't happen by using MySQL 5.0 or 5.1 anymore.

Please try our actual version MySQL 5.1 and let us know if all works fine for you.
[16 Feb 2009 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".