MySQL Bugs: #58164: Regular crash of server

Bug #58164	Regular crash of server
Submitted:	12 Nov 2010 13:36	Modified:	1 May 2013 16:37
Reporter:	Lionel Frachon	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: General	Severity:	S1 (Critical)
Version:	5.1.53	OS:	Linux (Debian 64bits)
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
Hello,

we face for a while regular crashes of MySql engine, generally under load. The reported query works ok (it is run a thousand times a day) and from the error stack dump, I suspect (possibly wrongly) an internal issue on threads or locking. 

We've been under 5.1.46 for about 6 months, and upgraded yesterday to 5.1.52 to ensure the problem has not been previsouly solved by an intermediate release.

Server is a 16 GB box with 2xQuad, Intel SSD (with binlogs & 1 database on it) and hard disks (for the other databases).

Any help much appreciated. 

Here is the dump stack (generally looking the same from crash to crash):

101112 12:00:12 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=8388608
max_used_connections=50
max_threads=128
threads_connected=49
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1705259 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x7fe53c185130
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x459f5f20 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x85520e]
/usr/sbin/mysqld(handle_segfault+0x322)[0x5c3422]
/lib/libpthread.so.0[0x7fe85ebf9a80]
/lib/libpthread.so.0(pthread_mutex_lock+0x1a)[0x7fe85ebf34aa]
/usr/sbin/mysqld(thr_lock+0x4c)[0x859aac]
/usr/sbin/mysqld(thr_multi_lock+0xaa)[0x859f2a]
/usr/sbin/mysqld(_Z17mysql_lock_tablesP3THDPP8st_tablejjPb+0x24c)[0x5bde6c]
/usr/sbin/mysqld(_Z11lock_tablesP3THDP10TABLE_LISTjPb+0x308)[0x60bf48]
/usr/sbin/mysqld(_Z28open_and_lock_tables_derivedP3THDP10TABLE_LISTb+0x3c)[0x61276c]
/usr/sbin/mysqld[0x5cdbb2]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c1)[0x5d0c01]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x2ae)[0x5d6a3e]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x585)[0x5d6fd5]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xe4)[0x5d8104]
/usr/sbin/mysqld(handle_one_connection+0x6f0)[0x5ca5f0]
/lib/libpthread.so.0[0x7fe85ebf1fc7]
/lib/libc.so.6(clone+0x6d)[0x7fe85e09059d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1732b5c0 = SELECT Remise, IDPointDeVente, ChiffreAffaire, IDArticle, Quantite, IDMagasin, ARecalculer, PrixUnitaireMoyen, IDAnnuelArticle, Annee FROM annuelarticle WHERE (((IDArticle = 6599) AND (IDPointDeVente = 8532)) AND (Annee = 2010))
thd->thread_id=9897
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
mysqld: my_new.cc:51: int __cxa_pure_virtual(): Assertion `! "Aborted: pure virtual method called."' failed.
Fatal signal 6 while backtracing
                                  

How to repeat:
Random under heavy load.

Hello,

to explain a little bit more the context if this can help understanding, this box is a slave for the data gathering server (running blackbox engine) and a master for the reporting database (after data has been processed on this very box).

This looks similar to Bug#57249, which also affects 5.1.

Out of curiosity, were you using any 5.1 version, before 5.1.46, which didn't have this problem?

Hello,

nope, we've started the project with 5.1.46

Rgds

Looking at the related bugs, and even if there is no apparent relationship, I realized the tuning was not done concerning number of procs/open files for mysql user.

I adjusted the parameters (16384 each) just in case...

A colleague suspect this might be a glibc related problem. Please, paste the output of: getconf GNU_LIBC_VERSION and rpm -q glibc.

Hi,

as requested :

# getconf GNU_LIBC_VERSION 
glibc 2.7

# dpkg -I libc6
[...]

ii libc6 2.7-18lenny2 GNU C Library: Shared libraries

Thanks

Didn't see the 2010-3847 CVE, so upgraded now to 2.7-18lenny6.

Hello,

is the information provided sufficient or pertinent regarding your colleagues' concerns ?

Thanks

Yes, but we guess it's something else. Have you been able to get a core file?

Hello,

if you could indicate/point me to/ how to get a core file, I'll be happy to do so.

Thanks

Sorry, found the doc, need to restart the server then, so it'll be for tonight.

Thank you for the feedback.

We will be waiting core file.

Hello,

another crash today, but sligthly different in the resulting tracefile (see attached) with core_file enabled and limited to 2GB.

Mysql error log

Attachment: bug_error_58164.txt (text/plain), 12.28 KiB.

And btw, there were no core.pid generated in mysql's basedir, whereas we've restarted the mysql server yesterday using 

[mysqld_safe]
core_file_size  =  2147483648

[mysqld]
core_file

and adding ulimit -c unlimited at the OS level. 

???

There are also some kernel related options related to core file. Try sysctl..

Added kernel.core_uses_pid =1 to /etc/sysctl.conf, "commited" to the system using sysctl -c & restarted the MySql server.

Thanks for the tip.

Hello,

crashed again this morning, with no core file (serached through the whole fs)  neither detailed dump 

Here is the log:

 101120  9:23:22 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=8388608
max_used_connections=46
max_threads=128
threads_connected=43
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1705259 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x7fb9c0055ad0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x47705f20 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x85520e]
/usr/sbin/mysqld(handle_segfault+0x322)[0x5c3422]
/lib/libpthread.so.0[0x7fbce33b6a80]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1737db00 = /*  */ SELECT utilisateur.MotDePasse, pointdevente.IDMagasin FROM (((droitsdacces INNER JOIN utilisateur ON (droitsdacces.IDUtilisateur = utilisateur.IDUtilisateur)) INNER JOIN pointdevente ON (pointdevente.IDMagasin = droitsdacces.IDMagasin)) INNER JOIN equipement ON (pointdevente.IDPointDeVente = equipement.IDPointDeVente)) INNER JOIN droitsdacces_serviceweb ON (droitsdacces_serviceweb.IDDroitsDAcces =droitsdacces.IDDroitsDAcces) WHERE (utilisateur.IDUtilisateur =8) AND(utilisateur.IDGestionnaire = 2) AND(pointdevente.IDPointDeVente = 6410 )AND(droitsdacces.Suspendu = 0) AND(droitsdacces_serviceweb.IDServiceWeb = 2) AND(equipement.IDEquipement = 9386)
thd->thread_id=3712
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

Help much appreciated, thanks !

What is the value of kernel.core_pattern? Also, you can check manually by killing mysqld with kill -s SIGSEGV pid. It should generate a core file, if it doesn't, something is wrong.

Another thing. Have you checked the system memory? Usually, when there is a bug, the stack trace is more or less consistent.

I Work with Lionel Frachon.
The value of kernel.core_pattern is :
/home/coredumps/core-%e-%s-%u-%g-%p-%t

Looks about right. When possible, try killing mysqld with kill -s SIGSEGV <pid> to check that a core file is being generated on crashes.

Hello,

We have 2 core dumps in the mentionned directory. Will upload them later, they're 13GB in size...

OK. Please compress the core files. In the mean time, please try getting a backtrace with the core file:

http://dev.mysql.com/doc/refman/5.1/en/using-gdb-on-mysqld.html

The relevant instructions are:

shell> gdb mysqld core
gdb>   backtrace full
gdb>   quit

Hello,

from the first core :

gdb mysqld core-mysqld-11-1001-106-7429-1290241402
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_mdns4_minimal.so.2
Reading symbols from /lib/libnss_dns.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
(no debugging symbols found)
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --log-error='.
Program terminated with signal 11, Segmentation fault.
[New process 8074]
[New process 8368]
[New process 8075]
[New process 13083]
[New process 7615]
[New process 7587]
[New process 7588]
[New process 8401]
[New process 7613]
[New process 8369]
[New process 7589]
[New process 7590]
[New process 7626]
[New process 7637]
[New process 7709]
[New process 7608]
[New process 7632]
[New process 7429]
[New process 7612]
[New process 7754]
[New process 7668]
[New process 7609]
[New process 7619]
[New process 7624]
[New process 7986]
[New process 7713]
[New process 7620]
[New process 7633]
[New process 7635]
[New process 7720]
[New process 7892]
[New process 7887]
[New process 7775]
[New process 7638]
[New process 7641]
[New process 7985]
[New process 7886]
[New process 7649]
[New process 7645]
[New process 7731]
[New process 8073]
[New process 7670]
[New process 7646]
[New process 7672]
[New process 7673]
[New process 7702]
[New process 7987]
[New process 7607]
[New process 7667]
[New process 7651]
[New process 7723]
[New process 7614]
[New process 7885]
[New process 7721]
[New process 7704]
[New process 7790]
[New process 7789]
[New process 7644]
#0  0x00007fbce33b3c36 in pthread_kill () from /lib/libpthread.so.0
(gdb) backtrace full
#0  0x00007fbce33b3c36 in pthread_kill () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00000000005c3455 in handle_segfault ()
No symbol table info available.
#2  <signal handler called>
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.
#4  0x0000000000559264 in Arg_comparator::set_compare_func ()
No symbol table info available.
#5  0x0000000000558f48 in Arg_comparator::set_cmp_func ()
No symbol table info available.
#6  0x000000000055dba5 in Item_equal::compare_const ()
No symbol table info available.
#7  0x000000000055dca0 in Item_equal::update_const ()
No symbol table info available.
#8  0x0000000000620a03 in ?? ()
No symbol table info available.
#9  0x00000000006209bc in ?? ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#10 0x00000000006315bb in ?? ()
No symbol table info available.
#11 0x0000000000631fa2 in ?? ()
No symbol table info available.
#12 0x0000000000634af6 in JOIN::optimize ()
No symbol table info available.
#13 0x000000000063d5a1 in mysql_select ()
No symbol table info available.
#14 0x000000000063df89 in handle_select ()
No symbol table info available.
#15 0x00000000005cdbf4 in ?? ()
No symbol table info available.
#16 0x00000000005d0c01 in mysql_execute_command ()
No symbol table info available.
#17 0x00000000005d6a3e in mysql_parse ()
No symbol table info available.
#18 0x00000000005d6fd5 in dispatch_command ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#19 0x00000000005d8104 in do_command ()
No symbol table info available.
#20 0x00000000005ca5f0 in handle_one_connection ()
No symbol table info available.
#21 0x00007fbce33aefc7 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#22 0x00007fbce284d64d in clone () from /lib/libc.so.6
No symbol table info available.
#23 0x0000000000000000 in ?? ()
No symbol table info available.

Second core gives 

 gdb mysqld core-mysqld-11-1001-106-19699-1290365416
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_mdns4_minimal.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
(no debugging symbols found)
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --log-error='.
Program terminated with signal 11, Segmentation fault.
[New process 20899]
[New process 20900]
[New process 20902]
[New process 20901]
[New process 20903]
[New process 21864]
[New process 21885]
[New process 20904]
[New process 20905]
[New process 20284]
[New process 20280]
[New process 20283]
[New process 20381]
[New process 21609]
[New process 20379]
[New process 20285]
[New process 20282]
[New process 20293]
[New process 4005]
[New process 20287]
[New process 20380]
[New process 20377]
[New process 20289]
[New process 20288]
[New process 20294]
[New process 20376]
[New process 20279]
[New process 19699]
[New process 20292]
[New process 20290]
[New process 19701]
[New process 20291]
[New process 19702]
[New process 20286]
[New process 20258]
[New process 20281]
[New process 20263]
[New process 20278]
[New process 20265]
[New process 20277]
[New process 20266]
[New process 20276]
[New process 20259]
[New process 20275]
[New process 20262]
[New process 20274]
[New process 20267]
[New process 19703]
[New process 20269]
[New process 20264]
[New process 20270]
[New process 20272]
[New process 20273]
[New process 20257]
[New process 20271]
[New process 29305]
[New process 20268]
[New process 20261]
[New process 19704]
#0  0x00007f9297e97c36 in pthread_kill () from /lib/libpthread.so.0
(gdb) backtrace full
#0  0x00007f9297e97c36 in pthread_kill () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00000000005c3455 in handle_segfault ()
No symbol table info available.
#2  <signal handler called>
No symbol table info available.
#3  0x000000000062c6ab in ?? ()
No symbol table info available.
#4  0x00000000006347ac in JOIN::optimize ()
No symbol table info available.
#5  0x000000000063d5a1 in mysql_select ()
No symbol table info available.
#6  0x000000000063df89 in handle_select ()
No symbol table info available.
#7  0x00000000005cdbf4 in ?? ()
No symbol table info available.
#8  0x00000000005d0c01 in mysql_execute_command ()
No symbol table info available.
#9  0x00000000005d6a3e in mysql_parse ()
No symbol table info available.
#10 0x00000000005d6fd5 in dispatch_command ()
No symbol table info available.
#11 0x00000000005d8104 in do_command ()
No symbol table info available.
#12 0x00000000005ca5f0 in handle_one_connection ()
No symbol table info available.
#13 0x00007f9297e92fc7 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#14 0x00007f929733164d in clone () from /lib/libc.so.6
No symbol table info available.
#15 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

You guys are building MySQL yourselves or are using one of our binaries packages?

Hello,

nope, using binary rpm converted through alien for Debian.

Hum, strange, the RPM mysqld is normally built with debug symbols. Did also a mysqld.debug (or something like) get installed? If yes, replace the regular mysqld with it (after making a copy..).

BTW, just to double check, those cores are from a regular crash, not from the KILL, right?

> BTW, just to double check, those cores are from a regular crash, not from the KILL, right?

Yes :)

find / -name mysql.debug did not find anything.

OK. Just wanted to double check because the stack traces are varying quite a bit in each crash. This is usually a signal of deeper problems.

> find / -name mysql.debug did not find anything.

myqsld, not mysql. It comes with MySQL-debuginfo rpm and is installed in:

/usr/lib/debug/usr/sbin/mysqld-debug.debug

Another thing, you guys are using the generic RPM?

Hello again,

pushing the 2 core dumps under cores_for_58164.tar.bz2 on ftp site

The mysqld.debug is not present on the server. 

If I install it through mysql debug package, should we expect perf degradation ?

> Another thing, you guys are using the generic RPM?

Using the rhel5 community-server rpm from the site. May this have an influence ?

Found out the mysqld-debug in /usr/sbin, so updated the main mysqld with this one. Original mysqld was saved under mysqld.original, and mysql server restarted.

> Using the rhel5 community-server rpm from the site. May this have an influence ?

Could have.. although its just a long shot. Using the generic tarballs is recommend anyway. Given the core files, I think that's all we need thus far. Thanks.

> Using the generic tarballs is recommend anyway.

Ok, will know about it.

Hi

a new crash this afternoon, here is relative information to it (remembering we used mysql-debug since the last crash, which makes me dubitative about the -debug one...

gdb /usr/sbin/mysqld-debug core-mysqld-11-1001-106-21620-1291821329
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)

warning: core file may not match specified executable file.
(no debugging symbols found)
(no debugging symbols found)
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --log-error='.
Program terminated with signal 11, Segmentation fault.
[New process 21817]
[New process 24969]
[New process 27637]
[New process 25894]
[New process 24254]
[New process 25036]
[New process 21773]
[New process 21667]
[New process 15039]
[New process 21665]
[New process 21633]
[New process 21634]
[New process 21671]
[New process 21644]
[New process 21635]
[New process 21620]
[New process 21816]
[New process 21676]
[New process 21646]
[New process 21636]
[New process 21649]
[New process 21664]
[New process 21673]
[New process 21675]
[New process 21651]
[New process 23290]
[New process 21772]
[New process 21666]
[New process 21663]
[New process 21685]
[New process 21642]
[New process 21677]
[New process 21681]
[New process 21679]
[New process 21652]
[New process 21774]
[New process 21672]
[New process 21770]
[New process 21776]
[New process 21682]
[New process 21656]
[New process 21684]
[New process 21643]
[New process 21775]
[New process 21674]
[New process 21657]
[New process 21680]
[New process 21650]
[New process 21669]
[New process 21645]
[New process 21678]
[New process 21653]
[New process 21654]
[New process 21668]
[New process 21683]
[New process 21655]
[New process 21670]
[New process 24253]
#0  0x00007fc3ac369c36 in ?? ()
(gdb) backtrace full
#0  0x00007fc3ac369c36 in ?? ()
No symbol table info available.
#1  0x0000000000a195b1 in yaSSL::Sessions::lookup ()
No symbol table info available.
#2  0x000000000064adbd in mysql_execute_command ()
No symbol table info available.
#3  0x00007fc3ac36ca80 in ?? ()
No symbol table info available.
#4  0x0000000000000001 in ?? ()
No symbol table info available.
#5  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) q

Sending the core file in next minutes, called core-mysqld-11-1001-106-21620-1291821329.bz2

Btw, upgraded to 5.1.53 with generic Linux rpms, as advised above.

Hello, upgraded yesterday but another crash today, here is gdb info info. Please update if you need the core file

# gdb mysqld ./core-mysqld-11-1001-106-20802-1291895515 
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_compat.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_mdns4_minimal.so.2
Reading symbols from /lib/libnss_dns.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2

(no debugging symbols found)
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --log-error='.
Program terminated with signal 11, Segmentation fault.
[New process 20859]
[New process 20985]
[New process 18217]
[New process 20986]
[New process 20804]
[New process 20815]
[New process 20987]
[New process 20805]
[New process 20802]
[New process 24691]
[New process 20827]
[New process 20988]
[New process 23836]
[New process 20809]
[New process 20817]
[New process 20822]
[New process 15803]
[New process 4079]
[New process 20845]
[New process 20806]
[New process 20816]
[New process 20814]
[New process 20833]
[New process 20811]
[New process 20830]
[New process 20834]
[New process 20813]
[New process 20839]
[New process 20819]
[New process 20838]
[New process 20837]
[New process 20821]
[New process 20843]
[New process 20836]
[New process 20828]
[New process 20846]
[New process 20825]
[New process 20832]
[New process 20820]
[New process 20842]
[New process 20807]
[New process 20841]
[New process 20823]
[New process 20840]
[New process 20826]
[New process 20824]
[New process 20835]
[New process 20818]
[New process 20829]
[New process 20810]
[New process 20844]
[New process 20831]
[New process 20847]
[New process 20812]
#0  0x00007f1f96806c36 in pthread_kill () from /lib/libpthread.so.0
(gdb) backtrace full
#0  0x00007f1f96806c36 in pthread_kill () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00000000005bc757 in handle_segfault ()
No symbol table info available.
#2  <signal handler called>
No symbol table info available.
#3  0x0000000000611d44 in ?? ()
No symbol table info available.
#4  0x000000000060f8b2 in ?? ()
No symbol table info available.
#5  0x000000000060bc83 in JOIN::optimize ()
No symbol table info available.
#6  0x000000000060f0e7 in mysql_select ()
No symbol table info available.
#7  0x000000000060a9df in handle_select ()
No symbol table info available.
#8  0x00000000005c9505 in mysql_execute_command ()
No symbol table info available.
#9  0x00000000005cdae4 in mysql_parse ()
No symbol table info available.
#10 0x00000000005c6475 in dispatch_command ()
No symbol table info available.
#11 0x00000000005c5ef0 in do_command ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#12 0x00000000005c4d26 in handle_one_connection ()
No symbol table info available.
#13 0x00007f1f96801fc7 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#14 0x00007f1f95ca064d in clone () from /lib/libc.so.6
No symbol table info available.
#15 0x0000000000000000 in ?? ()
No symbol table info available.

Hi Lionel, I am going to reproduce this bug, could you please spot me some questions:

- what exactly version of linux do you use? According to your libc version it must be some debian 5.0.x, but what is last number? (I am going to install it)

- do you use any preload/prelink?

- could you provide table definitions and common queries? (that I could  populate by random data and create similar workload)

- I see some yassl references in the backtrace, do you use ssl for connections?

- as you wrote it's slave of some master. Does that master have any otner slaves? if so, do they work properly?

- Do you have any problems on master?

Thanks in advance...

Hello Alexey,

here are some answers :
> what exactly version of linux do you use? According to your libc version it must be some debian 5.0.x, but what is last number? (I am going to install it)

It is Debian 5.0.7 x86_64.

> do you use any preload/prelink?
No

> could you provide table definitions and common queries? (that I could  populate by random data and create similar workload).

Yes, if it's not seeable by other users. To give a bit of context, the database runs partly over SSDs (binlogs & acquisition tables), has 16GB RAM of which 13 are allocated to MySql and does ~600 tps according to Munin.

> I see some yassl references in the backtrace, do you use ssl for connections?
Yes, for all the replications. Works well, btw.

> as you wrote it's slave of some master. Does that master have any otner slaves? if so,do they work properly?

No, it's the only slave. It's the processing one of our arch, gathers binlogs from the master (called acquisition, runs over blackbox engine as front-end with Glassfish AS), writes data in SSDs, then crontabs every minute to re-read and write to disk consolidated data).
It works aswell as a master of a final slave database which is aswell queried for R/O queries. 

> Do you have any problems on master?
No, running without a glitch for months.

> Thanks in advance...
Thank you ! (tm Elvis)

Hi Lionel, I installed debian 64bits, mysql from rpm, but so far I dont have any crashes. It means I need your table definitions and common queries. 

If it's possible attach pls them to this bug record (you can set privacy level "only developers" on upload page), or, if it's more convenient to you, you can send them to me to alexey.kishkin@oracle.com.

Thanks in advance.

Hello,

my CTO is not really comfortable about the whole database structure being sent.
I'm attaching some numbers 
to understand the volumetry context, time for me to discuss the point with him. The hostnames have been changed.

File is named Numbers-bug-data-58164.zip and will be up'd to your ftp in minutes.

Hello,

two other crashes from yesterday and this morning, here are the backtraces :

gdb mysqld core-mysqld-11-1001-106-20802-1291895515
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_mdns4_minimal.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
(no debugging symbols found)
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --log-error='.
Program terminated with signal 11, Segmentation fault.
[New process 20859]
[New process 20985]
[New process 18217]
[New process 20986]
[New process 20804]
[New process 20815]
[New process 20987]
[New process 20805]
[New process 20802]
[New process 24691]
[New process 20827]
[New process 20988]
[New process 23836]
[New process 20809]
[New process 20817]
[New process 20822]
[New process 15803]
[New process 4079]
[New process 20845]
[New process 20806]
[New process 20816]
[New process 20814]
[New process 20833]
[New process 20811]
[New process 20830]
[New process 20834]
[New process 20813]
[New process 20839]
[New process 20819]
[New process 20838]
[New process 20837]
[New process 20821]
[New process 20843]
[New process 20836]
[New process 20828]
[New process 20846]
[New process 20825]
[New process 20832]
[New process 20820]
[New process 20842]
[New process 20807]
[New process 20841]
[New process 20823]
[New process 20840]
[New process 20826]
[New process 20824]
[New process 20835]
[New process 20818]
[New process 20829]
[New process 20810]
[New process 20844]
[New process 20831]
[New process 20847]
[New process 20812]
#0  0x00007f1f96806c36 in pthread_kill () from /lib/libpthread.so.0
(gdb) backtrace full
#0  0x00007f1f96806c36 in pthread_kill () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00000000005bc757 in handle_segfault ()
No symbol table info available.
#2  <signal handler called>
No symbol table info available.
#3  0x0000000000611d44 in ?? ()
No symbol table info available.
#4  0x000000000060f8b2 in ?? ()
No symbol table info available.
#5  0x000000000060bc83 in JOIN::optimize ()
No symbol table info available.
#6  0x000000000060f0e7 in mysql_select ()
No symbol table info available.
#7  0x000000000060a9df in handle_select ()
No symbol table info available.
#8  0x00000000005c9505 in mysql_execute_command ()
No symbol table info available.
#9  0x00000000005cdae4 in mysql_parse ()
No symbol table info available.
#10 0x00000000005c6475 in dispatch_command ()
No symbol table info available.
#11 0x00000000005c5ef0 in do_command ()
No symbol table info available.
#12 0x00000000005c4d26 in handle_one_connection ()
No symbol table info available.
#13 0x00007f1f96801fc7 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#14 0x00007f1f95ca064d in clone () from /lib/libc.so.6
No symbol table info available.
#15 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) q

And second 

Serveur MASTER : /home/coredumps # ls -ltr
total 28389844
-rw------- 1 root mysql  4086894980 d�c  8 16:20 core-mysqld-11-1001-106-21620-1291821329.bz2
-rw------- 1 root mysql 15419346944 d�c  9 12:54 core-mysqld-11-1001-106-20802-1291895515
-rw------- 1 root mysql 15653310464 d�c 10 10:03 core-mysqld-11-1001-106-2200-1291971480
Serveur MASTER : /home/coredumps #  gdb mysqld core-mysqld-11-1001-106-2200-1291971480 
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_mdns4_minimal.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
(no debugging symbols found)
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --log-error='.
Program terminated with signal 11, Segmentation fault.
[New process 2930]
[New process 3058]
[New process 5739]
[New process 2899]
[New process 2885]
[New process 2888]
[New process 2204]
[New process 3296]
[New process 2892]
[New process 2946]
[New process 2205]
[New process 2891]
[New process 2200]
[New process 2913]
[New process 2206]
[New process 2207]
[New process 2918]
[New process 2887]
[New process 2920]
[New process 2889]
[New process 2886]
[New process 2914]
[New process 2894]
[New process 2921]
[New process 2893]
[New process 2905]
[New process 2898]
[New process 2906]
[New process 2924]
[New process 2910]
[New process 2902]
[New process 2926]
[New process 2922]
[New process 2903]
[New process 2923]
[New process 2896]
[New process 2890]
[New process 3057]
[New process 2897]
[New process 2917]
[New process 2904]
[New process 2900]
[New process 5807]
[New process 2901]
[New process 2915]
[New process 2919]
[New process 2912]
[New process 2907]
[New process 2895]
[New process 2911]
[New process 2909]
[New process 2908]
[New process 2916]
[New process 24000]
#0  0x00007f1b0c815c36 in pthread_kill () from /lib/libpthread.so.0
(gdb) backtrace full 
#0  0x00007f1b0c815c36 in pthread_kill () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00000000005bc757 in handle_segfault ()
No symbol table info available.
#2  <signal handler called>
No symbol table info available.
#3  0x0000000000675000 in SEL_ARG::clone ()
No symbol table info available.
#4  0x000000000067527a in SEL_ARG::clone_tree ()
No symbol table info available.
#5  0x000000000067aa7e in ?? ()
No symbol table info available.
#6  0x000000000067a278 in ?? ()
No symbol table info available.
#7  0x000000000067952b in ?? ()
No symbol table info available.
#8  0x000000000067595a in SQL_SELECT::test_quick_select ()
No symbol table info available.
#9  0x000000000060f3f1 in ?? ()
No symbol table info available.
#10 0x000000000060fe0b in ?? ()
No symbol table info available.
#11 0x000000000060bc83 in JOIN::optimize ()
No symbol table info available.
#12 0x000000000060f0e7 in mysql_select ()
No symbol table info available.
#13 0x000000000060a9df in handle_select ()
No symbol table info available.
#14 0x00000000005cc459 in ?? ()
No symbol table info available.
#15 0x00000000005c7b3a in mysql_execute_command ()
No symbol table info available.
#16 0x00000000005cdae4 in mysql_parse ()
No symbol table info available.
#17 0x00000000005c6475 in dispatch_command ()
No symbol table info available.
#18 0x00000000005c5ef0 in do_command ()
No symbol table info available.
#19 0x00000000005c4d26 in handle_one_connection ()
No symbol table info available.
#20 0x00007f1b0c810fc7 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#21 0x00007f1b0bcaf64d in clone () from /lib/libc.so.6
No symbol table info available.
#22 0x0000000000000000 in ?? ()
No symbol table info available.

Hello,

apart from giving the database structure, in which way can we help to help you identifying the issue ?

I'm far from being a debug specialist, but libpthread.so comes systematically in debug messages; any method to identify if it may come from there ?

if it's a memory corruption happening elsewhere, you can find it by running mysqld under valgrind.  but it'll run 100x times slower :(

Another crash this afternoon after changing innodb_max_dirty_pct to 95.

We had stopped the archive crons, so excluding them from equation. 

Just connecting now, the load of server was 48 (!), with no process consuming cpu (incl.mysql which was 0.0), then mysql restarted its standard usage (between 40 and 100% of one cpu)

Here is another backtrace :

Serveur MASTER : /home/coredumps # gdb /usr/sbin/mysqld core-mysqld-11-1001-106-26191-1292514460  
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_mdns4_minimal.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
(no debugging symbols found)
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --log-error='.
Program terminated with signal 11, Segmentation fault.
[New process 30972]
[New process 26191]
[New process 26198]
[New process 26200]
[New process 30367]
[New process 26199]
[New process 26972]
[New process 26201]
[New process 26974]
[New process 26976]
[New process 27147]
[New process 26973]
[New process 26980]
[New process 27121]
[New process 28659]
[New process 26990]
[New process 26983]
[New process 26997]
[New process 26975]
[New process 26996]
[New process 27142]
[New process 26999]
[New process 26995]
[New process 27123]
[New process 27006]
[New process 26979]
[New process 27011]
[New process 26991]
[New process 27002]
[New process 26985]
[New process 27001]
[New process 29418]
[New process 27009]
[New process 27007]
[New process 26986]
[New process 26988]
[New process 27124]
[New process 27000]
[New process 27003]
[New process 26993]
[New process 26994]
[New process 27005]
[New process 27008]
[New process 27004]
[New process 30366]
[New process 26981]
[New process 26998]
[New process 27012]
[New process 26992]
[New process 26982]
[New process 27125]
[New process 26987]
[New process 27010]
[New process 26984]
[New process 27122]
#0  0x00007f77c3563c36 in pthread_kill () from /lib/libpthread.so.0
(gdb) backtrace full
#0  0x00007f77c3563c36 in pthread_kill () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00000000005bc757 in handle_segfault ()
No symbol table info available.
#2  <signal handler called>
No symbol table info available.
#3  0x000000000070d490 in ha_archive::pack_row ()
No symbol table info available.
#4  0x000000000070d27d in ha_archive::real_write_row ()
No symbol table info available.
#5  0x000000000070df79 in ha_archive::optimize ()
No symbol table info available.
#6  0x000000000070de4d in ha_archive::repair ()
No symbol table info available.
#7  0x000000000068b0b3 in handler::ha_repair ()
No symbol table info available.
#8  0x000000000069ec91 in ?? ()
No symbol table info available.
#9  0x000000000069f355 in mysql_repair_table ()
No symbol table info available.
#10 0x00000000005c92d8 in mysql_execute_command ()
No symbol table info available.
#11 0x00000000005cdae4 in mysql_parse ()
No symbol table info available.
#12 0x00000000005c6475 in dispatch_command ()
No symbol table info available.
#13 0x00000000005c5ef0 in do_command ()
No symbol table info available.
#14 0x00000000005c4d26 in handle_one_connection ()
No symbol table info available.
#15 0x00007f77c355efc7 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#16 0x00007f77c29fd64d in clone () from /lib/libc.so.6
No symbol table info available.
#17 0x0000000000000000 in ?? ()
No symbol table info available.

Hi Lionel,

>two other crashes from yesterday and this morning, here are the backtraces :

>Serveur MASTER : /home/coredumps #  gdb mysqld core-mysqld-11-1001-106-2200-1291971480 

>gdb mysqld core-mysqld-11-1001-106-20802-1291895515

No, that core dumps were created 9 Dec and 10 Dec, (the name contains timestamp)

>apart from giving the database structure, in which way can we help to help you
identifying the issue ?

I still need some .. "point of support". If not the whole database, then at least some part.

20 Nov you submitted log (to this bug record), that contains one of the problem queries, I could start testing from this query, but I need structures of tables which are used in this query. I mean: utilisateur, pointdevente, droitsdacces, equipement and droitsdacces_serviceweb. If that table defs are still too sensitive information,  you could 'obfuscate' that table definitions (i.e. replace field names to a1,a2,a3... in the table structures and (of course) in the query).

Hello,

it looks like a side issue of not having enough possible open tables, or the innodb_max_dirty_pct flush (or anything in combination with both ^^). After tuning both (first to 16k-1, second to 0), we did not suffer any crash, although we had bigger volumes.

So dunno what to say about this bug, but maybe it may serve for diagnostic of similar issue...

2 other crashes today, unexpectingdly, here are the logs

110108 15:37:45 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=8388608
max_used_connections=66
max_threads=128
threads_connected=38
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1705260 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x7f0167375900
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4876a128 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace+0x3a)[0x828f3a]
/usr/sbin/mysqld(handle_segfault+0x1f7)[0x5bc637]
/lib/libpthread.so.0[0x7f04a5459a80]
/usr/sbin/mysqld(_ZN10Item_equal8containsEP5Field+0x31)[0x55dc51]
/usr/sbin/mysqld(_ZN10Item_field15find_item_equalEP10COND_EQUAL+0x54)[0x52b724]
/usr/sbin/mysqld[0x618df0]
/usr/sbin/mysqld[0x619325]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x1a63)[0x60d2d3]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xd7)[0x60f0e7]
/usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x15f)[0x60a9df]
/usr/sbin/mysqld[0x5cc459]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1ca)[0x5c7b3a]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x214)[0x5cdae4]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x435)[0x5c6475]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xd0)[0x5c5ef0]
/usr/sbin/mysqld(handle_one_connection+0x126)[0x5c4d26]
/lib/libpthread.so.0[0x7f04a5451fc7]
/lib/libc.so.6(clone+0x6d)[0x7f04a48f064d]

And 

Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x474f4128 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace+0x3a)[0x828f3a]
/usr/sbin/mysqld(handle_segfault+0x1f7)[0x5bc637]
/lib/libpthread.so.0[0x7f9a07b95a80]
/lib/libc.so.6(gsignal+0x35)[0x7f9a06f8eed5]
/lib/libc.so.6(abort+0x183)[0x7f9a06f903f3]
/lib/libc.so.6(__assert_fail+0xe9)[0x7f9a06f87dc9]
/usr/sbin/mysqld[0x819f8e]
/usr/sbin/mysqld[0x611b34]
/usr/sbin/mysqld[0x60f8b2]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x413)[0x60bc83]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xd7)[0x60f0e7]
/usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x15f)[0x60a9df]
/usr/sbin/mysqld[0x5cc459]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1ca)[0x5c7b3a]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x214)[0x5cdae4]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x435)[0x5c6475]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xd0)[0x5c5ef0]
/usr/sbin/mysqld(handle_one_connection+0x126)[0x5c4d26]
/lib/libpthread.so.0[0x7f9a07b8dfc7]
/lib/libc.so.6(clone+0x6d)[0x7f9a0702c64d]

Thanks, this becomes really problematic.

Hi Lionel,

There are at least 4 different crashes on this bug report.  Can you kindly test 5.5.31 or 5.6.11 once it comes out and let us know how it fares.  Recent versions have hundreds of bugfixes.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".