Description:
From time to time, mysql server crashes and we have to restart it manually.
--------------------------------------------------------------------------------
This is what we get in the log file:
------------------------------------
mysqld got signal 11;
...
key_buffer_size=402653184
read_buffer_size=2093056
max_used_connections=8
max_connections=100
threads_connected=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 802415 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x885d578
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfd9f1d8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80720d4
0x8250d48
0x827a209
0x827a0a3
0x8235a86
0x8236069
0x8094acb
0x808f3e4
0x808f403
0x823b67f
0x808f513
0x8081cab
0x807fac7
0x8080935
0x807c113
0x807bb5e
0x807b388
0x824e4fc
0x828452a
...
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x87a1870 = FLUSH TABLES WITH READ LOCK
thd->thread_id=16702
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
Number of processes running now: 1
mysqld process hanging, pid 3772 - killed
Resolved stack trace reads:
0x80720d4 handle_segfault + 420
0x8250d48 pthread_sighandler + 184
0x827a209 chunk_free + 297
0x827a0a3 free + 147
0x8235a86 my_no_flags_free + 22
0x8236069 free_root + 153
0x8094acb closefrm__FP8st_table + 203
0x808f3e4 intern_close_table__FP8st_table + 36
0x808f403 free_cache_entry__FP8st_table + 19
0x823b67f hash_delete + 719
0x808f513 close_cached_tables__FP3THDbP13st_table_list + 67
0x8081cab reload_acl_and_cache__FP3THDUlP13st_table_list + 379
0x807fac7 mysql_execute_command__Fv + 12023
0x8080935 mysql_parse__FP3THDPcUi + 149
0x807c113 dispatch_command__F19enum_server_commandP3THDPcUi + 1443
0x807bb5e do_command__FP3THD + 158
0x807b388 handle_one_connection + 648
0x824e4fc pthread_start_thread + 220
0x828452a thread_start + 4
--------------------------------------------------------------------------------
Some details on our hardware:
-----------------------------
# cat /proc/scsi/ft3xx/0
PROMISE FastTrak TX4000/376/378/S150 TX Series Linux Driver Version 1.00.0.19
Adapter1 - FastTrak 378
Array - Array[1] : 1X2 Mirror (OK)
Drive -
1 : Maxtor 6Y120M0 IDE1/Master 122941MB IRQ(22) UDMA6 Array[1]
3 : Maxtor 6Y120M0 IDE2/Master 122941MB IRQ(22) UDMA6 Array[1]
# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Pentium(R) 4 CPU 2.40GHz
stepping : 9
cpu MHz : 2405.520
cache size : 512 KB
physical id : 0
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 4797.23
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Pentium(R) 4 CPU 2.40GHz
stepping : 9
cpu MHz : 2405.520
cache size : 512 KB
physical id : 0
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 4810.34
# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 2113933312 1658507264 455426048 0 52858880 1179557888
Swap: 2089209856 0 2089209856
MemTotal: 2064388 kB
MemFree: 444752 kB
MemShared: 0 kB
Buffers: 51620 kB
Cached: 1151912 kB
SwapCached: 0 kB
Active: 859796 kB
ActiveAnon: 363260 kB
ActiveCache: 496536 kB
Inact_dirty: 229508 kB
Inact_laundry: 320 kB
Inact_clean: 478056 kB
Inact_target: 313536 kB
HighTotal: 1179584 kB
HighFree: 18424 kB
LowTotal: 884804 kB
LowFree: 426328 kB
SwapTotal: 2040244 kB
SwapFree: 2040244 kB
--------------------------------------------------------------------------------
Some details on the OS:
-----------------------
# uname -a
Linux parsys.olympia.it 2.4.20-31.9smp #1 SMP Tue Apr 13 17:40:10 EDT 2004 i686 i686 i386 GNU/Linux
Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc /usr/bin/cc
GCC: Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --host=i386-redhat-linux
Thread model: posix
gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)
Compilation info: CC='gcc' CFLAGS='-O2 -mcpu=i486 -fno-strength-reduce' CXX='gcc' CXXFLAGS='-O2 -mcpu=i486 -fno-strength-reduce -felide-constructors -fno-exceptions -fno-rtti ' LDFLAGS='' ASFLAGS=''
LIBC:
lrwxrwxrwx 1 root root 13 May 24 07:56 /lib/libc.so.6 -> libc-2.3.2.so
-rwxr-xr-x 1 root root 1476244 Nov 12 2003 /lib/libc-2.3.2.so
-rw-r--r-- 1 root root 2332200 Nov 12 2003 /usr/lib/libc.a
-rw-r--r-- 1 root root 204 Nov 12 2003 /usr/lib/libc.so
Configure command: ./configure '--disable-shared' '--with-mysqld-ldflags=-all-static' '--with-client-ldflags=-all-static' '--with-server-suffix=-standard' '--without-embedded-server' '--without-berkeley-db' '--with-innodb' '--without-vio' '--without-openssl' '--enable-assembler' '--enable-local-infile' '--with-mysqld-user=mysql' '--with-unix-socket-path=/var/lib/mysql/mysql.sock' '--prefix=/' '--with-extra-charsets=complex' '--exec-prefix=/usr' '--libexecdir=/usr/sbin' '--libdir=/usr/lib' '--sysconfdir=/etc' '--datadir=/usr/share' '--localstatedir=/var/lib/mysql' '--infodir=/usr/share/info' '--includedir=/usr/include' '--mandir=/usr/share/man' '--enable-thread-safe-client' '--with-comment=Official MySQL RPM' 'CC=gcc' 'CFLAGS=-O2 -mcpu=i486 -fno-strength-reduce' 'CXXFLAGS=-O2 -mcpu=i486 -fno-strength-reduce -felide-constructors -fno-exceptions -fno-rtti ' 'CXX=gcc'
How to repeat:
Instal RHL 9 on the above hardware, then official 4.0.20 mysql rpm, start the server, and after some time working ( possibly a lot days ), it crashes.
Suggested fix:
Now we have to restart it manually each time we get this problem.
Description: From time to time, mysql server crashes and we have to restart it manually. -------------------------------------------------------------------------------- This is what we get in the log file: ------------------------------------ mysqld got signal 11; ... key_buffer_size=402653184 read_buffer_size=2093056 max_used_connections=8 max_connections=100 threads_connected=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 802415 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x885d578 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfd9f1d8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x80720d4 0x8250d48 0x827a209 0x827a0a3 0x8235a86 0x8236069 0x8094acb 0x808f3e4 0x808f403 0x823b67f 0x808f513 0x8081cab 0x807fac7 0x8080935 0x807c113 0x807bb5e 0x807b388 0x824e4fc 0x828452a ... Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x87a1870 = FLUSH TABLES WITH READ LOCK thd->thread_id=16702 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 1 mysqld process hanging, pid 3772 - killed Resolved stack trace reads: 0x80720d4 handle_segfault + 420 0x8250d48 pthread_sighandler + 184 0x827a209 chunk_free + 297 0x827a0a3 free + 147 0x8235a86 my_no_flags_free + 22 0x8236069 free_root + 153 0x8094acb closefrm__FP8st_table + 203 0x808f3e4 intern_close_table__FP8st_table + 36 0x808f403 free_cache_entry__FP8st_table + 19 0x823b67f hash_delete + 719 0x808f513 close_cached_tables__FP3THDbP13st_table_list + 67 0x8081cab reload_acl_and_cache__FP3THDUlP13st_table_list + 379 0x807fac7 mysql_execute_command__Fv + 12023 0x8080935 mysql_parse__FP3THDPcUi + 149 0x807c113 dispatch_command__F19enum_server_commandP3THDPcUi + 1443 0x807bb5e do_command__FP3THD + 158 0x807b388 handle_one_connection + 648 0x824e4fc pthread_start_thread + 220 0x828452a thread_start + 4 -------------------------------------------------------------------------------- Some details on our hardware: ----------------------------- # cat /proc/scsi/ft3xx/0 PROMISE FastTrak TX4000/376/378/S150 TX Series Linux Driver Version 1.00.0.19 Adapter1 - FastTrak 378 Array - Array[1] : 1X2 Mirror (OK) Drive - 1 : Maxtor 6Y120M0 IDE1/Master 122941MB IRQ(22) UDMA6 Array[1] 3 : Maxtor 6Y120M0 IDE2/Master 122941MB IRQ(22) UDMA6 Array[1] # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Pentium(R) 4 CPU 2.40GHz stepping : 9 cpu MHz : 2405.520 cache size : 512 KB physical id : 0 siblings : 2 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm bogomips : 4797.23 processor : 1 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Pentium(R) 4 CPU 2.40GHz stepping : 9 cpu MHz : 2405.520 cache size : 512 KB physical id : 0 siblings : 2 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm bogomips : 4810.34 # cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 2113933312 1658507264 455426048 0 52858880 1179557888 Swap: 2089209856 0 2089209856 MemTotal: 2064388 kB MemFree: 444752 kB MemShared: 0 kB Buffers: 51620 kB Cached: 1151912 kB SwapCached: 0 kB Active: 859796 kB ActiveAnon: 363260 kB ActiveCache: 496536 kB Inact_dirty: 229508 kB Inact_laundry: 320 kB Inact_clean: 478056 kB Inact_target: 313536 kB HighTotal: 1179584 kB HighFree: 18424 kB LowTotal: 884804 kB LowFree: 426328 kB SwapTotal: 2040244 kB SwapFree: 2040244 kB -------------------------------------------------------------------------------- Some details on the OS: ----------------------- # uname -a Linux parsys.olympia.it 2.4.20-31.9smp #1 SMP Tue Apr 13 17:40:10 EDT 2004 i686 i686 i386 GNU/Linux Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc /usr/bin/cc GCC: Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2.2/specs Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --host=i386-redhat-linux Thread model: posix gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5) Compilation info: CC='gcc' CFLAGS='-O2 -mcpu=i486 -fno-strength-reduce' CXX='gcc' CXXFLAGS='-O2 -mcpu=i486 -fno-strength-reduce -felide-constructors -fno-exceptions -fno-rtti ' LDFLAGS='' ASFLAGS='' LIBC: lrwxrwxrwx 1 root root 13 May 24 07:56 /lib/libc.so.6 -> libc-2.3.2.so -rwxr-xr-x 1 root root 1476244 Nov 12 2003 /lib/libc-2.3.2.so -rw-r--r-- 1 root root 2332200 Nov 12 2003 /usr/lib/libc.a -rw-r--r-- 1 root root 204 Nov 12 2003 /usr/lib/libc.so Configure command: ./configure '--disable-shared' '--with-mysqld-ldflags=-all-static' '--with-client-ldflags=-all-static' '--with-server-suffix=-standard' '--without-embedded-server' '--without-berkeley-db' '--with-innodb' '--without-vio' '--without-openssl' '--enable-assembler' '--enable-local-infile' '--with-mysqld-user=mysql' '--with-unix-socket-path=/var/lib/mysql/mysql.sock' '--prefix=/' '--with-extra-charsets=complex' '--exec-prefix=/usr' '--libexecdir=/usr/sbin' '--libdir=/usr/lib' '--sysconfdir=/etc' '--datadir=/usr/share' '--localstatedir=/var/lib/mysql' '--infodir=/usr/share/info' '--includedir=/usr/include' '--mandir=/usr/share/man' '--enable-thread-safe-client' '--with-comment=Official MySQL RPM' 'CC=gcc' 'CFLAGS=-O2 -mcpu=i486 -fno-strength-reduce' 'CXXFLAGS=-O2 -mcpu=i486 -fno-strength-reduce -felide-constructors -fno-exceptions -fno-rtti ' 'CXX=gcc' How to repeat: Instal RHL 9 on the above hardware, then official 4.0.20 mysql rpm, start the server, and after some time working ( possibly a lot days ), it crashes. Suggested fix: Now we have to restart it manually each time we get this problem.