| Bug #40481 | Mysql Crash update to version 5.0.67 made it worse | ||
|---|---|---|---|
| Submitted: | 3 Nov 2008 16:40 | Modified: | 21 Dec 2008 7:09 |
| Reporter: | Moshe Sharon | Email Updates: | |
| Status: | No Feedback | Impact on me: | |
| Category: | MySQL Server | Severity: | S1 (Critical) |
| Version: | 5.0.67 | OS: | Linux (CentOS 5.2 64bit) |
| Assigned to: | CPU Architecture: | Any | |
[3 Nov 2008 16:40]
Moshe Sharon
[3 Nov 2008 16:51]
MySQL Verification Team
Thank you for the bug report. Your server version is quite older, could you please test with latest released version and inform the results?. Thanks in advance.
[9 Nov 2008 23:53]
Moshe Sharon
Hi We upgraded few days ago. and rebooted the server. but still today I had the same problem 2b87c3e82000-2b87c3e8c000 r-xp 00000000 fd:00 2523164 /lib64/libnss_files-2.5.so 2b87c3e8c000-2b87c408b000 ---p 0000a000 fd:00 2523164 /lib64/libnss_files-2.5.so 2b87c408b000-2b87c408c000 r--p 00009000 fd:00 2523164 /lib64/libnss_files-2.5.so 2b87c408c000-2b87c408d000 rw-p 0000a000 fd:00 2523164 /lib64/libnss_files-2.5.so 2b87c408d000-2b87df183000 rw-p 2b87c408d000 00:00 0 7fffe6c82000-7fffe6c98000 rw-p 7fffe6c82000 00:00 0 [stack] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso] 081110 1:07:01 - mysqld got signal 6; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=402653184 read_buffer_size=2093056 max_used_connections=367 max_connections=4000 threads_connected=86 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 16761184 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2aaab818aac0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x465b2fb0, backtrace may not be correct. Stack range sanity check OK, backtrace follows: (nil) Stack trace seems successful - bottom reached Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Reso lved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x148a6d40 = set autocommit=1 thd->thread_id=64686511 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. 081110 01:37:20 mysqld started 081110 1:37:20 [Note] /usr/libexec/mysqld: ready for connections. Version: '5.0.67-community' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution only killing all processes and restarting the server manually helped. has anybody has an idea how to work this out. Using mysql-debug is not an options since this is production enviroments and gets around 1,000,000 querys every 5 - 10 minutes. mysql-debug won't be able to respond to large amount Thanks in advance
[13 Nov 2008 11:49]
Moshe Sharon
we are now experiencing crach on daily basis ( on the latest version it was 2 - 3 days) please find below the error. Version: '5.0.67-community' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition (GPL) 081113 6:00:01 [ERROR] /usr/sbin/mysqld: Table './monitor/service_details_members' is marked as crashed and should be repaired 081113 6:00:01 [Warning] Checking table: './monitor/service_details_members' 081113 6:00:01 [ERROR] /usr/sbin/mysqld: Table './monitor/service_members_contactgroups' is marked as crashed and should be repaire d 081113 6:00:01 [Warning] Checking table: './monitor/service_members_contactgroups' *** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption: 0x00002aaad4243500 *** ======= Backtrace: ========= /lib64/libc.so.6[0x34c2c71d21] /lib64/libc.so.6(__libc_malloc+0x7d)[0x34c2c72efd] /usr/sbin/mysqld(my_malloc+0x32)[0x86bca2] /usr/sbin/mysqld(_ZN6String10real_allocEj+0x35)[0x5acc05] /usr/sbin/mysqld(handle_one_connection+0x745)[0x5cc405] /lib64/libpthread.so.0[0x34c3406307] /lib64/libc.so.6(clone+0x6d)[0x34c2cd1ded] ======= Memory map: ======== 00400000-00b38000 r-xp 00000000 fd:00 12727679 /usr/sbin/mysqld 00d37000-00daf000 rw-p 00737000 fd:00 12727679 /usr/sbin/mysqld 00daf000-00dc4000 rw-p 00daf000 00:00 0 06ec6000-0837f000 rw-p 06ec6000 00:00 0 4067c000-4067d000 ---p 4067c000 00:00 0 4067d000-406bd000 rw-p 4067d000 00:00 0 406fe000-406ff000 ---p 406fe000 00:00 0 406ff000-4073f000 rw-p 406ff000 00:00 0 4073f000-40740000 ---p 4073f000 00:00 0 40740000-40780000 rw-p 40740000 00:00 0 43fda000-43fdb000 ---p 43fda000 00:00 0 43fdb000-4401b000 rw-p 43fdb000 00:00 0 4401b000-4401c000 ---p 4401b000 00:00 0 4401c000-4405c000 rw-p 4401c000 00:00 0 440de000-440df000 ---p 440de000 00:00 0 440df000-4411f000 rw-p 440df000 00:00 0 4411f000-44120000 ---p 4411f000 00:00 0 44120000-44160000 rw-p 44120000 00:00 0 44160000-44161000 ---p 44160000 00:00 0 44161000-441a1000 rw-p 44161000 00:00 0 441a1000-441a2000 ---p 441a1000 00:00 0 skipping .... if needed please let me know 47018000-47019000 ---p 47018000 00:00 0 47019000-47059000 rw-p 47019000 00:00 0 34c2800000-34c281a000 r-xp 00000000 fd:00 2523420 /lib64/ld-2.5.so 34c2a1a000-34c2a1b000 r--p 0001a000 fd:00 2523420 /lib64/ld-2.5.so 34c2a1b000-34c2a1c000 rw-p 0001b000 fd:00 2523420 /lib64/ld-2.5.so 2aaacc9f4000-2aaad0000000 ---p 2aaacc9f4000 00:00 0 2aaad0000000-2aaad0b8f000 rw-p 2aaad0000000 00:00 0 2aaad0b8f000-2aaad4000000 ---p 2aaad0b8f000 00:00 0 2aaad4000000-2aaad4c7c000 rw-p 2aaad4000000 00:00 0 2aaad4c7c000-2aaad8000000 ---p 2aaad4c7c000 00:00 0 2aab5000d000-2aab5000e000 rw-p 2aab5000d000 00:00 0 2aab50019000-2aab5001d000 rw-p 2aab50019000 00:00 0 7fff5aa87000-7fff5aa9d000 rw-p 7fff5aa87000 00:00 0 [stack] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso] 081113 12:19:01 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=402653184 read_buffer_size=2097152 max_used_connections=352 max_connections=2000 threads_connected=73 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 8585216 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2aaad039c260 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x447b7fd0, backtrace may not be correct. Stack range sanity check OK, backtrace follows: (nil) Stack trace seems successful - bottom reached Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Reso lved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at (nil) is invalid pointer thd->thread_id=14931985 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 081113 13:30:02 [Note] /usr/sbin/mysqld: Normal shutdown this is a disaster since its working production enviroment.
[21 Nov 2008 7:09]
Sveta Smirnova
Thank you for the feedback. Error log shows different queries when server fails, additionally in fails in malloc. So there still possibility problem can be because out of memory. To check this, please, collect vmstat output (vmstat 10 SOME_LARGE_COUNT) in time when crash occurs and check system logs.
[22 Dec 2008 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
