*** glibc detected *** /usr/sbin/mysqld: free(): invalid next size (normal): 0x00002aace40d8e10 *** ======= Backtrace: ========= /lib64/libc.so.6[0x2adebad13d7f] /lib64/libc.so.6(cfree+0x4b)[0x2adebad141db] /usr/sbin/mysqld(delete_dynamic+0x25)[0x84ab25] /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECTD0Ev+0x24)[0x68a5e4] /usr/sbin/mysqld(_ZN10SQL_SELECT7cleanupEv+0x46)[0x689ec6] /usr/sbin/mysqld(_ZN10SQL_SELECTD1Ev+0x9)[0x689f09] /usr/sbin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyyb+0x9ef)[0x655a3f] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x474a)[0x5dc15a] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x1f3)[0x5ddd13] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x53a)[0x5de32a] /usr/sbin/mysqld(_Z10do_commandP3THD+0xde)[0x5df46e] /usr/sbin/mysqld(handle_one_connection+0x661)[0x5d2201] /lib64/libpthread.so.0[0x2adeba1b677d] /lib64/libc.so.6(clone+0x6d)[0x2adebad7625d] ======= Memory map: ======== 00400000-00a56000 r-xp 00000000 fd:03 577965 /usr/sbin/mysqld 00b55000-00c8a000 rw-p 00655000 fd:03 577965 /usr/sbin/mysqld 00c8a000-00ca2000 rw-p 00c8a000 00:00 0 00d89000-00e12000 rw-p 00789000 fd:03 577965 /usr/sbin/mysqld 1769e000-2ac28000 rw-p 1769e000 00:00 0 [heap] 40877000-40878000 ---p 40877000 00:00 0 40878000-41278000 rw-p 40878000 00:00 0 41278000-41279000 ---p 41278000 00:00 0 41279000-412a9000 rw-p 41279000 00:00 0 41950000-41951000 ---p 41950000 00:00 0 41951000-41981000 rw-p 41951000 00:00 0 41981000-41982000 ---p 41981000 00:00 0 41982000-419b2000 rw-p 41982000 00:00 0 419b2000-419b3000 ---p 419b2000 00:00 0 419b3000-419e3000 rw-p 419b3000 00:00 0 419e3000-419e4000 ---p 419e3000 00:00 0 419e4000-41a14000 rw-p 419e4000 00:00 0 41a14000-41a15000 ---p 41a14000 00:00 0 41a15000-41a45000 rw-p 41a15000 00:00 0 41a45000-41a46000 ---p 41a45000 00:00 0 41a46000-41a76000 rw-p 41a46000 00:00 0 41a76000-41a77000 ---p 41a76000 00:00 0 41a77000-41aa7000 rw-p 41a77000 00:00 0 41aa7000-41aa8000 ---p 41aa7000 00:00 0 41aa8000-41ad8000 rw-p 41aa8000 00:00 0 41ad8000-41ad9000 ---p 41ad8000 00:00 0 41ad9000-41b09000 rw-p 41ad9000 00:00 0 41b09000-41b0a000 ---p 41b09000 00:00 0 41b0a000-41b3a000 rw-p 41b0a000 00:00 0 41b50000-41b51000 ---p 41b50000 00:00 0 41b51000-41b81000 rw-p 41b51000 00:00 0 41ce4000-41ce5000 ---p 41ce4000 00:00 0 41ce5000-426e5000 rw-p 41ce5000 00:00 0 426e5000-426e6000 ---p 426e5000 00:00 0 426e6000-430e6000 rw-p 426e6000 00:00 0 430e6000-430e7000 ---p 430e6000 00:00 0 430e7000-43ae7000 rw-p 430e7000 00:00 0 43ae7000-43ae8000 ---p 43ae7000 00:00 0 43ae8000-444e8000 rw-p 43ae8000 00:00 0 444e8000-444e9000 ---p 444e8000 00:00 0 444e9000-44ee9000 rw-p 444e9000 00:00 0 44ee9000-44eea000 ---p 44ee9000 00:00 0 44eea000-458ea000 rw-p 44eea000 00:00 0 458ea000-458eb000 ---p 458ea000 00:00 0 458eb000-462eb000 rw-p 458eb000 00:00 0 462eb000-462ec000 ---p 462eb000 00:00 0 462ec000-46cec000 rw-p 462ec000 00:00 0 46cec000-46ced000 ---p 46cec000 00:00 0 46ced000-46d1d000 rw-p 46ced000 00:00 0 46d1d000-46d1e000 ---p 46d1d000 00:00 0 46d1e000-46d4e000 rw-p 46d1e000 00:00 0 46d4e000-46d4f000 ---p 46d4e000 00:00 0 46d4f000-46d7f000 rw-p 46d4f000 00:00 0 46d7f000-46d80000 ---p 46d7f000 00:00 0 46d80000-46db0000 rw-p 46d80000 00:00 0 46db0000-46db1000 ---p 46db0000 00:00 0 46db1000-46de1000 rw-p 46db1000 00:00 0 46de1000-46de2000 ---p 46de1000 00:00 0 46de2000-46e12000 rw-p 46de2000 00:00 0 46e12000-46e13000 ---p 46e12000 00:00 0 46e13000-46e43000 rw-p 46e13000 00:00 0 46e43000-46e44000 ---p 46e43000 00:00 0 46e44000-46e74000 rw-p 46e44000 00:00 0 46e74000-46e75000 ---p 46e74000 00:00 0 46e75000-46ea5000 rw-p 46e75000 00:00 0 46ea5000-46ea6000 ---p 46ea5000 00:00 0 46ea6000-46ed6000 rw-p 46ea6000 00:00 0 46ed6000-46ed7000 ---p 46ed6000 00:00 0 46ed7000-46f07000 rw-p 46ed7000 00:00 0 46f07000-46f08000 ---p 46f07000 00:00 0 46f08000-46f38000 rw-p 46f08000 00:00 0 46f38000-46f39000 ---p 46f38000 00:00 0 46f39000-46f69000 rw-p 46f39000 00:00 0 46f69000-46f6a000 ---p 46f69000 00:00 0 46f6a000-46f9a000 rw-p 46f6a000 00:00 0 46f9a000-46f9b000 ---p 46f9a000 00:00 0 46f9b000-46fcb000 rw-p 46f9b000 00:00 0 46fcb000-46fcc000 ---p 46fcb000 00:00 0 46fcc000-46ffc000 rw-p 46fcc000 00:00 0 46ffc000-46ffd000 ---p 46ffc000 00:00 0 46ffd000-4702d000 rw-p 46ffd000 00:00 0 4702d000-4702e000 ---p 4702d000 00:00 0 4702e000-4705e000 rw-p 4702e000 00:00 0 4705e000-4705f000 ---p 4705e000 00:00 0 4705f000-4708f000 rw-p 4705f000 00:00 0 4708f000-47090000 ---p 4708f000 00:00 0 47090000-470c0000 rw-p 47090000 00:00 0 470c0000-470c1000 ---p 470c0000 00:00 0 470c1000-470f1000 rw-p 470c1000 00:00 0 470f1000-470f2000 ---p 470f1000 00:00 0 470f2000-47122000 rw-p 470f2000 00:00 0 47122000-47123000 ---p 47122000 00:00 0 47123000-47153000 rw-p 47123000 00:00 0 47153000-47154000 ---p 47153000 00:00 0 47154000-47184000 rw-p 47154000 00:00 0 47184000-47185000 ---p 47184000 00:00 0 47185000-471b5000 rw-p 47185000 00:00 0 471b5000-471b6000 ---p 471b5000 00:00 0 471b6000-471e6000 rw-p 471b6000 00:00 0 471e6000-471e7000 ---p 471e6000 00:00 0 471e7000-47217000 rw-p 471e7000 00:00 0 47217000-47218000 ---p 47217000 00:00 0 47218000-47248000 rw-p 47218000 00:00 0 47248000-47249000 ---p 47248000 00:00 0 47249000-47279000 rw-p 47249000 00:00 0 47279000-4727a000 ---p 47279000 00:00 0 4727a000-472aa000 rw-p 4727a000 00:00 0 472aa000-472ab000 ---p 472aa000 00:00 0 472ab000-472db000 rw-p 472ab000 00:00 0 472db000-472dc000 ---p 472db000 00:00 0 472dc000-4730c000 rw-p 472dc000 00:00 0 4730c000-4730d000 ---p 4730c000 00:00 0 4730d000-4733d000 rw-p 4730d000 00:00 0 4733d000-4733e000 ---p 4733d000 00:00 0 4733e000-4736e000 rw-p 4733e000 00:00 0 4736e000-4736f000 ---p 4736e000 00:00 0 4736f000-4739f000 rw-p 4736f000 00:00 0 4739f000-473a0000 ---p 4739f000 00:00 0 473a0000-473d0000 rw-p 473a0000 00:00 0 473d0000-473d1000 ---p 473d0000 00:00 0 473d1000-47401000 rw-p 473d1000 00:00 0 47401000-47402000 ---p 47401000 00:00 0 47402000-47432000 rw-p 47402000 00:00 0 47432000-47433000 ---p 47432000 00:00 0 47433000-47463000 rw-p 47433000 00:00 0 35f6800000-35f6802000 r-xp 00000000 fd:00 161333 /lib64/libkeyutils-1.2.so 35f6802000-35f6a01000 ---p 00002000 fd:00 161333 /lib64/libkeyutils-1.2.so 35f6a01000-35f6a02000 rw-p 00001000 fd:00 161333 /lib64/libkeyutils-1.2.so 39db800000-39db83b000 r-xp 00000000 fd:00 161323 /lib64/libsepol.so.1 39db83b000-39dba3b000 ---p 0003b000 fd:00 161323 /lib64/libsepol.so.1 39dba3b000-39dba3c000 rw-p 0003b000 fd:00 161323 /lib64/libsepol.so.1 39dba3c000-39dba46000 rw-p 39dba3c000 00:00 0 2aaaaaaab000-2aaaaaaac000 rw-p 2aaaaaaab000 00:00 0 2aaaaaabc000-2aaaaaac9000 r-xp 00000000 fd:00 227875 /lib64/libgcc_s-4.1.2-20080825.so.1 2aaaaaac9000-2aaaaacc9000 ---p 0000d000 fd:00 227875 /lib64/libgcc_s-4.1.2-20080825.so.1 2aaaaacc9000-2aaaaacca000 rw-p 0000d000 fd:00 227875 /lib64/libgcc_s-4.1.2-20080825.so.1 2aaaaacca000-2aaaaad12000 rw-p 2aaaaacca000 00:00 0 2aaaaad23000-2aaaaad2d000 r-xp 00000000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaad2d000-2aaaaaf2c000 ---p 0000a000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaaf2c000-2aaaaaf2d000 r--p 00009000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaaf2d000-2aaaaaf2e000 rw-p 0000a000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaaf2e000-2aaaab20b000 r-xp 00000000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab20b000-2aaaab40a000 ---p 002dd000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab40a000-2aaaab435000 rw-p 002dc000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab435000-2aaaab445000 rw-p 2aaaab435000 00:00 0 2aaaab445000-2aaaab447000 r-xp 00000000 fd:00 161291 /lib64/libcom_err.so.2.1 2aaaab447000-2aaaab646000 ---p 00002000 fd:00 161291 /lib64/libcom_err.so.2.1 2aaaab646000-2aaaab647000 rw-p 00001000 fd:00 161291 /lib64/libcom_err.so.2.1 2aaaab647000-2aaaab65c000 r-xp 00000000 fd:00 225869 /lib64/libselinux.so.1 2aaaab65c000-2aaaab85c000 ---p 00015000 fd:00 225869 /lib64/libselinux.so.1 2aaaab85c000-2aaaab85e000 rw-p 00015000 fd:00 225869 /lib64/libselinux.so.1 2aaaab85e000-2aaaab85f000 rw-p 2aaaab85e000 00:00 0 2aaaab85f000-2aaaab870000 r-xp 00000000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaab870000-2aaaaba70000 ---p 00011000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaaba70000-2aaaaba71000 r--p 00011000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaaba71000-2aaaaba72000 rw-p 00012000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaaba72000-2aaaafa91000 rw-p 2aaaaba72000 00:00 0 2aaaafa92000-2aace0e98000 rw-p 2aaaafa92000 00:00 0 2aace4000000-2aace4788000 rw-p 2aace4000000 00:00 0 2aace4788000-2aace8000000 ---p 2aace4788000 00:00 0 2aace97a6000-2aacebb70000 rw-p 2aace97a6000 00:00 0 2aacec000000-2aacec320000 rw-p 2aacec000000 00:00 0 2aacec320000-2aacf0000000 ---p 2aacec320000 00:00 0 2adeb9f92000-2adeb9fae000 r-xp 00000000 fd:00 225840 /lib64/ld-2.5.so 2adeb9fae000-2adeb9fb0000 rw-p 2adeb9fae000 00:00 0 2adeba1ae000-2adeba1af000 r--p 0001c000 fd:00 225840 /lib64/ld-2.5.so 2adeba1af000-2adeba1b0000 rw-p 0001d000 fd:00 225840 /lib64/ld-2.5.so 2adeba1b0000-2adeba1c6000 r-xp 00000000 fd:00 161314 /lib64/libpthread-2.5.so 2adeba1c6000-2adeba3c5000 ---p 00016000 fd:00 161314 /lib64/libpthread-2.5.so 2adeba3c5000-2adeba3c6000 r--p 00015000 fd:00 161314 /lib64/libpthread-2.5.so 2adeba3c6000-2adeba3c7000 rw-p 00016000 fd:00 161314 /lib64/libpthread-2.5.so 2adeba3c7000-2adeba3cb000 rw-p 2adeba3c7000 00:00 0 2adeba3cb000-2adeba3cd000 r-xp 00000000 fd:00 161298 /lib64/libdl-2.5.so 2adeba3cd000-2adeba5cd000 ---p 00002000 fd:00 161298 /lib64/libdl-2.5.so 2adeba5cd000-2adeba5ce000 r--p 00002000 fd:00 161298 /lib64/libdl-2.5.so 2adeba5ce000-2adeba5cf000 rw-p 00003000 fd:00 161298 /lib64/libdl-2.5.so 2adeba5cf000-2adeba5d8000 r-xp 00000000 fd:00 161294 /lib64/libcrypt-2.5.so 2adeba5d8000-2adeba7d7000 ---p 00009000 fd:00 161294 /lib64/libcrypt-2.5.so 2adeba7d7000-2adeba7d8000 r--p 00008000 fd:00 161294 /lib64/libcrypt-2.5.so 2adeba7d8000-2adeba7d9000 rw-p 00009000 fd:00 161294 /lib64/libcrypt-2.5.so 2adeba7d9000-2adeba808000 rw-p 2adeba7d9000 00:00 0 2adeba808000-2adeba81d000 r-xp 00000000 fd:00 161302 /lib64/libnsl-2.5.so 2adeba81d000-2adebaa1c000 ---p 00015000 fd:00 161302 /lib64/libnsl-2.5.so 2adebaa1c000-2adebaa1d000 r--p 00014000 fd:00 161302 /lib64/libnsl-2.5.so 2adebaa1d000-2adebaa1e000 rw-p 00015000 fd:00 161302 /lib64/libnsl-2.5.so 2adebaa1e000-2adebaa20000 rw-p 2adebaa1e000 00:00 0 2adebaa20000-2adebaaa2000 r-xp 00000000 fd:00 161300 /lib64/libm-2.5.so 2adebaaa2000-2adebaca1000 ---p 00082000 fd:00 161300 /lib64/libm-2.5.so 2adebaca1000-2adebaca2000 r--p 00081000 fd:00 161300 /lib64/libm-2.5.so 2adebaca2000-2adebaca3000 rw-p 00082000 fd:00 161300 /lib64/libm-2.5.so 2adebaca3000-2adebadf0000 r-xp 00000000 fd:00 161289 /lib64/libc-2.5.so 2adebadf0000-2adebaff0000 ---p 0014d000 fd:00 161289 /lib64/libc-2.5.so 2adebaff0000-2adebaff4000 r--p 0014d000 fd:00 161289 /lib64/libc-2.5.so 2adebaff4000-2adebaff5000 rw-p 00151000 fd:00 161289 /lib64/libc-2.5.so 2adebaff5000-2adebaffc000 rw-p 2adebaff5000 00:00 0 7ffff4ec3000-7ffff4ed8000 rw-p 7ffffffe9000 00:00 0 [stack] 7ffff4ffd000-7ffff5000000 r-xp 7ffff4ffd000 00:00 0 [vdso] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall] 06:02:11 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=50 max_threads=2000 thread_count=33 connection_count=33 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2332895 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x2aace4038030 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 4702bf28 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x85395e] /usr/sbin/mysqld(handle_fatal_signal+0x38c)[0x6ae7ec] /lib64/libpthread.so.0[0x2adeba1bebe0] /lib64/libc.so.6(gsignal+0x35)[0x2adebacd3285] /lib64/libc.so.6(abort+0x110)[0x2adebacd4d30] /lib64/libc.so.6[0x2adebad0c2bb] /lib64/libc.so.6[0x2adebad13d7f] /lib64/libc.so.6(cfree+0x4b)[0x2adebad141db] /usr/sbin/mysqld(delete_dynamic+0x25)[0x84ab25] /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECTD0Ev+0x24)[0x68a5e4] /usr/sbin/mysqld(_ZN10SQL_SELECT7cleanupEv+0x46)[0x689ec6] /usr/sbin/mysqld(_ZN10SQL_SELECTD1Ev+0x9)[0x689f09] /usr/sbin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyyb+0x9ef)[0x655a3f] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x474a)[0x5dc15a] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x1f3)[0x5ddd13] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x53a)[0x5de32a] /usr/sbin/mysqld(_Z10do_commandP3THD+0xde)[0x5df46e] /usr/sbin/mysqld(handle_one_connection+0x661)[0x5d2201] /lib64/libpthread.so.0[0x2adeba1b677d] /lib64/libc.so.6(clone+0x6d)[0x2adebad7625d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (29949970): DELETE FROM TAX_QUEUE WHERE ar_trans_id = 350822426 Connection ID (thread ID): 39288 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 120401 01:02:11 mysqld_safe Number of processes running now: 0 120401 01:02:11 mysqld_safe mysqld restarted 120401 1:02:11 [Note] Plugin 'FEDERATED' is disabled. 120401 1:02:12 InnoDB: Initializing buffer pool, size = 8.0G 120401 1:02:13 InnoDB: Completed initialization of buffer pool InnoDB: Log scan progressed past the checkpoint lsn 1833 810293315 120401 1:02:13 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 1833 815536128 InnoDB: Doing recovery: scanned up to log sequence number 1833 820779008 InnoDB: Doing recovery: scanned up to log sequence number 1833 826021888 InnoDB: Doing recovery: scanned up to log sequence number 1833 831264768 InnoDB: Doing recovery: scanned up to log sequence number 1833 836507648 InnoDB: Doing recovery: scanned up to log sequence number 1833 841750528 InnoDB: Doing recovery: scanned up to log sequence number 1833 846993408 InnoDB: Doing recovery: scanned up to log sequence number 1833 852236288 InnoDB: Doing recovery: scanned up to log sequence number 1833 857479168 InnoDB: Doing recovery: scanned up to log sequence number 1833 862722048 InnoDB: Doing recovery: scanned up to log sequence number 1833 867964928 InnoDB: Doing recovery: scanned up to log sequence number 1833 873207808 InnoDB: Doing recovery: scanned up to log sequence number 1833 878450688 InnoDB: Doing recovery: scanned up to log sequence number 1833 883693568 InnoDB: Doing recovery: scanned up to log sequence number 1833 888936448 InnoDB: Doing recovery: scanned up to log sequence number 1833 894179328 InnoDB: Doing recovery: scanned up to log sequence number 1833 899422208 InnoDB: Doing recovery: scanned up to log sequence number 1833 904665088 InnoDB: Doing recovery: scanned up to log sequence number 1833 909907968 InnoDB: Doing recovery: scanned up to log sequence number 1833 915150848 InnoDB: Doing recovery: scanned up to log sequence number 1833 920393728 InnoDB: Doing recovery: scanned up to log sequence number 1833 925636608 InnoDB: Doing recovery: scanned up to log sequence number 1833 930879488 InnoDB: Doing recovery: scanned up to log sequence number 1833 936122368 InnoDB: Doing recovery: scanned up to log sequence number 1833 941365248 InnoDB: Doing recovery: scanned up to log sequence number 1833 946608128 InnoDB: Doing recovery: scanned up to log sequence number 1833 951851008 InnoDB: Doing recovery: scanned up to log sequence number 1833 952903091 InnoDB: Transaction 1 3926416715 was in the XA prepared state. InnoDB: 6 transaction(s) which must be rolled back or cleaned up InnoDB: in total 9 row operations to undo InnoDB: Trx id counter is 1 3926417152 120401 1:02:40 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 136328950, file name ./mysql-bin.004165 InnoDB: Starting in background the rollback of uncommitted transactions 120401 1:04:20 InnoDB: Rolling back trx with id 1 3926416736, 1 rows to undo 120401 1:04:20 InnoDB: Started; log sequence number 1833 952903091 120401 1:04:20 [Note] Recovering after a crash using mysql-bin InnoDB: Rolling back of trx id 1 3926416736 completed 120401 1:04:20 InnoDB: Rolling back trx with id 1 3926416726, 1 rows to undo InnoDB: Rolling back of trx id 1 3926416726 completed 120401 1:04:20 InnoDB: Rolling back trx with id 1 3926416725, 3 rows to undo InnoDB: Rolling back of trx id 1 3926416725 completed 120401 1:04:20 InnoDB: Rolling back trx with id 1 3926416716, 1 rows to undo InnoDB: Rolling back of trx id 1 3926416716 completed 120401 1:04:20 InnoDB: Rolling back trx with id 1 3926416684, 3 rows to undo InnoDB: Rolling back of trx id 1 3926416684 completed 120401 1:04:20 InnoDB: Rollback of non-prepared transactions completed 120401 1:04:21 [Note] Starting crash recovery... 120401 1:04:21 InnoDB: Starting recovery for XA transactions... 120401 1:04:21 InnoDB: Transaction 1 3926416715 in prepared state after recovery 120401 1:04:21 InnoDB: Transaction contains changes to 1 rows 120401 1:04:21 InnoDB: 1 transactions in prepared state after recovery 120401 1:04:21 [Note] Found 1 prepared transaction(s) in InnoDB 120401 1:04:21 [Note] Crash recovery finished. 120401 1:04:21 [Note] Event Scheduler: Loaded 0 events 120401 1:04:21 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.1.62-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) 120401 1:09:21 [Warning] Aborted connection 1416 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets) 120401 1:19:11 [Warning] Aborted connection 4887 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets) 06:51:30 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=37 max_threads=2000 thread_count=22 connection_count=22 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2332895 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x2aace43221a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 120401 01:51:30 mysqld_safe Number of processes running now: 0 120401 01:51:30 mysqld_safe mysqld restarted 120401 1:51:30 [Note] Plugin 'FEDERATED' is disabled. 120401 1:51:30 InnoDB: Initializing buffer pool, size = 8.0G 120401 1:51:32 InnoDB: Completed initialization of buffer pool InnoDB: Log scan progressed past the checkpoint lsn 1833 852421236 120401 1:51:32 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 1833 857664000 InnoDB: Doing recovery: scanned up to log sequence number 1833 862906880 InnoDB: Doing recovery: scanned up to log sequence number 1833 868149760 InnoDB: Doing recovery: scanned up to log sequence number 1833 873392640 InnoDB: Doing recovery: scanned up to log sequence number 1833 878635520 InnoDB: Doing recovery: scanned up to log sequence number 1833 883878400 InnoDB: Doing recovery: scanned up to log sequence number 1833 889121280 InnoDB: Doing recovery: scanned up to log sequence number 1833 894364160 InnoDB: Doing recovery: scanned up to log sequence number 1833 899607040 InnoDB: Doing recovery: scanned up to log sequence number 1833 904849920 InnoDB: Doing recovery: scanned up to log sequence number 1833 910092800 InnoDB: Doing recovery: scanned up to log sequence number 1833 915335680 InnoDB: Doing recovery: scanned up to log sequence number 1833 920578560 InnoDB: Doing recovery: scanned up to log sequence number 1833 925821440 InnoDB: Doing recovery: scanned up to log sequence number 1833 931064320 InnoDB: Doing recovery: scanned up to log sequence number 1833 936307200 InnoDB: Doing recovery: scanned up to log sequence number 1833 941550080 InnoDB: Doing recovery: scanned up to log sequence number 1833 946792960 InnoDB: Doing recovery: scanned up to log sequence number 1833 952035840 InnoDB: Doing recovery: scanned up to log sequence number 1833 957278720 InnoDB: Doing recovery: scanned up to log sequence number 1833 962521600 InnoDB: Doing recovery: scanned up to log sequence number 1833 967764480 InnoDB: Doing recovery: scanned up to log sequence number 1833 973007360 InnoDB: Doing recovery: scanned up to log sequence number 1833 978250240 InnoDB: Doing recovery: scanned up to log sequence number 1833 983493120 InnoDB: Doing recovery: scanned up to log sequence number 1833 988736000 InnoDB: Doing recovery: scanned up to log sequence number 1833 993978880 InnoDB: Doing recovery: scanned up to log sequence number 1833 999221760 InnoDB: Doing recovery: scanned up to log sequence number 1833 1004464640 InnoDB: Doing recovery: scanned up to log sequence number 1833 1009707520 InnoDB: Doing recovery: scanned up to log sequence number 1833 1014950400 InnoDB: Doing recovery: scanned up to log sequence number 1833 1020193280 InnoDB: Doing recovery: scanned up to log sequence number 1833 1025436160 InnoDB: Doing recovery: scanned up to log sequence number 1833 1030679040 InnoDB: Doing recovery: scanned up to log sequence number 1833 1035921920 InnoDB: Doing recovery: scanned up to log sequence number 1833 1041164800 InnoDB: Doing recovery: scanned up to log sequence number 1833 1046407680 InnoDB: Doing recovery: scanned up to log sequence number 1833 1051650560 InnoDB: Doing recovery: scanned up to log sequence number 1833 1056893440 InnoDB: Doing recovery: scanned up to log sequence number 1833 1062136320 InnoDB: Doing recovery: scanned up to log sequence number 1833 1067379200 InnoDB: Doing recovery: scanned up to log sequence number 1833 1072622080 InnoDB: Doing recovery: scanned up to log sequence number 1833 1077864960 InnoDB: Doing recovery: scanned up to log sequence number 1833 1083107840 InnoDB: Doing recovery: scanned up to log sequence number 1833 1084067743 120401 1:52:21 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 108295521, file name ./mysql-bin.004166 120401 1:54:46 InnoDB: Started; log sequence number 1833 1084067743 120401 1:54:46 [Note] Recovering after a crash using mysql-bin 120401 1:54:47 [Note] Starting crash recovery... 120401 1:54:47 [Note] Crash recovery finished. 120401 1:54:47 [Note] Event Scheduler: Loaded 0 events 120401 1:54:47 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.1.62-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) 120401 1:59:03 [Warning] Aborted connection 29 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets) 120401 2:22:56 [Warning] Aborted connection 5043 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets) 120401 9:59:03 [Warning] Aborted connection 701 to db: 'unconnected' user: 'root' host: 'localhost' (Got timeout reading communication packets) *** glibc detected *** /usr/sbin/mysqld: free(): invalid next size (fast): 0x0000000017857ea0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x2af2ddd78d7f] /lib64/libc.so.6(cfree+0x4b)[0x2af2ddd791db] /usr/sbin/mysqld(close_cached_file+0x4c)[0x84311c] /usr/sbin/mysqld(_Z8filesortP3THDP8st_tableP13st_sort_fieldjP10SQL_SELECTybPy+0x8f2)[0x69eaa2] /usr/sbin/mysqld[0x633eb7] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x617)[0x63ca67] /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x5eb)[0x63eb3b] /usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x180)[0x63ee40] /usr/sbin/mysqld[0x5d37c1] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x14ec)[0x5d8efc] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x1f3)[0x5ddd13] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x53a)[0x5de32a] /usr/sbin/mysqld(_Z10do_commandP3THD+0xde)[0x5df46e] /usr/sbin/mysqld(handle_one_connection+0x661)[0x5d2201] /lib64/libpthread.so.0[0x2af2dd21b77d] /lib64/libc.so.6(clone+0x6d)[0x2af2ddddb25d] ======= Memory map: ======== 00400000-00a56000 r-xp 00000000 fd:03 577965 /usr/sbin/mysqld 00b55000-00c8a000 rw-p 00655000 fd:03 577965 /usr/sbin/mysqld 00c8a000-00ca2000 rw-p 00c8a000 00:00 0 00d89000-00e12000 rw-p 00789000 fd:03 577965 /usr/sbin/mysqld 04de5000-1a9ac000 rw-p 04de5000 00:00 0 [heap] 40e27000-40e28000 ---p 40e27000 00:00 0 40e28000-41828000 rw-p 40e28000 00:00 0 41828000-41829000 ---p 41828000 00:00 0 41829000-42229000 rw-p 41829000 00:00 0 42229000-4222a000 ---p 42229000 00:00 0 4222a000-42c2a000 rw-p 4222a000 00:00 0 42c2a000-42c2b000 ---p 42c2a000 00:00 0 42c2b000-4362b000 rw-p 42c2b000 00:00 0 4362b000-4362c000 ---p 4362b000 00:00 0 4362c000-4402c000 rw-p 4362c000 00:00 0 4402c000-4402d000 ---p 4402c000 00:00 0 4402d000-44a2d000 rw-p 4402d000 00:00 0 44a2d000-44a2e000 ---p 44a2d000 00:00 0 44a2e000-4542e000 rw-p 44a2e000 00:00 0 4542e000-4542f000 ---p 4542e000 00:00 0 4542f000-45e2f000 rw-p 4542f000 00:00 0 45e2f000-45e30000 ---p 45e2f000 00:00 0 45e30000-46830000 rw-p 45e30000 00:00 0 46830000-46831000 ---p 46830000 00:00 0 46831000-46871000 rw-p 46831000 00:00 0 46871000-46872000 ---p 46871000 00:00 0 46872000-468b2000 rw-p 46872000 00:00 0 468b2000-468b3000 ---p 468b2000 00:00 0 468b3000-468f3000 rw-p 468b3000 00:00 0 468f3000-468f4000 ---p 468f3000 00:00 0 468f4000-46934000 rw-p 468f4000 00:00 0 46934000-46935000 ---p 46934000 00:00 0 46935000-46975000 rw-p 46935000 00:00 0 46975000-46976000 ---p 46975000 00:00 0 46976000-469b6000 rw-p 46976000 00:00 0 469b6000-469b7000 ---p 469b6000 00:00 0 469b7000-469f7000 rw-p 469b7000 00:00 0 469f7000-469f8000 ---p 469f7000 00:00 0 469f8000-46a38000 rw-p 469f8000 00:00 0 46a38000-46a39000 ---p 46a38000 00:00 0 46a39000-46a79000 rw-p 46a39000 00:00 0 46a79000-46a7a000 ---p 46a79000 00:00 0 46a7a000-46aba000 rw-p 46a7a000 00:00 0 46aba000-46abb000 ---p 46aba000 00:00 0 46abb000-46afb000 rw-p 46abb000 00:00 0 46afb000-46afc000 ---p 46afb000 00:00 0 46afc000-46b3c000 rw-p 46afc000 00:00 0 46b3c000-46b3d000 ---p 46b3c000 00:00 0 46b3d000-46b7d000 rw-p 46b3d000 00:00 0 46b7d000-46b7e000 ---p 46b7d000 00:00 0 46b7e000-46bbe000 rw-p 46b7e000 00:00 0 46bbe000-46bbf000 ---p 46bbe000 00:00 0 46bbf000-46bff000 rw-p 46bbf000 00:00 0 46bff000-46c00000 ---p 46bff000 00:00 0 46c00000-46c40000 rw-p 46c00000 00:00 0 46c40000-46c41000 ---p 46c40000 00:00 0 46c41000-46c81000 rw-p 46c41000 00:00 0 46c81000-46c82000 ---p 46c81000 00:00 0 46c82000-46cc2000 rw-p 46c82000 00:00 0 46cc2000-46cc3000 ---p 46cc2000 00:00 0 46cc3000-46d03000 rw-p 46cc3000 00:00 0 46d03000-46d04000 ---p 46d03000 00:00 0 46d04000-46d44000 rw-p 46d04000 00:00 0 46d44000-46d45000 ---p 46d44000 00:00 0 46d45000-46d85000 rw-p 46d45000 00:00 0 46d85000-46d86000 ---p 46d85000 00:00 0 46d86000-46dc6000 rw-p 46d86000 00:00 0 46dc6000-46dc7000 ---p 46dc6000 00:00 0 46dc7000-46e07000 rw-p 46dc7000 00:00 0 46e07000-46e08000 ---p 46e07000 00:00 0 46e08000-46e48000 rw-p 46e08000 00:00 0 46e48000-46e49000 ---p 46e48000 00:00 0 46e49000-46e89000 rw-p 46e49000 00:00 0 46e89000-46e8a000 ---p 46e89000 00:00 0 46e8a000-46eca000 rw-p 46e8a000 00:00 0 46eca000-46ecb000 ---p 46eca000 00:00 0 46ecb000-46f0b000 rw-p 46ecb000 00:00 0 46f0b000-46f0c000 ---p 46f0b000 00:00 0 46f0c000-46f4c000 rw-p 46f0c000 00:00 0 46f4c000-46f4d000 ---p 46f4c000 00:00 0 46f4d000-46f8d000 rw-p 46f4d000 00:00 0 46f8d000-46f8e000 ---p 46f8d000 00:00 0 46f8e000-46fce000 rw-p 46f8e000 00:00 0 46fce000-46fcf000 ---p 46fce000 00:00 0 46fcf000-4700f000 rw-p 46fcf000 00:00 0 4700f000-47010000 ---p 4700f000 00:00 0 47010000-47050000 rw-p 47010000 00:00 0 47050000-47051000 ---p 47050000 00:00 0 47051000-47091000 rw-p 47051000 00:00 0 47091000-47092000 ---p 47091000 00:00 0 47092000-470d2000 rw-p 47092000 00:00 0 470d2000-470d3000 ---p 470d2000 00:00 0 470d3000-47113000 rw-p 470d3000 00:00 0 47113000-47114000 ---p 47113000 00:00 0 47114000-47154000 rw-p 47114000 00:00 0 47154000-47155000 ---p 47154000 00:00 0 47155000-47195000 rw-p 47155000 00:00 0 47195000-47196000 ---p 47195000 00:00 0 47196000-471d6000 rw-p 47196000 00:00 0 471d6000-471d7000 ---p 471d6000 00:00 0 471d7000-47217000 rw-p 471d7000 00:00 0 47217000-47218000 ---p 47217000 00:00 0 47218000-47258000 rw-p 47218000 00:00 0 47258000-47259000 ---p 47258000 00:00 0 47259000-47299000 rw-p 47259000 00:00 0 47299000-4729a000 ---p 47299000 00:00 0 4729a000-472da000 rw-p 4729a000 00:00 0 472da000-472db000 ---p 472da000 00:00 0 472db000-4731b000 rw-p 472db000 00:00 0 4731b000-4731c000 ---p 4731b000 00:00 0 4731c000-4735c000 rw-p 4731c000 00:00 0 4735c000-4735d000 ---p 4735c000 00:00 0 4735d000-4739d000 rw-p 4735d000 00:00 0 4739d000-4739e000 ---p 4739d000 00:00 0 4739e000-473de000 rw-p 4739e000 00:00 0 473de000-473df000 ---p 473de000 00:00 0 473df000-4741f000 rw-p 473df000 00:00 0 4741f000-47420000 ---p 4741f000 00:00 0 47420000-47460000 rw-p 47420000 00:00 0 47460000-47461000 ---p 47460000 00:00 0 47461000-474a1000 rw-p 47461000 00:00 0 474a1000-474a2000 ---p 474a1000 00:00 0 474a2000-474e2000 rw-p 474a2000 00:00 0 474e2000-474e3000 ---p 474e2000 00:00 0 474e3000-47523000 rw-p 474e3000 00:00 0 47523000-47524000 ---p 47523000 00:00 0 47524000-47564000 rw-p 47524000 00:00 0 47564000-47565000 ---p 47564000 00:00 0 47565000-475a5000 rw-p 47565000 00:00 0 475a5000-475a6000 ---p 475a5000 00:00 0 475a6000-475e6000 rw-p 475a6000 00:00 0 475e6000-475e7000 ---p 475e6000 00:00 0 475e7000-47627000 rw-p 475e7000 00:00 0 47627000-47628000 ---p 47627000 00:00 0 47628000-47668000 rw-p 47628000 00:00 0 47668000-47669000 ---p 47668000 00:00 0 47669000-476a9000 rw-p 47669000 00:00 0 476a9000-476aa000 ---p 476a9000 00:00 0 476aa000-476ea000 rw-p 476aa000 00:00 0 476ea000-476eb000 ---p 476ea000 00:00 0 476eb000-4772b000 rw-p 476eb000 00:00 0 4772b000-4772c000 ---p 4772b000 00:00 0 4772c000-4776c000 rw-p 4772c000 00:00 0 4776c000-4776d000 ---p 4776c000 00:00 0 4776d000-477ad000 rw-p 4776d000 00:00 0 477ad000-477ae000 ---p 477ad000 00:00 0 477ae000-477ee000 rw-p 477ae000 00:00 0 477ee000-477ef000 ---p 477ee000 00:00 0 477ef000-4782f000 rw-p 477ef000 00:00 0 4782f000-47830000 ---p 4782f000 00:00 0 47830000-47870000 rw-p 47830000 00:00 0 47870000-47871000 ---p 47870000 00:00 0 47871000-478b1000 rw-p 47871000 00:00 0 478b1000-478b2000 ---p 478b1000 00:00 0 478b2000-478f2000 rw-p 478b2000 00:00 0 478f2000-478f3000 ---p 478f2000 00:00 0 478f3000-47933000 rw-p 478f3000 00:00 0 47933000-47934000 ---p 47933000 00:00 0 47934000-47974000 rw-p 47934000 00:00 0 35f6800000-35f6802000 r-xp 00000000 fd:00 161333 /lib64/libkeyutils-1.2.so 35f6802000-35f6a01000 ---p 00002000 fd:00 161333 /lib64/libkeyutils-1.2.so 35f6a01000-35f6a02000 rw-p 00001000 fd:00 161333 /lib64/libkeyutils-1.2.so 39db800000-39db83b000 r-xp 00000000 fd:00 161323 /lib64/libsepol.so.1 39db83b000-39dba3b000 ---p 0003b000 fd:00 161323 /lib64/libsepol.so.1 39dba3b000-39dba3c000 rw-p 0003b000 fd:00 161323 /lib64/libsepol.so.1 39dba3c000-39dba46000 rw-p 39dba3c000 00:00 0 2aaaaaabc000-2aaaaaac9000 r-xp 00000000 fd:00 227875 /lib64/libgcc_s-4.1.2-20080825.so.1.#prelink#.K3R3Kp (deleted) 2aaaaaac9000-2aaaaacc9000 ---p 0000d000 fd:00 227875 /lib64/libgcc_s-4.1.2-20080825.so.1.#prelink#.K3R3Kp (deleted) 2aaaaacc9000-2aaaaacca000 rw-p 0000d000 fd:00 227875 /lib64/libgcc_s-4.1.2-20080825.so.1.#prelink#.K3R3Kp (deleted) 2aaaaacca000-2aaaaad12000 rw-p 2aaaaacca000 00:00 0 2aaaaad23000-2aaaaad2d000 r-xp 00000000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaad2d000-2aaaaaf2c000 ---p 0000a000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaaf2c000-2aaaaaf2d000 r--p 00009000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaaf2d000-2aaaaaf2e000 rw-p 0000a000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaaf2e000-2aaaab20b000 r-xp 00000000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab20b000-2aaaab40a000 ---p 002dd000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab40a000-2aaaab435000 rw-p 002dc000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab435000-2aaaab445000 rw-p 2aaaab435000 00:00 0 2aaaab445000-2aaaab447000 r-xp 00000000 fd:00 161291 /lib64/libcom_err.so.2.1 2aaaab447000-2aaaab646000 ---p 00002000 fd:00 161291 /lib64/libcom_err.so.2.1 2aaaab646000-2aaaab647000 rw-p 00001000 fd:00 161291 /lib64/libcom_err.so.2.1 2aaaab647000-2aaaab65c000 r-xp 00000000 fd:00 225869 /lib64/libselinux.so.1 2aaaab65c000-2aaaab85c000 ---p 00015000 fd:00 225869 /lib64/libselinux.so.1 2aaaab85c000-2aaaab85e000 rw-p 00015000 fd:00 225869 /lib64/libselinux.so.1 2aaaab85e000-2aaaab85f000 rw-p 2aaaab85e000 00:00 0 2aaaab85f000-2aaaab870000 r-xp 00000000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaab870000-2aaaaba70000 ---p 00011000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaaba70000-2aaaaba71000 r--p 00011000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaaba71000-2aaaaba72000 rw-p 00012000 fd:00 161324 /lib64/libresolv-2.5.so 2aaaaba72000-2aace0e98000 rw-p 2aaaaba72000 00:00 0 2aace4000000-2aace65f2000 rw-p 2aace4000000 00:00 0 2aace65f2000-2aace8000000 ---p 2aace65f2000 00:00 0 2aace97a6000-2aacebb70000 rw-p 2aace97a6000 00:00 0 2aacec000000-2aacecae8000 rw-p 2aacec000000 00:00 0 2aacecae8000-2aacf0000000 ---p 2aacecae8000 00:00 0 2af2dcff7000-2af2dd013000 r-xp 00000000 fd:00 225840 /lib64/ld-2.5.so 2af2dd013000-2af2dd015000 rw-p 2af2dd013000 00:00 0 2af2dd213000-2af2dd214000 r--p 0001c000 fd:00 225840 /lib64/ld-2.5.so 2af2dd214000-2af2dd215000 rw-p 0001d000 fd:00 225840 /lib64/ld-2.5.so 2af2dd215000-2af2dd22b000 r-xp 00000000 fd:00 161314 /lib64/libpthread-2.5.so 2af2dd22b000-2af2dd42a000 ---p 00016000 fd:00 161314 /lib64/libpthread-2.5.so 2af2dd42a000-2af2dd42b000 r--p 00015000 fd:00 161314 /lib64/libpthread-2.5.so 2af2dd42b000-2af2dd42c000 rw-p 00016000 fd:00 161314 /lib64/libpthread-2.5.so 2af2dd42c000-2af2dd430000 rw-p 2af2dd42c000 00:00 0 2af2dd430000-2af2dd432000 r-xp 00000000 fd:00 161298 /lib64/libdl-2.5.so 2af2dd432000-2af2dd632000 ---p 00002000 fd:00 161298 /lib64/libdl-2.5.so 2af2dd632000-2af2dd633000 r--p 00002000 fd:00 161298 /lib64/libdl-2.5.so 2af2dd633000-2af2dd634000 rw-p 00003000 fd:00 161298 /lib64/libdl-2.5.so 2af2dd634000-2af2dd63d000 r-xp 00000000 fd:00 161294 /lib64/libcrypt-2.5.so 2af2dd63d000-2af2dd83c000 ---p 00009000 fd:00 161294 /lib64/libcrypt-2.5.so 2af2dd83c000-2af2dd83d000 r--p 00008000 fd:00 161294 /lib64/libcrypt-2.5.so 2af2dd83d000-2af2dd83e000 rw-p 00009000 fd:00 161294 /lib64/libcrypt-2.5.so 2af2dd83e000-2af2dd86d000 rw-p 2af2dd83e000 00:00 0 2af2dd86d000-2af2dd882000 r-xp 00000000 fd:00 161302 /lib64/libnsl-2.5.so 2af2dd882000-2af2dda81000 ---p 00015000 fd:00 161302 /lib64/libnsl-2.5.so 2af2dda81000-2af2dda82000 r--p 00014000 fd:00 161302 /lib64/libnsl-2.5.so 2af2dda82000-2af2dda83000 rw-p 00015000 fd:00 161302 /lib64/libnsl-2.5.so 2af2dda83000-2af2dda85000 rw-p 2af2dda83000 00:00 0 2af2dda85000-2af2ddb07000 r-xp 00000000 fd:00 161300 /lib64/libm-2.5.so 2af2ddb07000-2af2ddd06000 ---p 00082000 fd:00 161300 /lib64/libm-2.5.so 2af2ddd06000-2af2ddd07000 r--p 00081000 fd:00 161300 /lib64/libm-2.5.so 2af2ddd07000-2af2ddd08000 rw-p 00082000 fd:00 161300 /lib64/libm-2.5.so 2af2ddd08000-2af2dde55000 r-xp 00000000 fd:00 161289 /lib64/libc-2.5.so 2af2dde55000-2af2de055000 ---p 0014d000 fd:00 161289 /lib64/libc-2.5.so 2af2de055000-2af2de059000 r--p 0014d000 fd:00 161289 /lib64/libc-2.5.so 2af2de059000-2af2de05a000 rw-p 00151000 fd:00 161289 /lib64/libc-2.5.so 2af2de05a000-2af2de061000 rw-p 2af2de05a000 00:00 0 7fff0373c000-7fff03751000 rw-p 7ffffffe9000 00:00 0 [stack] 7fff037fd000-7fff03800000 r-xp 7fff037fd000 00:00 0 [vdso] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall] 15:17:17 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=67 max_threads=2000 thread_count=28 connection_count=28 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2332895 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x2aacec060a20 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 478aff28 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x85395e] /usr/sbin/mysqld(handle_fatal_signal+0x38c)[0x6ae7ec] /lib64/libpthread.so.0[0x2af2dd223be0] /lib64/libc.so.6(gsignal+0x35)[0x2af2ddd38285] /lib64/libc.so.6(abort+0x110)[0x2af2ddd39d30] /lib64/libc.so.6[0x2af2ddd712bb] /lib64/libc.so.6[0x2af2ddd78d7f] /lib64/libc.so.6(cfree+0x4b)[0x2af2ddd791db] /usr/sbin/mysqld(close_cached_file+0x4c)[0x84311c] /usr/sbin/mysqld(_Z8filesortP3THDP8st_tableP13st_sort_fieldjP10SQL_SELECTybPy+0x8f2)[0x69eaa2] /usr/sbin/mysqld[0x633eb7] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x617)[0x63ca67] /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x5eb)[0x63eb3b] /usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x180)[0x63ee40] /usr/sbin/mysqld[0x5d37c1] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x14ec)[0x5d8efc] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x1f3)[0x5ddd13] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x53a)[0x5de32a] /usr/sbin/mysqld(_Z10do_commandP3THD+0xde)[0x5df46e] /usr/sbin/mysqld(handle_one_connection+0x661)[0x5d2201] /lib64/libpthread.so.0[0x2af2dd21b77d] /lib64/libc.so.6(clone+0x6d)[0x2af2ddddb25d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (1717a0a0): SELECT I.invoice_number as sentinvoiceid, I.customer_id as customerid, I.invoice_date as invoicedate, I.paid_amount as amountpaid, I.due_date as duedate, I.payment_due as paymentdue, I.new_charges as newcharges FROM INVOICE I LEFT JOIN INVOICE_INFO II on II.invoice_info_id = I.invoice_info_id LEFT JOIN INVOICE_OUTPUT_METHOD IOM on IOM.invoice_output_method_id = II.invoice_output_method_id WHERE I.customer_id = 1933705 AND I.bill_date > DATE_SUB(CURDATE(),INTERVAL 1 YEAR) AND (NOT (I.location IS NULL) OR NOT (IOM.invoice_output_method = 'Paper')) ORDER BY I.invoice_date DESC Connection ID (thread ID): 259549 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 120401 10:17:18 mysqld_safe Number of processes running now: 0 120401 10:17:18 mysqld_safe mysqld restarted 120401 10:17:18 [Note] Plugin 'FEDERATED' is disabled. 120401 10:17:18 InnoDB: Initializing buffer pool, size = 8.0G 120401 10:17:19 InnoDB: Completed initialization of buffer pool InnoDB: Log scan progressed past the checkpoint lsn 1833 2423555661 120401 10:17:20 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 1833 2423604303 120401 10:17:20 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 703256085, file name ./mysql-bin.004167 120401 10:17:22 InnoDB: Started; log sequence number 1833 2423604303 120401 10:17:22 [Note] Recovering after a crash using mysql-bin 120401 10:17:25 [Note] Starting crash recovery... 120401 10:17:25 [Note] Crash recovery finished. 120401 10:17:25 [Note] Event Scheduler: Loaded 0 events 120401 10:17:25 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.1.62-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) *** glibc detected *** /usr/sbin/mysqld: corrupted double-linked list: 0x00000000299411b0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x378c671cf0] /lib64/libc.so.6(__libc_malloc+0x6e)[0x378c67374e] /usr/sbin/mysqld(my_malloc+0x32)[0x843512] /usr/sbin/mysqld(hp_get_new_block+0x56)[0x73a0f6] /usr/sbin/mysqld(heap_write+0x17a)[0x7384aa] /usr/sbin/mysqld(_ZN7ha_heap9write_rowEPh+0x42)[0x7351e2] /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x4f)[0x6a5adf] /usr/sbin/mysqld[0x6309e2] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld[0x62cb19] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x63)[0x62cc73] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld[0x62cb19] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x63)[0x62cc73] /usr/sbin/mysqld[0x62cb19] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x63)[0x62cc73] /usr/sbin/mysqld[0x62d06d] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x90b)[0x63cd5b] /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x5eb)[0x63eb3b] /usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x180)[0x63ee40] /usr/sbin/mysqld[0x5d37c1] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x14ec)[0x5d8efc] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x1f3)[0x5ddd13] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x53a)[0x5de32a] /usr/sbin/mysqld(_Z10do_commandP3THD+0xde)[0x5df46e] /usr/sbin/mysqld(handle_one_connection+0x661)[0x5d2201] /lib64/libpthread.so.0[0x378ce0677d] /lib64/libc.so.6(clone+0x6d)[0x378c6d325d] ======= Memory map: ======== 00400000-00a56000 r-xp 00000000 fd:03 577887 /usr/sbin/mysqld 00b55000-00c8a000 rw-p 00655000 fd:03 577887 /usr/sbin/mysqld 00c8a000-00ca2000 rw-p 00c8a000 00:00 0 00d89000-00e12000 rw-p 00789000 fd:03 577887 /usr/sbin/mysqld 16baf000-2c29b000 rw-p 16baf000 00:00 0 [heap] 40401000-40402000 ---p 40401000 00:00 0 40402000-40e02000 rw-p 40402000 00:00 0 40e02000-40e03000 ---p 40e02000 00:00 0 40e03000-41803000 rw-p 40e03000 00:00 0 41803000-41804000 ---p 41803000 00:00 0 41804000-41844000 rw-p 41804000 00:00 0 41844000-41845000 ---p 41844000 00:00 0 41845000-41885000 rw-p 41845000 00:00 0 41885000-41886000 ---p 41885000 00:00 0 41886000-418c6000 rw-p 41886000 00:00 0 418df000-418e0000 ---p 418df000 00:00 0 418e0000-41920000 rw-p 418e0000 00:00 0 41920000-41921000 ---p 41920000 00:00 0 41921000-41961000 rw-p 41921000 00:00 0 41961000-41962000 ---p 41961000 00:00 0 41962000-419a2000 rw-p 41962000 00:00 0 419a2000-419a3000 ---p 419a2000 00:00 0 419a3000-419e3000 rw-p 419a3000 00:00 0 419e3000-419e4000 ---p 419e3000 00:00 0 419e4000-41a24000 rw-p 419e4000 00:00 0 41a24000-41a25000 ---p 41a24000 00:00 0 41a25000-41a65000 rw-p 41a25000 00:00 0 41bd3000-41bd4000 ---p 41bd3000 00:00 0 41bd4000-425d4000 rw-p 41bd4000 00:00 0 425d4000-425d5000 ---p 425d4000 00:00 0 425d5000-42fd5000 rw-p 425d5000 00:00 0 42fd5000-42fd6000 ---p 42fd5000 00:00 0 42fd6000-439d6000 rw-p 42fd6000 00:00 0 439d6000-439d7000 ---p 439d6000 00:00 0 439d7000-443d7000 rw-p 439d7000 00:00 0 443d7000-443d8000 ---p 443d7000 00:00 0 443d8000-44dd8000 rw-p 443d8000 00:00 0 44dd8000-44dd9000 ---p 44dd8000 00:00 0 44dd9000-457d9000 rw-p 44dd9000 00:00 0 457d9000-457da000 ---p 457d9000 00:00 0 457da000-461da000 rw-p 457da000 00:00 0 461da000-461db000 ---p 461da000 00:00 0 461db000-4621b000 rw-p 461db000 00:00 0 4621b000-4621c000 ---p 4621b000 00:00 0 4621c000-4625c000 rw-p 4621c000 00:00 0 4625c000-4625d000 ---p 4625c000 00:00 0 4625d000-4629d000 rw-p 4625d000 00:00 0 4629d000-4629e000 ---p 4629d000 00:00 0 4629e000-462de000 rw-p 4629e000 00:00 0 462de000-462df000 ---p 462de000 00:00 0 462df000-4631f000 rw-p 462df000 00:00 0 4631f000-46320000 ---p 4631f000 00:00 0 46320000-46360000 rw-p 46320000 00:00 0 46360000-46361000 ---p 46360000 00:00 0 46361000-463a1000 rw-p 46361000 00:00 0 463a1000-463a2000 ---p 463a1000 00:00 0 463a2000-463e2000 rw-p 463a2000 00:00 0 463e2000-463e3000 ---p 463e2000 00:00 0 463e3000-46423000 rw-p 463e3000 00:00 0 46423000-46424000 ---p 46423000 00:00 0 46424000-46464000 rw-p 46424000 00:00 0 46464000-46465000 ---p 46464000 00:00 0 46465000-464a5000 rw-p 46465000 00:00 0 464a5000-464a6000 ---p 464a5000 00:00 0 464a6000-464e6000 rw-p 464a6000 00:00 0 464e6000-464e7000 ---p 464e6000 00:00 0 464e7000-46527000 rw-p 464e7000 00:00 0 46527000-46528000 ---p 46527000 00:00 0 46528000-46568000 rw-p 46528000 00:00 0 46568000-46569000 ---p 46568000 00:00 0 46569000-465a9000 rw-p 46569000 00:00 0 465a9000-465aa000 ---p 465a9000 00:00 0 465aa000-465ea000 rw-p 465aa000 00:00 0 465ea000-465eb000 ---p 465ea000 00:00 0 465eb000-4662b000 rw-p 465eb000 00:00 0 4662b000-4662c000 ---p 4662b000 00:00 0 4662c000-4666c000 rw-p 4662c000 00:00 0 4666c000-4666d000 ---p 4666c000 00:00 0 4666d000-466ad000 rw-p 4666d000 00:00 0 466ad000-466ae000 ---p 466ad000 00:00 0 466ae000-466ee000 rw-p 466ae000 00:00 0 466ee000-466ef000 ---p 466ee000 00:00 0 466ef000-4672f000 rw-p 466ef000 00:00 0 4672f000-46730000 ---p 4672f000 00:00 0 46730000-46770000 rw-p 46730000 00:00 0 46770000-46771000 ---p 46770000 00:00 0 46771000-467b1000 rw-p 46771000 00:00 0 467b1000-467b2000 ---p 467b1000 00:00 0 467b2000-467f2000 rw-p 467b2000 00:00 0 467f2000-467f3000 ---p 467f2000 00:00 0 467f3000-46833000 rw-p 467f3000 00:00 0 46833000-46834000 ---p 46833000 00:00 0 46834000-46874000 rw-p 46834000 00:00 0 46874000-46875000 ---p 46874000 00:00 0 46875000-468b5000 rw-p 46875000 00:00 0 468b5000-468b6000 ---p 468b5000 00:00 0 468b6000-468f6000 rw-p 468b6000 00:00 0 468f6000-468f7000 ---p 468f6000 00:00 0 468f7000-46937000 rw-p 468f7000 00:00 0 46937000-46938000 ---p 46937000 00:00 0 46938000-46978000 rw-p 46938000 00:00 0 46978000-46979000 ---p 46978000 00:00 0 46979000-469b9000 rw-p 46979000 00:00 0 469b9000-469ba000 ---p 469b9000 00:00 0 469ba000-469fa000 rw-p 469ba000 00:00 0 469fa000-469fb000 ---p 469fa000 00:00 0 469fb000-46a3b000 rw-p 469fb000 00:00 0 46a3b000-46a3c000 ---p 46a3b000 00:00 0 46a3c000-46a7c000 rw-p 46a3c000 00:00 0 46a7c000-46a7d000 ---p 46a7c000 00:00 0 46a7d000-46abd000 rw-p 46a7d000 00:00 0 46abd000-46abe000 ---p 46abd000 00:00 0 46abe000-46afe000 rw-p 46abe000 00:00 0 46afe000-46aff000 ---p 46afe000 00:00 0 46aff000-46b3f000 rw-p 46aff000 00:00 0 46b3f000-46b40000 ---p 46b3f000 00:00 0 46b40000-46b80000 rw-p 46b40000 00:00 0 46b80000-46b81000 ---p 46b80000 00:00 0 46b81000-46bc1000 rw-p 46b81000 00:00 0 46bc1000-46bc2000 ---p 46bc1000 00:00 0 46bc2000-46c02000 rw-p 46bc2000 00:00 0 46c02000-46c03000 ---p 46c02000 00:00 0 46c03000-46c43000 rw-p 46c03000 00:00 0 46c43000-46c44000 ---p 46c43000 00:00 0 46c44000-46c84000 rw-p 46c44000 00:00 0 46c84000-46c85000 ---p 46c84000 00:00 0 46c85000-46cc5000 rw-p 46c85000 00:00 0 46cc5000-46cc6000 ---p 46cc5000 00:00 0 46cc6000-46d06000 rw-p 46cc6000 00:00 0 46d06000-46d07000 ---p 46d06000 00:00 0 46d07000-46d47000 rw-p 46d07000 00:00 0 46d47000-46d48000 ---p 46d47000 00:00 0 46d48000-46d88000 rw-p 46d48000 00:00 0 46d88000-46d89000 ---p 46d88000 00:00 0 46d89000-46dc9000 rw-p 46d89000 00:00 0 46dc9000-46dca000 ---p 46dc9000 00:00 0 46dca000-46e0a000 rw-p 46dca000 00:00 0 46e0a000-46e0b000 ---p 46e0a000 00:00 0 46e0b000-46e4b000 rw-p 46e0b000 00:00 0 46e4b000-46e4c000 ---p 46e4b000 00:00 0 46e4c000-46e8c000 rw-p 46e4c000 00:00 0 46e8c000-46e8d000 ---p 46e8c000 00:00 0 46e8d000-46ecd000 rw-p 46e8d000 00:00 0 46ecd000-46ece000 ---p 46ecd000 00:00 0 46ece000-46f0e000 rw-p 46ece000 00:00 0 46f0e000-46f0f000 ---p 46f0e000 00:00 0 46f0f000-46f4f000 rw-p 46f0f000 00:00 0 46f4f000-46f50000 ---p 46f4f000 00:00 0 46f50000-46f90000 rw-p 46f50000 00:00 0 46f90000-46f91000 ---p 46f90000 00:00 0 46f91000-46fd1000 rw-p 46f91000 00:00 0 46fd1000-46fd2000 ---p 46fd1000 00:00 0 46fd2000-47012000 rw-p 46fd2000 00:00 0 47012000-47013000 ---p 47012000 00:00 0 47013000-47053000 rw-p 47013000 00:00 0 47053000-47054000 ---p 47053000 00:00 0 47054000-47094000 rw-p 47054000 00:00 0 47094000-47095000 ---p 47094000 00:00 0 47095000-470d5000 rw-p 47095000 00:00 0 470d5000-470d6000 ---p 470d5000 00:00 0 470d6000-47116000 rw-p 470d6000 00:00 0 47116000-47117000 ---p 47116000 00:00 0 47117000-47157000 rw-p 47117000 00:00 0 47157000-47158000 ---p 47157000 00:00 0 47158000-47198000 rw-p 47158000 00:00 0 47198000-47199000 ---p 47198000 00:00 0 47199000-471d9000 rw-p 47199000 00:00 0 471d9000-471da000 ---p 471d9000 00:00 0 471da000-4721a000 rw-p 471da000 00:00 0 378c200000-378c21c000 r-xp 00000000 fd:00 161284 /lib64/ld-2.5.so 378c41c000-378c41d000 r--p 0001c000 fd:00 161284 /lib64/ld-2.5.so 378c41d000-378c41e000 rw-p 0001d000 fd:00 161284 /lib64/ld-2.5.so 378c600000-378c74d000 r-xp 00000000 fd:00 161305 /lib64/libc-2.5.so 378c74d000-378c94d000 ---p 0014d000 fd:00 161305 /lib64/libc-2.5.so 378c94d000-378c951000 r--p 0014d000 fd:00 161305 /lib64/libc-2.5.so 378c951000-378c952000 rw-p 00151000 fd:00 161305 /lib64/libc-2.5.so 378c952000-378c957000 rw-p 378c952000 00:00 0 378ca00000-378ca02000 r-xp 00000000 fd:00 161317 /lib64/libdl-2.5.so 378ca02000-378cc02000 ---p 00002000 fd:00 161317 /lib64/libdl-2.5.so 378cc02000-378cc03000 r--p 00002000 fd:00 161317 /lib64/libdl-2.5.so 378cc03000-378cc04000 rw-p 00003000 fd:00 161317 /lib64/libdl-2.5.so 378ce00000-378ce16000 r-xp 00000000 fd:00 161306 /lib64/libpthread-2.5.so 378ce16000-378d015000 ---p 00016000 fd:00 161306 /lib64/libpthread-2.5.so 378d015000-378d016000 r--p 00015000 fd:00 161306 /lib64/libpthread-2.5.so 378d016000-378d017000 rw-p 00016000 fd:00 161306 /lib64/libpthread-2.5.so 378d017000-378d01b000 rw-p 378d017000 00:00 0 378d200000-378d282000 r-xp 00000000 fd:00 161331 /lib64/libm-2.5.so 378d282000-378d481000 ---p 00082000 fd:00 161331 /lib64/libm-2.5.so 378d481000-378d482000 r--p 00081000 fd:00 161331 /lib64/libm-2.5.so 378d482000-378d483000 rw-p 00082000 fd:00 161331 /lib64/libm-2.5.so 378da00000-378da15000 r-xp 00000000 fd:00 161455 /lib64/libselinux.so.1 378da15000-378dc15000 ---p 00015000 fd:00 161455 /lib64/libselinux.so.1 378dc15000-378dc17000 rw-p 00015000 fd:00 161455 /lib64/libselinux.so.1 378dc17000-378dc18000 rw-p 378dc17000 00:00 0 378e200000-378e23b000 r-xp 00000000 fd:00 161451 /lib64/libsepol.so.1 378e23b000-378e43b000 ---p 0003b000 fd:00 161451 /lib64/libsepol.so.1 378e43b000-378e43c000 rw-p 0003b000 fd:00 161451 /lib64/libsepol.so.1 378e43c000-378e446000 rw-p 378e43c000 00:00 0 378e600000-378e615000 r-xp 00000000 fd:00 161413 /lib64/libnsl-2.5.so 378e615000-378e814000 ---p 00015000 fd:00 161413 /lib64/libnsl-2.5.so 378e814000-378e815000 r--p 00014000 fd:00 161413 /lib64/libnsl-2.5.so 378e815000-378e816000 rw-p 00015000 fd:00 161413 /lib64/libnsl-2.5.so 378e816000-378e818000 rw-p 378e816000 00:00 0 378ea00000-378ea0d000 r-xp 00000000 fd:00 161335 /lib64/libgcc_s-4.1.2-20080825.so.1 378ea0d000-378ec0d000 ---p 0000d000 fd:00 161335 /lib64/libgcc_s-4.1.2-20080825.so.1 378ec0d000-378ec0e000 rw-p 0000d000 fd:00 161335 /lib64/libgcc_s-4.1.2-20080825.so.1 378ee00000-378ee09000 r-xp 00000000 fd:00 161472 /lib64/libcrypt-2.5.so 378ee09000-378f008000 ---p 00009000 fd:00 161472 /lib64/libcrypt-2.5.so 378f008000-378f009000 r--p 00008000 fd:00 161472 /lib64/libcrypt-2.5.so 378f009000-378f00a000 rw-p 00009000 fd:00 161472 /lib64/libcrypt-2.5.so 378f00a000-378f038000 rw-p 378f00a000 00:00 0 378fa00000-378fa11000 r-xp 00000000 fd:00 161495 /lib64/libresolv-2.5.so 378fa11000-378fc11000 ---p 00011000 fd:00 161495 /lib64/libresolv-2.5.so 378fc11000-378fc12000 r--p 00011000 fd:00 161495 /lib64/libresolv-2.5.so 378fc12000-378fc13000 rw-p 00012000 fd:00 161495 /lib64/libresolv-2.5.so 378fc13000-378fc15000 rw-p 378fc13000 00:00 0 378fe00000-378fe02000 r-xp 00000000 fd:00 161496 /lib64/libcom_err.so.2.1 378fe02000-3790001000 ---p 00002000 fd:00 161496 /lib64/libcom_err.so.2.1 3790001000-3790002000 rw-p 00001000 fd:00 161496 /lib64/libcom_err.so.2.1 3791e00000-3791e02000 r-xp 00000000 fd:00 161494 /lib64/libkeyutils-1.2.so 3791e02000-3792001000 ---p 00002000 fd:00 161494 /lib64/libkeyutils-1.2.so 3792001000-3792002000 rw-p 00001000 fd:00 161494 /lib64/libkeyutils-1.2.so 2aaaaaabe000-2aaaaab06000 rw-p 2aaaaaabe000 00:00 0 2aaaaab17000-2aaaaab21000 r-xp 00000000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaab21000-2aaaaad20000 ---p 0000a000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaad20000-2aaaaad21000 r--p 00009000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaad21000-2aaaaad22000 rw-p 0000a000 fd:00 161318 /lib64/libnss_files-2.5.so 2aaaaad22000-2aaaaafff000 r-xp 00000000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaaafff000-2aaaab1fe000 ---p 002dd000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab1fe000-2aaaab229000 rw-p 002dc000 fd:00 226544 /lib64/libnss_ldap-2.5.so 2aaaab229000-2aace065d000 rw-p 2aaaab229000 00:00 0 2aace4000000-2aace5302000 rw-p 2aace4000000 00:00 0 2aace5302000-2aace8000000 ---p 2aace5302000 00:00 0 2aace8f6b000-2aaceb335000 rw-p 2aace8f6b000 00:00 0 2aacec000000-2aacec2b7000 rw-p 2aacec000000 00:00 0 2aacec2b7000-2aacf0000000 ---p 2aacec2b7000 00:00 0 2b51c6ae0000-2b51c6ae2000 rw-p 2b51c6ae0000 00:00 0 2b51c6af3000-2b51c6af6000 rw-p 2b51c6af3000 00:00 0 7fff8b2af000-7fff8b2c4000 rw-p 7ffffffe9000 00:00 0 [stack] 7fff8b3a8000-7fff8b3ab000 r-xp 7fff8b3a8000 00:00 0 [vdso] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall] 21:56:12 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=72 max_threads=2000 thread_count=29 connection_count=29 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2332895 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x2aace4092b30 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 47051f28 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x85395e] /usr/sbin/mysqld(handle_fatal_signal+0x38c)[0x6ae7ec] /lib64/libpthread.so.0[0x378ce0ebe0] /lib64/libc.so.6(gsignal+0x35)[0x378c630285] /lib64/libc.so.6(abort+0x110)[0x378c631d30] /lib64/libc.so.6[0x378c6692bb] /lib64/libc.so.6[0x378c671cf0] /lib64/libc.so.6(__libc_malloc+0x6e)[0x378c67374e] /usr/sbin/mysqld(my_malloc+0x32)[0x843512] /usr/sbin/mysqld(hp_get_new_block+0x56)[0x73a0f6] /usr/sbin/mysqld(heap_write+0x17a)[0x7384aa] /usr/sbin/mysqld(_ZN7ha_heap9write_rowEPh+0x42)[0x7351e2] /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x4f)[0x6a5adf] /usr/sbin/mysqld[0x6309e2] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld[0x62cb19] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x63)[0x62cc73] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fe)[0x62ce0e] /usr/sbin/mysqld[0x62cb19] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x63)[0x62cc73] /usr/sbin/mysqld[0x62cb19] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x63)[0x62cc73] /usr/sbin/mysqld[0x62d06d] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x90b)[0x63cd5b] /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x5eb)[0x63eb3b] /usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x180)[0x63ee40] /usr/sbin/mysqld[0x5d37c1] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x14ec)[0x5d8efc] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x1f3)[0x5ddd13] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x53a)[0x5de32a] /usr/sbin/mysqld(_Z10do_commandP3THD+0xde)[0x5df46e] /usr/sbin/mysqld(handle_one_connection+0x661)[0x5d2201] /lib64/libpthread.so.0[0x378ce0677d] /lib64/libc.so.6(clone+0x6d)[0x378c6d325d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (2aacec061540): is an invalid pointer Connection ID (thread ID): 401555 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 120401 16:56:13 mysqld_safe Number of processes running now: 0 120401 16:56:13 mysqld_safe mysqld restarted 120401 16:56:13 [Note] Plugin 'FEDERATED' is disabled. 120401 16:56:13 InnoDB: Initializing buffer pool, size = 8.0G 120401 16:56:15 InnoDB: Completed initialization of buffer pool InnoDB: Log scan progressed past the checkpoint lsn 1833 2478966248 120401 16:56:15 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 1833 2479023090 120401 16:56:15 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 42382561, file name ./mysql-bin.004168 120401 16:56:17 InnoDB: Started; log sequence number 1833 2479023090 120401 16:56:17 [Note] Recovering after a crash using mysql-bin 120401 16:56:17 [Note] Starting crash recovery... 120401 16:56:17 [Note] Crash recovery finished. 120401 16:56:17 [Note] Event Scheduler: Loaded 0 events 120401 16:56:17 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.1.62-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) 120401 19:00:52 [Warning] Statement may not be safe to log in statement format. Statement: UPDATE PAYMENT_INFO SET agency_id = 93 WHERE customer_id = 2745820 LIMIT 1