*~*~ console message *~*~ *** glibc detected *** /usr/local/mysql/bin/mysqld: free (): invalid pointer: 0x00007f47601a5010 *** ======= Backtrace: ========= /lib/libc.so.6(+0x775b6)[0x7f665de225b6] /lib/libc.so.6(cfree+0x73)[0x7f665de28e83] /usr/local/mysql/bin/mysqld(end_io_cache+0x4b)[0x7817fb] /usr/local/mysql/bin/mysqld(_ZN14select_to_file8send_eofEv+0x28)[0x54e118] /usr/local/mysql/bin/mysqld[0x59ca2e] /usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x925)[0x5b0505] /usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1 _ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+ 0x1c3)[0x5ac253] /usr/local/mysql/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x179)[ 0x5b1fb9] /usr/local/mysql/bin/mysqld[0x5703d1] /usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x226f)[0x5759ff] /usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x15a)[0x577d ea] /usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x 1321)[0x579961] /usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x337)[0x60d917] /usr/local/mysql/bin/mysqld(handle_one_connection+0x54)[0x60d984] /usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x5b)[0x909b8b] /lib/libpthread.so.0(+0x69ca)[0x7f665ebfe9ca] /lib/libc.so.6(clone+0x6d)[0x7f665de9170d] ======= Memory map: ======== 00400000-00ce5000 r-xp 00000000 fb:02 12326391 /usr/lo cal/mysql/bin/mysqld 00de4000-00efa000 rw-p 008e4000 fb:02 12326391 /usr/lo cal/mysql/bin/mysqld 00efa000-00f25000 rw-p 00000000 00:00 0 0279b000-183c8000 rw-p 00000000 00:00 0 [heap] 7f473c000000-7f473dac9000 rw-p 00000000 00:00 0 7f473dac9000-7f4740000000 ---p 00000000 00:00 0 7f4747fd5000-7f4748111000 rw-s 00001000 fb:00 10913655 /data/m ysql/sancp/sancp_tnwp02243_20101206_3#P#p7#SP#p7sp5.MYD 7f4748111000-7f47482d5000 rw-s 00000000 fb:00 10913597 /data/m ysql/sancp/sancp_tnwp02243_20101206_3#P#p4#SP#p4sp0.MYD 7f47482d5000-7f47484b0000 rw-s 00000000 fb:00 10913528 /data/m ysql/sancp/sancp_tnwp02243_20101206_2#P#p7#SP#p7sp5.MYD 7f47484b0000-7f475c1a4000 rw-s 00000000 fb:02 13107222 /tmp/#s qle20_19_2.MYD 7f475c1a5000-7f47641a6000 rw-p 00000000 00:00 0 7f47641a6000-7f47646db000 rw-s 00000000 fb:00 10909595 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp7.MYD 7f47646db000-7f47649f9000 rw-s 00000000 fb:00 10909593 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp6.MYD 7f47649f9000-7f47682d9000 rw-s 00000000 fb:00 10909591 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp5.MYD 7f47682d9000-7f47688d1000 rw-s 00000000 fb:00 10909589 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp4.MYD 7f47688d1000-7f47693fb000 rw-s 00000000 fb:00 10909587 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp3.MYD 7f47693fb000-7f4769b79000 rw-s 00000000 fb:00 10909585 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp2.MYD 7f4769b79000-7f476ac49000 rw-s 00000000 fb:00 10909583 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp1.MYD 7f476ac49000-7f476b72b000 rw-s 00000000 fb:00 10909581 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p7#SP#p7sp0.MYD 7f476b72b000-7f476bc6e000 rw-s 00000000 fb:00 10909579 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp7.MYD 7f476bc6e000-7f476bfc8000 rw-s 00000000 fb:00 10909577 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp6.MYD 7f476bfc8000-7f476df11000 rw-s 00000000 fb:00 10909575 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp5.MYD 7f476df11000-7f476e7b8000 rw-s 00000000 fb:00 10909573 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp4.MYD 7f476e7b8000-7f476f1e5000 rw-s 00000000 fb:00 10909571 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp3.MYD 7f476f1e5000-7f476fa1f000 rw-s 00000000 fb:00 10909569 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp2.MYD 7f476fa1f000-7f4770140000 rw-s 00000000 fb:00 10909567 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp1.MYD 7f4770140000-7f4770ba0000 rw-s 00000000 fb:00 10909565 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p6#SP#p6sp0.MYD 7f4770ba0000-7f477143a000 rw-s 00000000 fb:00 10909563 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p5#SP#p5sp7.MYD 7f477143a000-7f4771bea000 rw-s 00000000 fb:00 10909561 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p5#SP#p5sp6.MYD 7f4771bea000-7f4773b48000 rw-s 00000000 fb:00 10909559 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p5#SP#p5sp5.MYD 7f4773b48000-7f47747b5000 rw-s 00000000 fb:00 10909557 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p5#SP#p5sp4.MYD 7f47747b5000-7f4775193000 rw-s 00000000 fb:00 10909555 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p5#SP#p5sp3.MYD 7f4775193000-7f4775988000 rw-s 00000000 fb:00 10909553 /data/m ysql/sancp/sancp_tnwp02241_20101206_2#P#p5#SP#p5sp2.MYD *~*~ error log *~*~ 110608 19:43:23 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=25769803776 read_buffer_size=67108864 max_used_connections=8 max_threads=151 thread_count=8 connection_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 37537462 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x15d73690 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f665eee0e20 thread_stack 0x40000 /usr/local/mysql/bin/mysqld(my_print_stacktrace+0x39)[0x796d89] /usr/local/mysql/bin/mysqld(handle_segfault+0x380)[0x4fd110] /lib/libpthread.so.0(+0xf8f0)[0x7f665ec078f0] /lib/libc.so.6(gsignal+0x35)[0x7f665dddea75] /lib/libc.so.6(abort+0x180)[0x7f665dde25c0] /lib/libc.so.6(+0x6d4fb)[0x7f665de184fb] /lib/libc.so.6(+0x775b6)[0x7f665de225b6] /lib/libc.so.6(cfree+0x73)[0x7f665de28e83] /usr/local/mysql/bin/mysqld(end_io_cache+0x4b)[0x7817fb] /usr/local/mysql/bin/mysqld(_ZN14select_to_file8send_eofEv+0x28)[0x54e118] /usr/local/mysql/bin/mysqld[0x59ca2e] /usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x925)[0x5b0505] /usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x1c3)[0x5ac253] /usr/local/mysql/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x179)[0x5b1fb9] /usr/local/mysql/bin/mysqld[0x5703d1] /usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x226f)[0x5759ff] /usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x15a)[0x577dea] /usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1321)[0x579961] /usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x337)[0x60d917] /usr/local/mysql/bin/mysqld(handle_one_connection+0x54)[0x60d984] /usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x5b)[0x909b8b] /lib/libpthread.so.0(+0x69ca)[0x7f665ebfe9ca] /lib/libc.so.6(clone+0x6d)[0x7f665de9170d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x15d7e450): SELECT `sid`,`start_time`,`ip_proto`,`dst_ip`,`dst_port`,`src_ip`,`src_port`,`dst_pkts`,`dst_bytes`,`src_pkts`,`src_bytes`,`dst_flags`,`src_flags`, '137' FROM `sancp`.`sancp_tnwp02238_eth3_20101205_0` WHERE `ip_proto` IN(17) AND (`src_port` IN(514)) INTO OUTFILE '/data/tmp/infile-132-1319-sancp_tnwp02238_eth3_20101205_0-flip.txt' Connection ID (thread ID): 26 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file 110608 19:45:43 mysqld_safe Starting mysqld daemon with databases from /data/mysql 110608 19:45:44 [Warning] Changed limits: max_open_files: 1048576 max_connections: 151 table_cache: 524207 110608 19:45:46 [Note] Plugin 'FEDERATED' is disabled. 110608 19:45:46 InnoDB: The InnoDB memory heap is disabled 110608 19:45:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins 110608 19:45:46 InnoDB: Compressed tables use zlib 1.2.3 110608 19:45:46 InnoDB: Using Linux native AIO 110608 19:45:46 InnoDB: Initializing buffer pool, size = 10.0G 110608 19:45:47 InnoDB: Completed initialization of buffer pool 110608 19:45:47 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 1607871797602 110608 19:45:47 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 1607877040128 InnoDB: Doing recovery: scanned up to log sequence number 1607882283008 InnoDB: Doing recovery: scanned up to log sequence number 1607887525888 InnoDB: Doing recovery: scanned up to log sequence number 1607892768768 InnoDB: Doing recovery: scanned up to log sequence number 1607898011648 InnoDB: Doing recovery: scanned up to log sequence number 1607903254528 InnoDB: Doing recovery: scanned up to log sequence number 1607908497408 InnoDB: Doing recovery: scanned up to log sequence number 1607913740288 InnoDB: Doing recovery: scanned up to log sequence number 1607918983168 InnoDB: Doing recovery: scanned up to log sequence number 1607924226048 InnoDB: Doing recovery: scanned up to log sequence number 1607929468928 InnoDB: Doing recovery: scanned up to log sequence number 1607934711808 InnoDB: Doing recovery: scanned up to log sequence number 1607939954688 InnoDB: Doing recovery: scanned up to log sequence number 1607945197568 InnoDB: Doing recovery: scanned up to log sequence number 1607950440448 InnoDB: Doing recovery: scanned up to log sequence number 1607955683328 InnoDB: Doing recovery: scanned up to log sequence number 1607960926208 InnoDB: Doing recovery: scanned up to log sequence number 1607966169088 InnoDB: Doing recovery: scanned up to log sequence number 1607971411968 InnoDB: Doing recovery: scanned up to log sequence number 1607976654848 InnoDB: Doing recovery: scanned up to log sequence number 1607981897728 InnoDB: Doing recovery: scanned up to log sequence number 1607987140608 InnoDB: Doing recovery: scanned up to log sequence number 1607992383488 InnoDB: Doing recovery: scanned up to log sequence number 1607997626368 InnoDB: Doing recovery: scanned up to log sequence number 1608002869248 InnoDB: Doing recovery: scanned up to log sequence number 1608008112128 InnoDB: Doing recovery: scanned up to log sequence number 1608013355008 InnoDB: Doing recovery: scanned up to log sequence number 1608018597888 InnoDB: Doing recovery: scanned up to log sequence number 1608023840768 InnoDB: Doing recovery: scanned up to log sequence number 1608029083648 InnoDB: Doing recovery: scanned up to log sequence number 1608034326528 InnoDB: Doing recovery: scanned up to log sequence number 1608039569408 InnoDB: Doing recovery: scanned up to log sequence number 1608044812288 InnoDB: Doing recovery: scanned up to log sequence number 1608050055168 InnoDB: Doing recovery: scanned up to log sequence number 1608055298048 InnoDB: Doing recovery: scanned up to log sequence number 1608060540928 InnoDB: Doing recovery: scanned up to log sequence number 1608065783808 InnoDB: Doing recovery: scanned up to log sequence number 1608071026688 InnoDB: Doing recovery: scanned up to log sequence number 1608076269568 InnoDB: Doing recovery: scanned up to log sequence number 1608081512448 InnoDB: Doing recovery: scanned up to log sequence number 1608086755328 InnoDB: Doing recovery: scanned up to log sequence number 1608091998208 InnoDB: Doing recovery: scanned up to log sequence number 1608097241088 InnoDB: Doing recovery: scanned up to log sequence number 1608102483968 InnoDB: Doing recovery: scanned up to log sequence number 1608107726848 InnoDB: Doing recovery: scanned up to log sequence number 1608112969728 InnoDB: Doing recovery: scanned up to log sequence number 1608118212608 InnoDB: Doing recovery: scanned up to log sequence number 1608123455488 InnoDB: Doing recovery: scanned up to log sequence number 1608128698368 InnoDB: Doing recovery: scanned up to log sequence number 1608133941248 InnoDB: Doing recovery: scanned up to log sequence number 1608139184128 InnoDB: Doing recovery: scanned up to log sequence number 1608144427008 InnoDB: Doing recovery: scanned up to log sequence number 1608149669888 InnoDB: Doing recovery: scanned up to log sequence number 1608154912768 InnoDB: Doing recovery: scanned up to log sequence number 1608160155648 InnoDB: Doing recovery: scanned up to log sequence number 1608165398528 InnoDB: Doing recovery: scanned up to log sequence number 1608170641408 InnoDB: Doing recovery: scanned up to log sequence number 1608175884288 InnoDB: Doing recovery: scanned up to log sequence number 1608181127168 InnoDB: Doing recovery: scanned up to log sequence number 1608186370048 InnoDB: Doing recovery: scanned up to log sequence number 1608191612928 InnoDB: Doing recovery: scanned up to log sequence number 1608196855808 InnoDB: Doing recovery: scanned up to log sequence number 1608202098688 InnoDB: Doing recovery: scanned up to log sequence number 1608207341568 InnoDB: Doing recovery: scanned up to log sequence number 1608212584448 InnoDB: Doing recovery: scanned up to log sequence number 1608217827328 InnoDB: Doing recovery: scanned up to log sequence number 1608223070208 InnoDB: Doing recovery: scanned up to log sequence number 1608228313088 InnoDB: Doing recovery: scanned up to log sequence number 1608233555968 InnoDB: Doing recovery: scanned up to log sequence number 1608238798848 InnoDB: Doing recovery: scanned up to log sequence number 1608244041728 InnoDB: Doing recovery: scanned up to log sequence number 1608249284608 InnoDB: Doing recovery: scanned up to log sequence number 1608254527488 InnoDB: Doing recovery: scanned up to log sequence number 1608259770368 InnoDB: Doing recovery: scanned up to log sequence number 1608265013248 InnoDB: Doing recovery: scanned up to log sequence number 1608270256128 InnoDB: Doing recovery: scanned up to log sequence number 1608275499008 InnoDB: Doing recovery: scanned up to log sequence number 1608280741888 InnoDB: Doing recovery: scanned up to log sequence number 1608285984768 InnoDB: Doing recovery: scanned up to log sequence number 1608291227648 InnoDB: Doing recovery: scanned up to log sequence number 1608296470528 InnoDB: Doing recovery: scanned up to log sequence number 1608301713408 InnoDB: Doing recovery: scanned up to log sequence number 1608306956288 InnoDB: Doing recovery: scanned up to log sequence number 1608312199168 InnoDB: Doing recovery: scanned up to log sequence number 1608317442048 InnoDB: Doing recovery: scanned up to log sequence number 1608322684928 InnoDB: Doing recovery: scanned up to log sequence number 1608327927808 InnoDB: Doing recovery: scanned up to log sequence number 1608333170688 InnoDB: Doing recovery: scanned up to log sequence number 1608338413568 InnoDB: Doing recovery: scanned up to log sequence number 1608343656448 InnoDB: Doing recovery: scanned up to log sequence number 1608348899328 InnoDB: Doing recovery: scanned up to log sequence number 1608354142208 InnoDB: Doing recovery: scanned up to log sequence number 1608359385088 InnoDB: Doing recovery: scanned up to log sequence number 1608364627968 InnoDB: Doing recovery: scanned up to log sequence number 1608369870848 InnoDB: Doing recovery: scanned up to log sequence number 1608375113728 InnoDB: Doing recovery: scanned up to log sequence number 1608380356608 InnoDB: Doing recovery: scanned up to log sequence number 1608385599488 InnoDB: Doing recovery: scanned up to log sequence number 1608390842368 InnoDB: Doing recovery: scanned up to log sequence number 1608396085248 InnoDB: Doing recovery: scanned up to log sequence number 1608401328128 InnoDB: Doing recovery: scanned up to log sequence number 1608406571008 InnoDB: Doing recovery: scanned up to log sequence number 1608411813888 InnoDB: Doing recovery: scanned up to log sequence number 1608417056768 InnoDB: Doing recovery: scanned up to log sequence number 1608422299648 InnoDB: Doing recovery: scanned up to log sequence number 1608427542528 InnoDB: Doing recovery: scanned up to log sequence number 1608432785408 InnoDB: Doing recovery: scanned up to log sequence number 1608438028288 InnoDB: Doing recovery: scanned up to log sequence number 1608443271168 InnoDB: Doing recovery: scanned up to log sequence number 1608448514048 InnoDB: Doing recovery: scanned up to log sequence number 1608453756928 InnoDB: Doing recovery: scanned up to log sequence number 1608458999808 InnoDB: Doing recovery: scanned up to log sequence number 1608464242688 InnoDB: Doing recovery: scanned up to log sequence number 1608469485568 InnoDB: Doing recovery: scanned up to log sequence number 1608474728448 InnoDB: Doing recovery: scanned up to log sequence number 1608479971328 InnoDB: Doing recovery: scanned up to log sequence number 1608485214208 InnoDB: Doing recovery: scanned up to log sequence number 1608490457088 InnoDB: Doing recovery: scanned up to log sequence number 1608495699968 InnoDB: Doing recovery: scanned up to log sequence number 1608500942848 InnoDB: Doing recovery: scanned up to log sequence number 1608506185728 InnoDB: Doing recovery: scanned up to log sequence number 1608511428608 InnoDB: Doing recovery: scanned up to log sequence number 1608516671488 InnoDB: Doing recovery: scanned up to log sequence number 1608521914368 InnoDB: Doing recovery: scanned up to log sequence number 1608527157248 InnoDB: Doing recovery: scanned up to log sequence number 1608532400128 InnoDB: Doing recovery: scanned up to log sequence number 1608537643008 InnoDB: Doing recovery: scanned up to log sequence number 1608541536135 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 11863242 row operations to undo InnoDB: Trx id counter is EFC00 110608 19:46:21 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Starting in background the rollback of uncommitted transactions 110608 19:46:50 InnoDB: Rolling back trx with id EFA5D, 11863242 rows to undo InnoDB: Progress in percents: 1110608 19:46:50 InnoDB: Waiting for the background threads to start 110608 19:46:51 InnoDB: 1.1.7 started; log sequence number 1608541536135 110608 19:46:51 [Note] Event Scheduler: Loaded 0 events 110608 19:46:51 [Note] /usr/local/mysql/bin/mysqld: ready for connections. Version: '5.5.13-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL) 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 InnoDB: Rolling back of trx id EFA5D completed 110608 19:52:27 InnoDB: Rollback of non-prepared transactions completed