130424 9:34:36 [Note] Slave SQL thread initialized, starting replication in log 'veso3dblx01-bin.000158' at position 403016942, relay log './mysqld-relay-bin.000429' position: 403017094 130424 9:34:36 InnoDB: Assertion failure in thread 140597431609088 in file row0mysql.c line 1468 InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 07:34:36 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=2147483648 read_buffer_size=67108864 max_used_connections=0 max_threads=5000 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 657514183 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fec82a351e0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 130424 9:34:36 [Note] Event Scheduler: Loaded 0 events 130424 9:34:36 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.30-1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) stack_bottom = 7fdf63f799a0 thread_stack 0x100000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7fec80e129d9] /usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x7fec80cfb498] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7fec804b0030] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fec7ed43475] /lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7fec7ed466f0] /usr/sbin/mysqld(+0x559bf3)[0x7fec80e38bf3] /usr/sbin/mysqld(+0x546de6)[0x7fec80e25de6] /usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhPh+0x80)[0x7fec80d020e0] /usr/sbin/mysqld(_ZN21Update_rows_log_event11do_exec_rowEPK14Relay_log_info+0x135)[0x7fec80d99855] /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x1c0)[0x7fec80d901a0] 130424 9:34:36 [Note] Slave I/O thread: connected to master 'replication@veso3dblx01.virtual-expo.com:3306',replication started in log 'veso3dblx01-bin.000158' at position 403083323 /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0xfd)[0x7fec80ba89ad] /usr/sbin/mysqld(handle_slave_sql+0x873)[0x7fec80ba97e3] /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fec804a7b50] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fec7edeba7d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 1 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130424 09:34:36 mysqld_safe Number of processes running now: 0 130424 09:34:36 mysqld_safe mysqld restarted 130424 9:34:37 [Note] Plugin 'FEDERATED' is disabled. 130424 9:34:37 InnoDB: The InnoDB memory heap is disabled 130424 9:34:37 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130424 9:34:37 InnoDB: Compressed tables use zlib 1.2.7 130424 9:34:37 InnoDB: Using Linux native AIO 130424 9:34:37 InnoDB: Initializing buffer pool, size = 48.0G 130424 9:34:39 InnoDB: Completed initialization of buffer pool 130424 9:34:39 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 675530664891 130424 9:34:39 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 675535907328 InnoDB: Doing recovery: scanned up to log sequence number 675541150208 InnoDB: Doing recovery: scanned up to log sequence number 675546393088 InnoDB: Doing recovery: scanned up to log sequence number 675551635968 InnoDB: Doing recovery: scanned up to log sequence number 675556878848 InnoDB: Doing recovery: scanned up to log sequence number 675562121728 InnoDB: Doing recovery: scanned up to log sequence number 675567364608 InnoDB: Doing recovery: scanned up to log sequence number 675572607488 InnoDB: Doing recovery: scanned up to log sequence number 675577850368 InnoDB: Doing recovery: scanned up to log sequence number 675583093248 InnoDB: Doing recovery: scanned up to log sequence number 675588336128 InnoDB: Doing recovery: scanned up to log sequence number 675593579008 InnoDB: Doing recovery: scanned up to log sequence number 675598821888 InnoDB: Doing recovery: scanned up to log sequence number 675604064768 InnoDB: Doing recovery: scanned up to log sequence number 675609307648 InnoDB: Doing recovery: scanned up to log sequence number 675614550528 InnoDB: Doing recovery: scanned up to log sequence number 675619793408 InnoDB: Doing recovery: scanned up to log sequence number 675625036288 InnoDB: Doing recovery: scanned up to log sequence number 675630279168 InnoDB: Doing recovery: scanned up to log sequence number 675635522048 InnoDB: Doing recovery: scanned up to log sequence number 675640764928 InnoDB: Doing recovery: scanned up to log sequence number 675646007808 InnoDB: Doing recovery: scanned up to log sequence number 675651250688 InnoDB: Doing recovery: scanned up to log sequence number 675656493568 InnoDB: Doing recovery: scanned up to log sequence number 675661736448 InnoDB: Doing recovery: scanned up to log sequence number 675666979328 InnoDB: Doing recovery: scanned up to log sequence number 675672222208 InnoDB: Doing recovery: scanned up to log sequence number 675677465088 InnoDB: Doing recovery: scanned up to log sequence number 675682707968 InnoDB: Doing recovery: scanned up to log sequence number 675687950848 InnoDB: Doing recovery: scanned up to log sequence number 675693193728 InnoDB: Doing recovery: scanned up to log sequence number 675698436608 InnoDB: Doing recovery: scanned up to log sequence number 675703679488 InnoDB: Doing recovery: scanned up to log sequence number 675708922368 InnoDB: Doing recovery: scanned up to log sequence number 675714165248 InnoDB: Doing recovery: scanned up to log sequence number 675719408128 InnoDB: Doing recovery: scanned up to log sequence number 675724651008 InnoDB: Doing recovery: scanned up to log sequence number 675729893888 InnoDB: Doing recovery: scanned up to log sequence number 675735136768 InnoDB: Doing recovery: scanned up to log sequence number 675740379648 InnoDB: Doing recovery: scanned up to log sequence number 675745622528 InnoDB: Doing recovery: scanned up to log sequence number 675750865408 InnoDB: Doing recovery: scanned up to log sequence number 675756108288 InnoDB: Doing recovery: scanned up to log sequence number 675761351168 InnoDB: Doing recovery: scanned up to log sequence number 675766594048 InnoDB: Doing recovery: scanned up to log sequence number 675771836928 InnoDB: Doing recovery: scanned up to log sequence number 675777079808 InnoDB: Doing recovery: scanned up to log sequence number 675782322688 InnoDB: Doing recovery: scanned up to log sequence number 675787565568 InnoDB: Doing recovery: scanned up to log sequence number 675792808448 InnoDB: Doing recovery: scanned up to log sequence number 675798051328 InnoDB: Doing recovery: scanned up to log sequence number 675803294208 InnoDB: Doing recovery: scanned up to log sequence number 675808537088 InnoDB: Doing recovery: scanned up to log sequence number 675813779968 InnoDB: Doing recovery: scanned up to log sequence number 675819022848 InnoDB: Doing recovery: scanned up to log sequence number 675824265728 InnoDB: Doing recovery: scanned up to log sequence number 675829508608 InnoDB: Doing recovery: scanned up to log sequence number 675834751488 InnoDB: Doing recovery: scanned up to log sequence number 675839994368 InnoDB: Doing recovery: scanned up to log sequence number 675845237248 InnoDB: Doing recovery: scanned up to log sequence number 675850480128 InnoDB: Doing recovery: scanned up to log sequence number 675855723008 InnoDB: Doing recovery: scanned up to log sequence number 675860965888 InnoDB: Doing recovery: scanned up to log sequence number 675866208768 InnoDB: Doing recovery: scanned up to log sequence number 675871451648 InnoDB: Doing recovery: scanned up to log sequence number 675876694528 InnoDB: Doing recovery: scanned up to log sequence number 675881937408 InnoDB: Doing recovery: scanned up to log sequence number 675887180288 InnoDB: Doing recovery: scanned up to log sequence number 675892423168 InnoDB: Doing recovery: scanned up to log sequence number 675897666048 InnoDB: Doing recovery: scanned up to log sequence number 675902908928 InnoDB: Doing recovery: scanned up to log sequence number 675908151808 InnoDB: Doing recovery: scanned up to log sequence number 675913394688 InnoDB: Doing recovery: scanned up to log sequence number 675918637568 InnoDB: Doing recovery: scanned up to log sequence number 675923880448 InnoDB: Doing recovery: scanned up to log sequence number 675929123328 InnoDB: Doing recovery: scanned up to log sequence number 675934366208 InnoDB: Doing recovery: scanned up to log sequence number 675939609088 InnoDB: Doing recovery: scanned up to log sequence number 675944851968 InnoDB: Doing recovery: scanned up to log sequence number 675950094848 InnoDB: Doing recovery: scanned up to log sequence number 675955337728 InnoDB: Doing recovery: scanned up to log sequence number 675960580608 InnoDB: Doing recovery: scanned up to log sequence number 675965823488 InnoDB: Doing recovery: scanned up to log sequence number 675971066368 InnoDB: Doing recovery: scanned up to log sequence number 675976309248 InnoDB: Doing recovery: scanned up to log sequence number 675981552128 InnoDB: Doing recovery: scanned up to log sequence number 675986795008 InnoDB: Doing recovery: scanned up to log sequence number 675992037888 InnoDB: Doing recovery: scanned up to log sequence number 675997280768 InnoDB: Doing recovery: scanned up to log sequence number 676002523648 InnoDB: Doing recovery: scanned up to log sequence number 676007766528 InnoDB: Doing recovery: scanned up to log sequence number 676013009408 InnoDB: Doing recovery: scanned up to log sequence number 676018252288 InnoDB: Doing recovery: scanned up to log sequence number 676023495168 InnoDB: Doing recovery: scanned up to log sequence number 676028738048 InnoDB: Doing recovery: scanned up to log sequence number 676033980928 InnoDB: Doing recovery: scanned up to log sequence number 676039223808 InnoDB: Doing recovery: scanned up to log sequence number 676044466688 InnoDB: Doing recovery: scanned up to log sequence number 676049709568 InnoDB: Doing recovery: scanned up to log sequence number 676054952448 InnoDB: Doing recovery: scanned up to log sequence number 676060195328 InnoDB: Doing recovery: scanned up to log sequence number 676065438208 InnoDB: Doing recovery: scanned up to log sequence number 676070681088 InnoDB: Doing recovery: scanned up to log sequence number 676075923968 InnoDB: Doing recovery: scanned up to log sequence number 676081166848 InnoDB: Doing recovery: scanned up to log sequence number 676086409728 InnoDB: Doing recovery: scanned up to log sequence number 676091652608 InnoDB: Doing recovery: scanned up to log sequence number 676096895488 InnoDB: Doing recovery: scanned up to log sequence number 676102138368 InnoDB: Doing recovery: scanned up to log sequence number 676107381248 InnoDB: Doing recovery: scanned up to log sequence number 676112624128 InnoDB: Doing recovery: scanned up to log sequence number 676117867008 InnoDB: Doing recovery: scanned up to log sequence number 676123109888 InnoDB: Doing recovery: scanned up to log sequence number 676128352768 InnoDB: Doing recovery: scanned up to log sequence number 676133595648 InnoDB: Doing recovery: scanned up to log sequence number 676138838528 InnoDB: Doing recovery: scanned up to log sequence number 676144081408 InnoDB: Doing recovery: scanned up to log sequence number 676149324288 InnoDB: Doing recovery: scanned up to log sequence number 676154567168 InnoDB: Doing recovery: scanned up to log sequence number 676159810048 InnoDB: Doing recovery: scanned up to log sequence number 676165052928 InnoDB: Doing recovery: scanned up to log sequence number 676170295808 InnoDB: Doing recovery: scanned up to log sequence number 676175538688 InnoDB: Doing recovery: scanned up to log sequence number 676180781568 InnoDB: Doing recovery: scanned up to log sequence number 676186024448 InnoDB: Doing recovery: scanned up to log sequence number 676191267328 InnoDB: Doing recovery: scanned up to log sequence number 676196510208 InnoDB: Doing recovery: scanned up to log sequence number 676201753088 InnoDB: Doing recovery: scanned up to log sequence number 676206995968 InnoDB: Doing recovery: scanned up to log sequence number 676212238848 InnoDB: Doing recovery: scanned up to log sequence number 676217481728 InnoDB: Doing recovery: scanned up to log sequence number 676222724608 InnoDB: Doing recovery: scanned up to log sequence number 676227967488 InnoDB: Doing recovery: scanned up to log sequence number 676233210368 InnoDB: Doing recovery: scanned up to log sequence number 676238453248 InnoDB: Doing recovery: scanned up to log sequence number 676243696128 InnoDB: Doing recovery: scanned up to log sequence number 676248939008 InnoDB: Doing recovery: scanned up to log sequence number 676254181888 InnoDB: Doing recovery: scanned up to log sequence number 676259424768 InnoDB: Doing recovery: scanned up to log sequence number 676264667648 InnoDB: Doing recovery: scanned up to log sequence number 676269910528 InnoDB: Doing recovery: scanned up to log sequence number 676275153408 InnoDB: Doing recovery: scanned up to log sequence number 676278161570 130424 9:34:59 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 1014005658, file name /mydata/binlog/veso3dblx02-bin.000139 130424 9:35:10 InnoDB: Waiting for the background threads to start 130424 9:35:11 InnoDB: 5.5.30 started; log sequence number 676278161570 130424 9:35:11 [Note] Recovering after a crash using /mydata/binlog/veso3dblx02-bin 130424 9:35:11 [Note] Starting crash recovery... 130424 9:35:11 [Note] Crash recovery finished. 130424 9:35:11 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306 130424 9:35:11 [Note] - '0.0.0.0' resolves to '0.0.0.0'; 130424 9:35:11 [Note] Server socket created on IP: '0.0.0.0'. 130424 9:35:11 [Warning] 'user' entry 'root@veso3rdblx01' ignored in --skip-name-resolve mode. 130424 9:35:11 [Warning] 'user' entry '@veso3rdblx01' ignored in --skip-name-resolve mode. 130424 9:35:11 [Warning] 'proxies_priv' entry '@ root@veso3rdblx01' ignored in --skip-name-resolve mode. 130424 9:35:11 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem. 130424 9:35:11 [Note] Event Scheduler: Loaded 0 events 130424 9:35:11 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.30-1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) 130424 9:35:11 [Note] Slave SQL thread initialized, starting replication in log 'veso3dblx01-bin.000158' at position 403016942, relay log './mysqld-relay-bin.000429' position: 403017094 130424 9:35:11 InnoDB: Assertion failure in thread 140079148250880 in file row0mysql.c line 1468 InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be I