Description:
Hi,
1) Recently convert db to innodb.
2) Flushed tables with read lock and then moved data directory to another partition. T
3) Created symlink
4) Unlock Tables
5) 3 days later MySQL Crashes and will not start.
-- Problem points the newly converted/symlinked db.
6) Got system backup by dropping and recreating on of the tabls InnoDB was compling about.
7) When trying to check/repair/dump or select MySQL crashes.
8) I could not get a strack trace on the current system so I copied the innodb data file and the indivdual .ibd files to dev system (running 5.0.38 x86). I was able to reproduce the problem and run a strack trace as follows:
9) i HA
Version: '5.0.42-enterprise-log' socket: '/tmp/mysql.sock' port: 3306 MySQL Enterprise Server (Commercial)
071122 2:01:00InnoDB: Assertion failure in thread 40971 in file btr0pcur.c line 403
InnoDB: Failing assertion: page_is_comp(next_page) == page_is_comp(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
071122 2:01:00 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=524288000
read_buffer_size=8384512
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2149999 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x8c35d98
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbf85d458, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80a4dd3
0x830c7c8
0x82039c8
0x81dbe64
0x8149024
0x81492bf
0x813a2bf
0x8157012
0x815628f
0x80b82f6
0x80bd6fc
0x80b5618
0x80b4eb4
0x80b43db
0x8309f7c
0x835902a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Reso
lved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8c82210 = ALTER TABLE tasklist_extended_values type=myisam
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
//
// Stack Strace
//
0x80a4dd3 handle_segfault + 511
0x830c7c8 pthread_sighandler + 184
0x82039c8 btr_pcur_move_to_next_page + 644
0x81dbe64 row_search_for_mysql + 9504
0x8149024 general_fetch__11ha_innobasePcUiUi + 72
0x81492bf rnd_next__11ha_innobasePc + 95
0x813a2bf rr_sequential__FP14st_read_record + 123
0x8157012 copy_data_between_tables__FP8st_tableT0Rt4List1Z12create_fieldbUiP8st_orderPUxT622enum_enable_or_disable + 1342
0x815628f mysql_alter_table__FP3THDPcT1P24st_ha_create_informationP13st_table_listP10Alter_infoUiP8st_orderb + 7015
0x80b82f6 mysql_execute_command__FP3THD + 5810
0x80bd6fc mysql_parse__FP3THDPcUi + 204
0x80b5618 dispatch_command__F19enum_server_commandP3THDPcUi + 1880
0x80b4eb4 do_command__FP3THD + 204
0x80b43db handle_one_connection + 819
0x8309f7c pthread_start_thread + 220
0x835902a thread_start + 4
================================================================================
Original mysql.err from the initial crash:
InnoDB: Page directory corruption: supremum not pointed to
071121 8:31:50 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 0000 (NOTE: hex dump is all zeros)
...
071121 8:31:50 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page directory corruption: supremum not pointed to
071121 8:31:50 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 00000000000000
071121 8:31:50 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
071121 8:31:50InnoDB: Assertion failure in thread 1142479168 in file rem0rec.c line 339
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
071121 8:31:50 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=3221225472
read_buffer_size=2093056
max_used_connections=185
max_connections=200
threads_connected=28
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 3964126 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x2aac992d5510
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x4418d068, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Reso
lved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x157f0220 = SELECT SUM(text.value) FROM tasks t LEFT JOIN tasklist_extended_values text ON text.task_id=t.task_id WHE
RE t.regard_type='patient' AND t.regarding=8 AND text.ext_name_id=8 AND t.complete_date>=CONCAT(LEFT(DATE_SUB(NOW(), INTERVAL 1 MONT
H), 7), '-01')
thd->thread_id=8087359
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
Number of processes running now: 0
How to repeat:
Working on this.
Suggested fix:
Not crash when moving to next page?
