MySQL Bugs: #770: mysqld-4.0.13 server crash on linux

Bug #770	mysqld-4.0.13 server crash on linux
Submitted:	1 Jul 2003 3:19	Modified:	24 Sep 2003 4:17
Reporter:	Martin Mokrejs	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: MyISAM storage engine	Severity:	S2 (Serious)
Version:	4.0.13	OS:	Linux (Linux 2.4.19)
Assigned to:	Sergei Golubchik	CPU Architecture:	Any

Description:
0x80702cb handle_segfault + 447
0x8282488 pthread_sighandler + 184
0x8283a23 __pthread_unlock + 147
0x8280ca4 pthread_mutex_unlock + 164
0x827ed89 pthread_cond_wait + 345
0x80d028f safe_remove_from_cache__FP3THDP8st_table + 79
0x80d1636 mysql_alter_table__FP3THDPcT1P24st_ha_create_informationP13st_table_listRt4List1Z12create_fieldRt4List1Z3KeyRt4List1Z10Alter_dr + 838
0x807b487 mysql_execute_command__Fv + 4563
0x807e166 mysql_parse__FP3THDPcUi + 146
0x80797ad dispatch_command__F19enum_server_commandP3THDPcUi + 1461
0x80791ed do_command__FP3THD + 149
0x8078a0f handle_one_connection + 631
0x827fc3c pthread_start_thread + 220
0x82b53fa thread_start + 4

How to repeat:
This bug was hit on our servers several times as I see now in errorlog. The queries(when listed in crashdump) were:

thd->query at 0x14dcee30 = select manual from blast where prot_data_id=1789
thd->thread_id=55201

thd->query at 0x84f9f98 = /*!40000 ALTER TABLE known3d ENABLE KEYS */
thd->thread_id=8

thd->query at 0x12770ff8 = /*!40000 ALTER TABLE blast ENABLE KEYS */
thd->thread_id=32359

I can repeat the fix with one table:

mysql> check table contig_data;
ERROR 2013: Lost connection to MySQL server during query
mysql>

Please download the testtables from http://pedant.gsf.de/Caenorhabditis_elegans_WS75.tar.gz
as I am unable to connect to support.mysql.com via ftp.

Thank you for your bug report.

This bug looks definitely like the one fixed in 4.0.14, which should be out in 7 - 10 days.

Try out our 4.0.14 binary when out and if it does not solve a problem, let us know.

No, it is still present in 4.0.14-standard official binaries:

mysql@vrapenec mysql $ PATH=/usr/local/mysql-standard-4.0.14-pc-linux-i686/bin:$PATH mysql -h127.0.0.1 -upedant -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8 to server version: 4.0.14-debug-debug

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use Caenorhabditis_elegans_WS75
Database changed
mysql> check table contig_data;
ERROR 2013: Lost connection to MySQL server during query
mysql> 030828 11:04:53  mysqld ended

mysql> quit
Bye
mysql@vrapenec mysql $ 

# /usr/local/mysql-standard-4.0.14-pc-linux-i686/bin/resolve_stack_dump -s ./symbols -n ./cr
0x80731ef set_maximum_open_files__FUi + 115
0x82b27c8 strncmp + 104
0x826d354 my_fopen + 276
0x826d1f7 strmake_root + 67
0x826c6d6 my_b_vprintf + 274
0x827d398 sdissect + 944
0x80d1e5d create_table_from_items__FP3THDP24st_ha_create_informationPCcT2Pt4List1Z12create_fieldPt4List1Z3KeyPt4List1Z4ItemPP13st_mysql_l + 441
0x80e2144 init_cache__11Query_cache + 416
0x80e2708 append_result_data__11Query_cachePP17Query_cache_blockUlPcP17Query_cache_block + 32
0x807fd92 append_file_to_dir__FP3THDPPcPc + 138
0x80829f5 yyparse__Fv + 10989
0x807db21 mysql_execute_command__Fv + 12941
0x807d503 mysql_execute_command__Fv + 11375
0x807cb2e mysql_execute_command__Fv + 8858
0x82aff7c chunk_free + 508
0x82e588a __strtof_internal + 250

I've started mysql from rootshell with open files limit 1024. I do not see the usual warning that mysqld could not setrlimit the number of file to a higher value.

I tried then:

# ulimit -a; PATH=/usr/local/mysql-standard-4.0.14-pc-linux-i686/bin:$PATH //usr/local/mysql-standard-4.0.14-pc-linux-i686/support-files/mysql.server start
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 30000
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 7168
virtual memory        (kbytes, -v) unlimited
vrapenec root # Starting mysqld daemon with databases from /var/lib/mysql

vrapenec root # 030828 11:15:00  mysqld ended

vrapenec root # 

mysqld crashed again on check table contig_data from a client. Tested on linux 2.4.22-pre7.

Warning: Ignoring user change to 'mysql' because the user was set to 'mysql' earlier on the command line
030828 11:13:42  Warning: Asked for 196608 thread stack, but got 126976
030828 11:13:42  /usr/local/mysql/bin/mysqld: Can't create/write to file '/usr/local/mysql/data/vrapenec.gsf.de.pid' (Errcode: 13)
/usr/local/mysql/bin/mysqld: ready for connections.
Version: '4.0.14-debug-debug'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306
mysqld got signal 11;

[cut]

Server has defined:

[mysqld]
set-variable    = open_files_limit=30000

It's mysqld/myisamchk crash on corrupted compressed table
MyISAM generally doesn't handle them nicely, it assumes that compressed tables - as they are readonly - cannot be corrupted (unless it's a hardware problem, and you cannot trust mysqld binary either).

Anyway I fixed this particular crash.

Hi,
  I aggree the server does not crash on repair, but I still cannot repair the table. It seem s like an endless loop:

$ cd Caenorhabditis_elegans_WS75
$ cp contig_data.MYD contig_data.MYD-

mysql> truncate table contig_data;

$ mysqladmin flush-tables
$ myisampack contig_data
$ cp contig_data.MYD- contig_data.MYD
$ /usr/local/mysql-standard-4.0.15-pc-linux-i686/bin/myisamchk -v -v -v -o contig_data
- recovering (with keycache) MyISAM-table 'contig_data'
Data records: 0
Found wrong record at 144

and I can wait forever. Same with "-r".

Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

There was a fix for this in 4.0.15 that temporary solved this problem, but the fix caused myisamchk to fail for other tables.
The correct fix will be available in 4.0.16
(Fix tested with table in this bug report and the table given in bug 1304)