Bug #20979 5.1 crashes like a drunk on a bike
Submitted: 11 Jul 2006 23:03 Modified: 23 Aug 2006 18:11
Reporter: Jim Winstead Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.1.11 OS:Linux (FC3, 2.6.12)
Assigned to: Assigned Account CPU Architecture:Any
Tags: crash, innodb

[11 Jul 2006 23:03] Jim Winstead
Description:
bugs.mysql.com is running on 5.1. it falls down a lot.

How to repeat:
use 5.1 on a production site. here are some of the stack dumps:

0x81b1877 handle_segfault + 423
0xb77420 (?)
(nil)
0x8263e62 index_read__11ha_innobasePcPCcUi16ha_rkey_function + 450
0x8209529 join_read_key__FP13st_join_table + 169
0x82088a0 sub_select__FP4JOINP13st_join_tableb + 208
0x8208a59 evaluate_join_record__FP4JOINP13st_join_tableiPc + 377
0x82088ac sub_select__FP4JOINP13st_join_tableb + 220
0x8208590 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 528
0x81f98bb exec__4JOIN + 1403
0x81fac87 mysql_select__FP3THDPPP4ItemP13st_table_listUiRt4List1Z4ItemP4ItemUiP8st_orderT7T5T7UlP13select_resultP18st_select_lex_unitP13s + 887
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command__FP3THD + 1539
0x81cd288 mysql_parse__FP3THDPcUi + 328
0x81c4adf dispatch_command__F19enum_server_commandP3THDPcUi + 1903
0x81c435e do_command__FP3THD + 190
0x81c3864 handle_one_connection + 772

0x81b1877 handle_segfault + 423
0x74e420 (?)
0x793d3b0 (?)
0x819f044 store__15Field_varstringPCcUiP15charset_info_st + 84
0x8295855 do_field_string__FP10Copy_field + 117
0x82955a3 do_outer_field_null__FP10Copy_field + 83
0x820e039 copy_fields__FP15TMP_TABLE_PARAM + 41
0x820aa3d end_write_group__FP4JOINP13st_join_tableb + 461
0x8208a59 evaluate_join_record__FP4JOINP13st_join_tableiPc + 377
0x82088ac sub_select__FP4JOINP13st_join_tableb + 220
0x8208a59 evaluate_join_record__FP4JOINP13st_join_tableiPc + 377
0x82088ac sub_select__FP4JOINP13st_join_tableb + 220
0x8208a59 evaluate_join_record__FP4JOINP13st_join_tableiPc + 377
0x82088ac sub_select__FP4JOINP13st_join_tableb + 220
0x8208a59 evaluate_join_record__FP4JOINP13st_join_tableiPc + 377
0x82088ac sub_select__FP4JOINP13st_join_tableb + 220
0x8208a59 evaluate_join_record__FP4JOINP13st_join_tableiPc + 377
0x82088ac sub_select__FP4JOINP13st_join_tableb + 220
0x8208590 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 528
0x81f98bb exec__4JOIN + 1403
0x81fac87 mysql_select__FP3THDPPP4ItemP13st_table_listUiRt4List1Z4ItemP4ItemUiP8st_orderT7T5T7UlP13select_resultP18st_select_lex_unitP13s + 887
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command__FP3THD + 1539
0x81cd288 mysql_parse__FP3THDPcUi + 328
0x81c4adf dispatch_command__F19enum_server_commandP3THDPcUi + 1903
0x81c435e do_command__FP3THD + 190
0x81c3864 handle_one_connection + 772

0x81b1877 handle_segfault + 423
0x2c3420 (?)
0xaae8ab8 _end + 37032880
0x81fba77 make_join_statistics__FP4JOINP13st_table_listP4ItemP16st_dynamic_array + 3271
0x81f7f1d optimize__4JOIN + 1165
0x81fac2e mysql_select__FP3THDPPP4ItemP13st_table_listUiRt4List1Z4ItemP4ItemUiP8st_orderT7T5T7UlP13select_resultP18st_select_lex_unitP13s + 798
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command__FP3THD + 1539
0x81cd288 mysql_parse__FP3THDPcUi + 328
0x81c4adf dispatch_command__F19enum_server_commandP3THDPcUi + 1903
0x81c435e do_command__FP3THD + 190
0x81c3864 handle_one_connection + 772

0x81b1877 handle_segfault + 423
0x768420 (?)
(nil)
0x81ed2ec open_and_lock_tables__FP3THDP13st_table_list + 28
0x82128bd mysql_insert__FP3THDP13st_table_listRt4List1Z4ItemRt4List1Zt4List1Z4ItemT2T215enum_duplicatesb + 429
0x81c7f7b mysql_execute_command__FP3THD + 8027
0x81cd288 mysql_parse__FP3THDPcUi + 328
0x81c4adf dispatch_command__F19enum_server_commandP3THDPcUi + 1903
0x81c435e do_command__FP3THD + 190
0x81c3864 handle_one_connection + 772
[12 Jul 2006 15:05] Jim Winstead
some of the crashes since the bug was filed (and with better traces after using nm --demangle):

0x81b1877 handle_segfault + 423
0x4da420 (?)
(nil)
0x81ed2ec open_and_lock_tables(THD *, st_table_list *) + 28
0x81c6449 mysql_execute_command(THD *) + 1065
0x81cd288 mysql_parse(THD *, char *, unsigned int) + 328
0x81c4adf dispatch_command(enum_server_command, THD *, char *, unsigned int) + 1903
0x81c435e do_command(THD *) + 190
0x81c3864 handle_one_connection + 772
0x50c341 (?)
0x2cefee (?)

0x81b1877 handle_segfault + 423
0xdd6420 (?)
0x1 (?)
0x8263e62 ha_innobase::index_read(char *, char const *, unsigned int, ha_rkey_function) + 450
0x8209529 join_read_key(st_join_table *) + 169
0x82088a0 sub_select(JOIN *, st_join_table *, bool) + 208
0x8208a59 evaluate_join_record(JOIN *, st_join_table *, int, char *) + 377
0x82088ac sub_select(JOIN *, st_join_table *, bool) + 220
0x8208590 do_select(JOIN *, List<Item> *, st_table *, Procedure *) + 528
0x81fa793 JOIN::exec(void) + 5203
0x81fac87 mysql_select(THD *, Item ***, st_table_list *, unsigned int, List<Item> &, Item *, unsigned int, st_order *, st_order *, Item * + 887
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command(THD *) + 1539
0x81cd288 mysql_parse(THD *, char *, unsigned int) + 328
0x81c4adf dispatch_command(enum_server_command, THD *, char *, unsigned int) + 1903
0x81c435e do_command(THD *) + 190
0x81c3864 handle_one_connection + 772
0x50c341 (?)
0x2cefee (?)

0x81b1877 handle_segfault + 423
0x39c420 (?)
0x84f0931 TMP_TABLE_PARAM::cleanup(void) + 49
0x84ee97b Field_blob::free(void) + 43
0x8207e41 free_tmp_table(THD *, st_table *) + 193
0x82ba566 st_select_lex_unit::cleanup(void) + 102
0x82b89f4 mysql_union(THD *, st_lex *, select_result *, st_select_lex_unit *, unsigned long) + 100
0x81f707f Letext + 79
0x81c6623 mysql_execute_command(THD *) + 1539
0x81cd288 mysql_parse(THD *, char *, unsigned int) + 328
0x81c4adf dispatch_command(enum_server_command, THD *, char *, unsigned int) + 1903
0x81c435e do_command(THD *) + 190
0x81c3864 handle_one_connection + 772
0x50c341 (?)
0x2cefee (?)

0x81b1877 handle_segfault + 423
0x94b420 (?)
0x75d450 (?)
0x819f044 Field_varstring::store(char const *, unsigned int, charset_info_st *) + 84
0x8295855 do_field_string(Copy_field *) + 117
0x82955a3 do_outer_field_null(Copy_field *) + 83
0x820e039 copy_fields(TMP_TABLE_PARAM *) + 41
0x820aa3d end_write_group(JOIN *, st_join_table *, bool) + 461
0x8208bf0 evaluate_null_complemented_join_record(JOIN *, st_join_table *) + 288
0x82088ce sub_select(JOIN *, st_join_table *, bool) + 254
0x8208a59 evaluate_join_record(JOIN *, st_join_table *, int, char *) + 377
0x82088ac sub_select(JOIN *, st_join_table *, bool) + 220
0x8208a59 evaluate_join_record(JOIN *, st_join_table *, int, char *) + 377
0x82088ac sub_select(JOIN *, st_join_table *, bool) + 220
0x8208a59 evaluate_join_record(JOIN *, st_join_table *, int, char *) + 377
0x82088ac sub_select(JOIN *, st_join_table *, bool) + 220
0x8208590 do_select(JOIN *, List<Item> *, st_table *, Procedure *) + 528
0x81f98bb JOIN::exec(void) + 1403
0x81fac87 mysql_select(THD *, Item ***, st_table_list *, unsigned int, List<Item> &, Item *, unsigned int, st_order *, st_order *, Item * + 887
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command(THD *) + 1539
0x81cd288 mysql_parse(THD *, char *, unsigned int) + 328
0x81c4adf dispatch_command(enum_server_command, THD *, char *, unsigned int) + 1903
0x81c435e do_command(THD *) + 190
0x81c3864 handle_one_connection + 772
0x50c341 (?)
0x2cefee (?)

0x81b1877 handle_segfault + 423
0xfac420 (?)
(nil)
0x26327a (?)
0x269abf (?)
0x269e3a (?)
0x8468e16 my_no_flags_free + 22
0x846951c free_root + 124
0x8207e80 free_tmp_table(THD *, st_table *) + 256
0x81fa896 JOIN::destroy(void) + 182
0x82ba6d1 st_select_lex::cleanup(void) + 33
0x81facf1 mysql_select(THD *, Item ***, st_table_list *, unsigned int, List<Item> &, Item *, unsigned int, st_order *, st_order *, Item * + 993
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command(THD *) + 1539
0x81cd288 mysql_parse(THD *, char *, unsigned int) + 328
0x81c4adf dispatch_command(enum_server_command, THD *, char *, unsigned int) + 1903
0x81c435e do_command(THD *) + 190
0x81c3864 handle_one_connection + 772
0x50c341 (?)
0x2cefee (?)

0x81b1877 handle_segfault + 423
0xb5b420 (?)
0x10000 (?)
0x81abe79 Protocol::send_fields(List<Item> *, unsigned int) + 233
0x81a6f2f select_send::send_fields(List<Item> &, unsigned int) + 47
0x81fa77d JOIN::exec(void) + 5181
0x81fac87 mysql_select(THD *, Item ***, st_table_list *, unsigned int, List<Item> &, Item *, unsigned int, st_order *, st_order *, Item * + 887
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command(THD *) + 1539
0x81cd288 mysql_parse(THD *, char *, unsigned int) + 328
0x81c4adf dispatch_command(enum_server_command, THD *, char *, unsigned int) + 1903
0x81c435e do_command(THD *) + 190
0x81c3864 handle_one_connection + 772
0x50c341 (?)
0x2cefee (?)

0x81b1877 handle_segfault + 423
0x382420 (?)
0x475b450 (?)
0x819f044 Field_varstring::store(char const *, unsigned int, charset_info_st *) + 84
0x8295855 do_field_string(Copy_field *) + 117
0x82955a3 do_outer_field_null(Copy_field *) + 83
0x820e039 copy_fields(TMP_TABLE_PARAM *) + 41
0x820aa3d end_write_group(JOIN *, st_join_table *, bool) + 461
0x8208bf0 evaluate_null_complemented_join_record(JOIN *, st_join_table *) + 288
0x82088ce sub_select(JOIN *, st_join_table *, bool) + 254
0x8208a59 evaluate_join_record(JOIN *, st_join_table *, int, char *) + 377
0x82088ac sub_select(JOIN *, st_join_table *, bool) + 220
0x8208a59 evaluate_join_record(JOIN *, st_join_table *, int, char *) + 377
0x82088ac sub_select(JOIN *, st_join_table *, bool) + 220
0x8208a59 evaluate_join_record(JOIN *, st_join_table *, int, char *) + 377
0x82088ac sub_select(JOIN *, st_join_table *, bool) + 220
0x8208590 do_select(JOIN *, List<Item> *, st_table *, Procedure *) + 528
0x81f98bb JOIN::exec(void) + 1403
0x81fac87 mysql_select(THD *, Item ***, st_table_list *, unsigned int, List<Item> &, Item *, unsigned int, st_order *, st_order *, Item * + 887
0x81f7111 Letext + 225
0x81c6623 mysql_execute_command(THD *) + 1539
0x81cd288 mysql_parse(THD *, char *, unsigned int) + 328
0x81c4adf dispatch_command(enum_server_command, THD *, char *, unsigned int) + 1903
0x81c435e do_command(THD *) + 190
0x81c3864 handle_one_connection + 772
0x50c341 (?)
0x2cefee (?)
[13 Jul 2006 18:07] Jim Winstead
another crash after upgrading to the dynamic generic RPMs:

[jwinstead2@bugs ~]$ resolve_stack_dump -s mysqld.sym -n stack 0x81d9210 handle_segfault + 416
0x9c5420 (?)
0xaaefba8 _end + 37393304
0x81d4047 mysql_lock_tables(THD*, st_table**, unsigned int, unsigned int, bool*) + 55
0x8216f77 lock_tables(THD*, st_table_list*, unsigned int, bool*) + 391
0x8216ca9 open_and_lock_tables(THD*, st_table_list*) + 89
0x81f216d mysql_execute_command(THD*) + 765
0x81f9abb mysql_parse(THD*, char*, unsigned int) + 347
0x81f07e3 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 1123
0x81f032d do_command(THD*) + 141
0x81ef85f handle_one_connection + 655
0x50c341 (?)
0x2cefee (?)
[14 Jul 2006 9:43] Valeriy Kravchuk
Thank you for a problem report. Please, specify what exact SQL statements lead to that crashes (if you can identify them). We have to create a repeatable (and, better, simple) test case(s) for this bug9s) to be fixed.
[18 Jul 2006 22:41] Jim Winstead
SELECT SQL_CALC_FOUND_ROWS
                  bugdb.*,
                  UNIX_TIMESTAMP(ts1) AS entered,
                  UNIX_TIMESTAMP(ts2) AS modified,
                  UNIX_TIMESTAMP(eta) AS eta,
                  TO_DAYS(NOW())-TO_DAYS(ts2) AS unchanged,
                  CONCAT(user.firstname,' ',user.lastname)
                    AS assign_name,
                  user.firstname, user.lastname,
                  user.email,
                  CONCAT(u2.firstname, ' ', u2.lastname) AS lead_name,
                  u2.firstname AS lead_firstname,
                  u2.lastname AS lead_lastname,
                  u2.email AS lead_email,
                  1 AS relevance
             FROM bugdb
             LEFT JOIN user
                    ON bugdb.assign = user.user_id AND bugdb.assign != 0
             LEFT JOIN user AS u2
                    ON bugdb.lead = u2.user_id AND bugdb.lead != 0
          WHERE 1 AND status='Need Doc Info'
 AND severity != 4  GROUP BY id ORDER BY relevance DESC LIMIT 0,10

bugdb:

CREATE TABLE `bugdb` (
  `id` int(8) NOT NULL AUTO_INCREMENT,
  `private` tinyint(1) NOT NULL DEFAULT '0',
  `bug_type` varchar(32) DEFAULT NULL,
  `reporter` int(11) NOT NULL DEFAULT '0',
  `ip` int(10) unsigned NOT NULL DEFAULT '0',
  `sdesc` varchar(80) NOT NULL DEFAULT '',
  `ldesc` mediumtext NOT NULL,
  `suggestedfix` mediumtext NOT NULL,
  `howtorepeat` mediumtext NOT NULL,
  `php_version` varchar(100) DEFAULT NULL,
  `php_os` varchar(100) NOT NULL DEFAULT '',
  `status` varchar(20) DEFAULT NULL,
  `severity` tinyint(4) NOT NULL DEFAULT '0',
  `priority` tinyint(4) NOT NULL DEFAULT '0',
  `ts1` datetime DEFAULT NULL,
  `ts2` datetime DEFAULT NULL,
  `eta` datetime DEFAULT NULL,
  `assign` int(10) unsigned NOT NULL DEFAULT '0',
  `reviewer` int(10) unsigned NOT NULL DEFAULT '0',
  `verifier` int(10) unsigned DEFAULT NULL,
  `lead` int(10) unsigned NOT NULL DEFAULT '0',
  `showstopper` tinyint(4) NOT NULL DEFAULT '0',
  `notinreleasedversions` tinyint(4) NOT NULL DEFAULT '0',
  `affectscustomer` tinyint(4) NOT NULL DEFAULT '0',
  `affectedissues` varchar(255) NOT NULL DEFAULT '',
  `mysqlnetwork` tinyint(4) DEFAULT NULL,
  `qatestneeded` tinyint(1) DEFAULT NULL,
  `changeset` varchar(255) DEFAULT NULL,
  `metrics_ref` varchar(40) DEFAULT NULL,
  `internal_sev` tinyint(3) unsigned DEFAULT NULL,
  `internal_order` tinyint(3) unsigned DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `php_version` (`php_version`(1)),
  KEY `severity` (`severity`),
  KEY `bug_type` (`bug_type`),
  KEY `status` (`status`),
  KEY `internal_sev` (`internal_sev`),
  KEY `priority` (`priority`),
  KEY `internal_order` (`internal_order`),
  FULLTEXT KEY `email` (`sdesc`,`ldesc`),
  FULLTEXT KEY `metrics_ref` (`metrics_ref`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

user:

CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `bugs`.`user` AS select `web`.`user`.`user_id` AS `user_id`,`web`.`user`.`email` AS `email`,`web`.`user`.`firstname` AS `firstname`,`web`.`user`.`lastname` AS `lastname`,((if(`bugs`.`user_details`.`developer`,32,0) | if(`bugs`.`user_details`.`reviewer`,64,0)) | if(`bugs`.`user_details`.`lead`,128,0)) AS `gid` from (`web`.`user` left join `bugs`.`user_details` on((`web`.`user`.`user_id` = `bugs`.`user_details`.`user`)))

stack trace:

0x81d9210 handle_segfault + 416
0x41d420 (?)
0x20 (?)
0x8216c88 open_and_lock_tables(THD*, st_table_list*) + 56
0x81f216d mysql_execute_command(THD*) + 765
0x81f9abb mysql_parse(THD*, char*, unsigned int) + 347
0x81f07e3 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 1123
0x81f032d do_command(THD*) + 141
0x81ef85f handle_one_connection + 655
0xaba341 (?)
0x9a46fe (?)

of course, it does not happen every time this query is run.
[19 Jul 2006 13:37] Petr Chardin
Some table definitions are definitely missing e.g. user table,
used in the query.
[25 Jul 2006 18:23] Jim Winstead
here's a different backtrace:

0x81d9210 handle_segfault + 416
0xf36420 (?)
0x828eb24 handler::drop_table(char const*) + 20
0x84261e8 mi_close + 680
0x828eb24 handler::drop_table(char const*) + 20
0x82336c7 free_tmp_table(THD*, st_table*) + 151
0x82f967c st_select_lex_unit::cleanup() + 204
0x82f807d mysql_union(THD*, st_lex*, select_result*, st_select_lex_unit*, unsigned long) + 93
0x8221e87 handle_select(THD*, st_lex*, select_result*, unsigned long) + 71
0x81f21cd mysql_execute_command(THD*) + 861
0x81f9abb mysql_parse(THD*, char*, unsigned int) + 347
0x81f07e3 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 1123
0x81f032d do_command(THD*) + 141
0x81ef85f handle_one_connection + 655
0xaba341 (?)
0x1d86fe (?)

query:

(SELECT id, UNIX_TIMESTAMP(ts) AS entered, user, comment, private,
                         email, CONCAT(firstname, ' ', lastname) AS name,
                     NULL AS file_fname, NULL AS file_ctype, 0 AS file_len
                    FROM bugdb_comments
                LEFT JOIN user ON user = user_id
               WHERE bug = '12720')
                 UNION
             (SELECT id, UNIX_TIMESTAMP(ts) AS entered, user,
                     description AS comment, private, email,
                     CONCAT(firstname, ' ', lastname) AS name,
                         file_fname, file_ctype, length(file_content) AS file_len
                    FROM bugdb_files
                LEFT JOIN user ON user = user_id
               WHERE bug = '12720')
             ORDER BY entered
[3 Aug 2006 21:46] Heikki Tuuri
Hi!

This might be the same bug as http://bugs.mysql.com/bug.php?id=20213

Please test the patch I posted there.

Regards,

Heikki
[19 Aug 2006 0:22] Jim Winstead
We're running a version of MySQL 5.1.11 with Heikki's patch on bugs.mysql.com now. We'll see how it goes.
[23 Aug 2006 18:11] Jim Winstead
The patch for Bug #20213 does appear to fix this problem, so I'm marking this bug as a duplicate of that one.
[23 Aug 2006 18:17] Heikki Tuuri
Jim,

thank you. I apologize that we introduced the ultra-critical memory corruption bug to ha_innodb.cc in 5.1.11. Jan was leaving at that time and I was traveling at the MySQL UC. Those are the excuses for this bug.

Regards,

Heikki