Bug #46941 crash with lower_case_table_names=2 and foreign key data dictionary confusion
Submitted: 26 Aug 18:23 Modified: 18 Nov 12:27
Reporter: Shane Bester
Status: In progress
Category:Server Severity:S1 (Critical)
Version:5.1.30, 5.1.39 OS:Microsoft Windows ((macosx, windows))
Assigned to: Lars-Erik Bjørk Target Version:5.1+
Tags: regression
Triage: Triaged: D1 (Critical)

[26 Aug 18:23] Shane Bester
Description:
When adding/dropping foreign keys and lower_case_table_names=2, server can crash.  This
happens on file-systems with case insensitive names.

Version: '5.1.39-debug'  socket: ''  port: 3306  Source distribution
090826 18:15:14 [ERROR] Table xy/activewfset contains fewer indexes inside InnoDB than
are defined in the MySQL .frm file. Have you mi

090826 18:15:14 [ERROR] Innodb could not find key n:o 2 with name FK2439AFAD7BCE1348 from
dict cache for table xy/xxxxxxx

mysqld.exe!dict_index_contains_col_or_prefix()[dict0dict.c:543]
mysqld.exe!build_template()[ha_innodb.cc:3441]
mysqld.exe!ha_innobase::index_read()[ha_innodb.cc:4506]
mysqld.exe!ha_innobase::index_first()[ha_innodb.cc:4809]
mysqld.exe!join_read_first()[sql_select.cc:11855]
mysqld.exe!sub_select()[sql_select.cc:11160]
mysqld.exe!do_select()[sql_select.cc:10917]
mysqld.exe!JOIN::exec()[sql_select.cc:2213]
mysqld.exe!mysql_select()[sql_select.cc:2404]
mysqld.exe!handle_select()[sql_select.cc:268]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5011]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2206]
mysqld.exe!mysql_parse()[sql_parse.cc:5931]
mysqld.exe!dispatch_command()[sql_parse.cc:1213]
mysqld.exe!do_command()[sql_parse.cc:854]
mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
mysqld.exe!pthread_start()[my_winthread.c:87]
mysqld.exe!_callthreadstart()[thread.c:293]

didn't affect 5.0 or 6.0.

How to repeat:
start server with --lower-case-table-names=2 and import the privately attached dumpfile. 
Make sure lower-case-table-names is set at 2.
[26 Aug 19:09] Valeriy Kravchuk
Verified with a bit smaller test case (will attach later) on Mac OS X with latest 5.1.39
from bzr. In the error log I've got:

Version: '5.1.39-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
090826 20:05:46 [ERROR] Table xy/activewfset contains fewer indexes inside InnoDB than
are defined in the MySQL .frm file. Have you mixed up .frm files from different
installations? See http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting.html

090826 20:05:46 [ERROR] Innodb could not find key n:o 2 with name FK2439AFAD7BCE1348 from
dict cache for table xy/activewfset
090826 20:05:46 - mysqld got signal 10 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337716 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x1123618
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xb026af64 thread_stack 0x30000
0   mysqld                              0x0056a953 my_print_stacktrace + 44
1   mysqld                              0x000f82c3 handle_segfault + 853
2   libSystem.B.dylib                   0x940472bb _sigtramp + 43
3   ???                                 0xffffffff 0x0 + 4294967295
4   mysqld                              0x0041be79
_ZN11ha_innobase21store_key_val_for_rowEjPcjPKh + 1959
5   mysqld                              0x0041d827
_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function + 253
6   mysqld                              0x0041e2e1 _ZN11ha_innobase11index_firstEPh +
143
7   mysqld                              0x0019078e
_Z20init_read_record_seqP13st_join_table + 658
8   mysqld                              0x0018cdff _Z10sub_selectP4JOINP13st_join_tableb
+ 273
9   mysqld                              0x0018d23c _Z10sub_selectP4JOINP13st_join_tableb
+ 1358
10  mysqld                              0x001a5810 _ZN4JOIN4execEv + 9078
11  mysqld                              0x001a5caa
_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex
+ 866
12  mysqld                              0x001a602e
_Z13handle_selectP3THDP6st_lexP13select_resultm + 558
13  mysqld                              0x00109ee6
_Z20prepare_schema_tableP3THDP6st_lexP11Table_ident18enum_schema_tables + 2096
14  mysqld                              0x00110954 _Z21mysql_execute_commandP3THD + 2896
15  mysqld                              0x0011a9e4 _Z11mysql_parseP3THDPKcjPS2_ + 580
16  mysqld                              0x0011b7a2
_Z16dispatch_command19enum_server_commandP3THDPcj + 3080
17  mysqld                              0x0011cab6 _Z10do_commandP3THD + 654
18  mysqld                              0x00107b64 handle_one_connection + 366
19  libSystem.B.dylib                   0x9400c095 _pthread_start + 321
20  libSystem.B.dylib                   0x9400bf52 thread_start + 34
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1125028 = select * from XY.ActiveWFSet limit 1
thd->thread_id=12
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
090826 20:05:46 mysqld_safe mysqld restarted
090826 20:05:47 [Warning] Setting lower_case_table_names=2 because file system for
/Users/openxs/dbs/5.1/var/ is case insensitive
...
[26 Aug 22:14] Calvin Sun
I am able to reproduce the crash on Windows XP. The call stack:

007808CA    mysqld.exe!dict_index_contains_col_or_prefix()[dict0dict.c:543]
00750A91    mysqld.exe!build_template()[ha_innodb.cc:3370]
00751DA2    mysqld.exe!ha_innobase::index_read()[ha_innodb.cc:4435]
007528CC    mysqld.exe!ha_innobase::index_first()[ha_innodb.cc:4738]
00627429    mysqld.exe!join_read_first()[sql_select.cc:11793]
00625A25    mysqld.exe!sub_select()[sql_select.cc:11098]
0062556F    mysqld.exe!do_select()[sql_select.cc:10855]
00610905    mysqld.exe!JOIN::exec()[sql_select.cc:2199]
00610FF1    mysqld.exe!mysql_select()[sql_select.cc:2380]
0060A90D    mysqld.exe!handle_select()[sql_select.cc:268]
0059869F    mysqld.exe!execute_sqlcom_select()[sql_parse.cc:4911]
00591008    mysqld.exe!mysql_execute_command()[sql_parse.cc:2204]
0059A801    mysqld.exe!mysql_parse()[sql_parse.cc:5831]
0058F05E    mysqld.exe!dispatch_command()[sql_parse.cc:1216]
0058E6D4    mysqld.exe!do_command()[sql_parse.cc:857]
0049B086    mysqld.exe!handle_one_connection()[sql_connect.cc:1115]
0088DED6    mysqld.exe!pthread_start()[my_winthread.c:85]
00869EE1    mysqld.exe!_callthreadstart()[thread.c:293]
00869E87    mysqld.exe!_threadstart()[thread.c:277]
7C80B713    kernel32.dll!GetModuleFileNameA()
[10 Sep 19:50] Calvin Sun
This appears to be a server bug. Somehow the table cache is messed up when
lower_case_table_names=2. I can repeat the crash when table_open_cache is 64 (default).
But the crash goes away after increasing the table_open_cache to 256 (or even 100).

Another indicator is the error such as:
090909 18:44:19 [ERROR] Table xy/activewfset contains fewer indexes inside InnoDB than
are defined in the MySQL .frm file. Have you mixed up .frm files from different
installations?

When this happens (just before the crash), table->s->keys = 4 while
ib_table->indexes->count = 2. 2 (in InnoDB) is the right number. 4 is the number prior to
drop database. So, the server reuses the old slot that has wrong info.