Bug #10625 | DECIMAL tables corrupt after upgrade 5.0.4 -> 5.0.6 | ||
---|---|---|---|
Submitted: | 13 May 2005 16:18 | Modified: | 25 May 2005 23:20 |
Reporter: | Hakan Küçükyılmaz | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server | Severity: | S1 (Critical) |
Version: | 5.0.6-beta | OS: | Linux (SLES 9 x86_64) |
Assigned to: | Paul DuBois | CPU Architecture: | Any |
[13 May 2005 16:18]
Hakan Küçükyılmaz
[13 May 2005 17:03]
Heikki Tuuri
This is probably an InnoDB bug. --Heikki
[13 May 2005 17:20]
Heikki Tuuri
Hi! ha_innodb.cc: templ->mysql_col_len = (ulint) field->pack_length(); templ->type = index->table->cols[i].type.mtype; templ->mysql_type = (ulint)field->type(); row0sel.c: } else if (templ->type == DATA_MYSQL) { memcpy(dest, data, len); ut_a(templ->mysql_col_len >= len); ut_a(templ->mbmaxlen >= templ->mbminlen); ut_a(templ->mbmaxlen > templ->mbminlen || templ->mysql_col_len == len); /* The following assertion would fail for old tables containing UTF-8 ENUM columns due to Bug #9526. */ ut_ad(!templ->mbmaxlen || !(templ->mysql_col_len % templ->mbmaxlen)); ut_a(len * templ->mbmaxlen >= templ->mysql_col_len); if (templ->mbminlen != templ->mbmaxlen) { /* Pad with spaces. This undoes the stripping done in row0mysql.ic, function row_mysql_store_col_in_innobase_format(). */ memset(dest + len, 0x20, templ->mysql_col_len - len); } The crash would be explained if mysql_col_len is SMALLER than len, which is 3. Regards, Heikki
[13 May 2005 17:26]
Heikki Tuuri
Hakan, it cannot be smaller because an assertion checks it beforehand! ut_a(templ->mysql_col_len >= len); Hakan, please check with gdb what is the value of *template in the crash. I want to know what is mysql_col_len there! Regards, Heikki
[13 May 2005 17:30]
Heikki Tuuri
Hakan, of course I meant gdb> frame 1 gdb> print *templ (not *template) Also give the full table definition. The bug maybe is associated with the column type having wrong pack_lenght in MySQL. Regards, Heikki
[17 May 2005 10:06]
Hakan Küçükyılmaz
Heikki, (gdb) frame 1 #1 0x00000000007622e2 in row_sel_field_store_in_mysql_format ( dest=0x37aa708 "\200\002\207", ' ' <repeats 197 times>..., templ=0x2a9870b988, data=0x2a99400eb0 "\200\002\207B", len=3) at row0sel.c:2378 2378 memset(dest + len, 0x20, templ->mysql_col_len - len); Current language: auto; currently c (gdb) print *templ $1 = {col_no = 2, rec_field_no = 4, mysql_col_offset = 32, mysql_col_len = 2, mysql_null_byte_offset = 0, mysql_null_bit_mask = 0, type = 4, mysql_type = 246, mysql_length_bytes = 0, charset = 63, mbminlen = 1, mbmaxlen = 1, is_unsigned = 0} (gdb) TABLE: CREATE TABLE "DDNTF" ( "TABNAME" char(30) collate latin1_bin NOT NULL default '', "BLOCKNR" tinyint(3) unsigned NOT NULL default '0', "FIELDSLG" decimal(4,0) NOT NULL default '0', "FIELDS" blob, PRIMARY KEY ("TABNAME","BLOCKNR") ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin I have a full dump (34MB) of the table. I can upload it somewhere if you want. Regards, Hakan
[17 May 2005 12:02]
Heikki Tuuri
Hakan, thank you. What I do not understand is why the assertion does not fail as len = 3, and templ->mysql_col_len = 2! I will study this further. --Heikki
[17 May 2005 13:01]
Hakan Küçükyılmaz
We stepped through the code and at row0sel.c:2362 templ->mysql_col_len is 30 and len is 30, too. Please note that this happens on a 64-bit machine. Regards, Hakan
[17 May 2005 13:53]
Marko Mäkelä
Hakan, can you do the following: break row0sel.c:2362 ignore 1 <a very large number> run (crash occurs) info breakpoints ignore 1 <number reported above, minus 1> run (breakpoint occurs) display *templ n n n ... (until crash on line 2378) Does the contents of templ change? It shouldn't.
[18 May 2005 11:42]
Hakan Küçükyılmaz
The contents of templ does not change. Regards, Hakan
[20 May 2005 14:47]
Heikki Tuuri
Hakan, "FIELDSLG" decimal(4,0) NOT NULL default '0', The templ in the crash is for storing the DECIMAL column. The internal format for the column is probably 2 bytes. Jani fixed a bug in DECIMAL key value length about a week ago. That bug fix may have triggered this crash. Actually, if the MySQL pack_length() for DECIMAL was wrong in 5.0.5, then old tables may crash with the fixed 5.0.6. When did you create and populate that table? I will study more. Regards, Heikki
[21 May 2005 15:37]
Heikki Tuuri
Hi! Looks like Jani's patch on May 12th, 2005 did change the pack_length of NEWDECIMAL when he fixed http://bugs.mysql.com/bug.php?id=10465 This is an incompatible change, and any old table that contains DECIMAL and was created with 5.0.? - 5.0.5 must be rebuilt when upgrading to 5.0.6. I have reopened Bug #10465, and asked Jani to document it prominently. Regards, Heikki http://lists.mysql.com/internals/24831 ------------------------------------------------ D 1.259 05/05/12 17:36:29+03:00 jani@a193-229-222-105.elisa-laajakaista.fi 466 4 65 6/1/8580 P sql/field.cc C Fixed bug 10465. ... --- 1.258/sql/field.cc Thu May 5 18:05:56 2005 +++ 1.259/sql/field.cc Thu May 12 17:36:29 2005 @@ -8060,7 +8060,12 @@ } break; case MYSQL_TYPE_NEWDECIMAL: - key_length= pack_length= my_decimal_get_binary_size(length, decimals); + key_length= pack_length= + my_decimal_get_binary_size(my_decimal_length_to_precision(length, + decimals, + flags & + UNSIGNED_FLAG), + decimals); break; default: key_length= pack_length= calc_pack_length(sql_type, length); The code is now: /***************************************************************************** Handling of field and create_field *****************************************************************************/ void create_field::create_length_to_internal_length(void) { switch (sql_type) { case MYSQL_TYPE_TINY_BLOB: case MYSQL_TYPE_MEDIUM_BLOB: case MYSQL_TYPE_LONG_BLOB: case MYSQL_TYPE_BLOB: case MYSQL_TYPE_VAR_STRING: case MYSQL_TYPE_STRING: case MYSQL_TYPE_VARCHAR: length*= charset->mbmaxlen; key_length= length; pack_length= calc_pack_length(sql_type, length); break; case MYSQL_TYPE_ENUM: case MYSQL_TYPE_SET: /* Pack_length already calculated in sql_parse.cc */ length*= charset->mbmaxlen; key_length= pack_length; break; case MYSQL_TYPE_BIT: if (f_bit_as_char(pack_flag)) { key_length= pack_length= ((length + 7) & ~7) / 8; } else { pack_length= length / 8; /* We need one extra byte to store the bits we save among the null bits *\ / key_length= pack_length + test(length & 7); } break; case MYSQL_TYPE_NEWDECIMAL: key_length= pack_length= my_decimal_get_binary_size(my_decimal_length_to_precision(length, decimals, flags & UNSIGNED_FLAG), decimals); break; default: key_length= pack_length= calc_pack_length(sql_type, length); break; } }
[25 May 2005 6:47]
Heikki Tuuri
Hakan, when I look at your table dump from 5.0.5, the ddntf.sql file, the type of the column is DECIMAL(5,0). But in the bug report above you give it as DECIMAL(4,0). The templ in the stack trace has mysql_col_len = 2, which suggests that the type in that case was DECIMAL(4,0). What is the correct table definition to reproduce the problem? Is it possible that you have mixed an .frm file where the type is DECIMAL(4,0) to an .frm file where it is DECIMAL(5,0)? Regards, Heikki
[25 May 2005 7:20]
Heikki Tuuri
Hakan, ok I found the explanation. I imported your table dump with 5.0.4. There SHOW CREATE TABLE gives DECIMAL(5, 0) as the type. Then I upgrade to 5.0.7. There it gives DECIMAL(4, 0) as the type! I checked what is field->pack_length() in 5.0.4: it is 3. But in 5.0.7 it is only 2! Since the code in Jani's patch does not get executed in 5.0.7, it cannot explain this. Something else has changed. Holyfoot edited decimal.h and my_decimal.h on May 5th. Maybe he can explain this? I was able to crash 5.0.7 with your query. Looks like the bug seriously corrupts the stack, which explains why the program appeared to crash in an impossible place memset(). Conclusion: 1) Jani's bug fix was necessary because all 5.0.4 tables with an indexed DECIMAL column were corrupt. 2) There was also some other change in the calculation of pack_length() for a DECIMAL in an .frm file, and that change makes at least DECIMAL(5,0) tables corrupt after an upgrade 5.0.4 -> 5.0.7. Regards, Heikki heikki@hundin:~/mysql-5.0.4-beta/client> ./mysql test Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 to server version: 5.0.4-beta-log Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> show create table DDNTF; +-------+----------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- --------------------------------------------------------------------+ | Table | Create Table | +-------+----------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- --------------------------------------------------------------------+ | DDNTF | CREATE TABLE `DDNTF` ( `TABNAME` char(30) collate latin1_bin NOT NULL default '', `BLOCKNR` tinyint(3) unsigned NOT NULL default '0', `FIELDSLG` decimal(5,0) NOT NULL default '0', `FIELDS` blob, PRIMARY KEY (`TABNAME`,`BLOCKNR`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin | +-------+----------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- --------------------------------------------------------------------+ 1 row in set (0.01 sec) mysql> exit Bye heikki@hundin:~/mysql-5.0.4-beta/client> ./mysqladmin shutdown heikki@hundin:~/mysql-5.0.4-beta/client> ./mysql test Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 to server version: 5.0.7-beta-debug-log Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> show create table DDNTF; +-------+----------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- --------------------------------------------------------------------+ | Table | Create Table | +-------+----------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- --------------------------------------------------------------------+ | DDNTF | CREATE TABLE `DDNTF` ( `TABNAME` char(30) collate latin1_bin NOT NULL default '', `BLOCKNR` tinyint(3) unsigned NOT NULL default '0', `FIELDSLG` decimal(4,0) NOT NULL default '0', `FIELDS` blob, PRIMARY KEY (`TABNAME`,`BLOCKNR`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin | +-------+----------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- --------------------------------------------------------------------+ 1 row in set (0.05 sec) mysql>
[25 May 2005 7:44]
Heikki Tuuri
Hi! I updated the synopsis of this bug to reflect that also MyISAM tables are corrupt after an upgrade. Regards, Heikki heikki@hundin:~/mysql-5.0.4-beta/client> ./mysql test Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 to server version: 5.0.4-beta-log Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> create table m(a DECIMAL(5, 0) NOT NULL PRIMARY KEY) type=myisam; Query OK, 0 rows affected, 1 warning (0.02 sec) mysql> insert into m values (12345); Query OK, 1 row affected (0.00 sec) mysql> insert into m values (123); Query OK, 1 row affected (0.00 sec) mysql> select * from m; +-------+ | a | +-------+ | 123 | | 12345 | +-------+ 2 rows in set (0.00 sec) mysql> show create table m; +-------+----------------------------------------------------------------------- --------------------------------------+ | Table | Create Table | +-------+----------------------------------------------------------------------- --------------------------------------+ | m | CREATE TABLE `m` ( `a` decimal(5,0) NOT NULL, PRIMARY KEY (`a`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 | +-------+----------------------------------------------------------------------- --------------------------------------+ 1 row in set (0.00 sec) mysql> exit Bye heikki@hundin:~/mysql-5.0.4-beta/client> ./mysqladmin shutdown heikki@hundin:~/mysql-5.0.4-beta/client> cd heikki@hundin:~> cd mysql-5.0 heikki@hundin:~/mysql-5.0> cd client heikki@hundin:~/mysql-5.0/client> ./mysql test Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 to server version: 5.0.7-beta-debug-log Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> select * from m; +----+ | a | +----+ | 48 | | 0 | +----+ 2 rows in set (0.01 sec) mysql> show create table m; +-------+----------------------------------------------------------------------- -----------------------------------------+ | Table | Create Table | +-------+----------------------------------------------------------------------- -----------------------------------------+ | m | CREATE TABLE `m` ( `a` decimal(4,0) NOT NULL, PRIMARY KEY (`a`(3)) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 | +-------+----------------------------------------------------------------------- -----------------------------------------+ 1 row in set (0.00 sec)
[25 May 2005 10:24]
Heikki Tuuri
Hi! A final note: the original crash happened because the InnoDB internal data type was DATA_BINARY, and field_end was smaller than pad_ptr (len = 3, but mysql_col_len=2). Then it tried to pad 4 GB with spaces, corrupting stacks. Regards, Heikki Breakpoint 1, row_sel_store_mysql_rec ( mysql_rec=0x8c5d460 "¥TCALS", ' ' <repeats 25 times>, '¥' <repeats 65 times> , prebuilt=0x40acd268, rec=0x40dfce84 "TCALS", ' ' <repeats 25 times>, "\001", offsets=0x63292cec) at row0sel.c:2416 2416 mem_heap_t* extern_field_heap = NULL; Current language: auto; currently c (gdb) next 2426 if (prebuilt->blob_heap != NULL) { (gdb) 2431 for (i = 0; i < prebuilt->n_template; i++) { (gdb) 2433 templ = prebuilt->mysql_template + i; (gdb) 2435 data = rec_get_nth_field(rec, offsets, (gdb) 2438 if (rec_offs_nth_extern(offsets, templ->rec_field_no)) { (gdb) 2458 if (len != UNIV_SQL_NULL) { (gdb) 2459 if (templ->type == DATA_BLOB) { (gdb) 2502 row_sel_field_store_in_mysql_format( (gdb) 2507 if (extern_field_heap) { (gdb) 2512 if (templ->mysql_null_bit_mask) { (gdb) 2431 for (i = 0; i < prebuilt->n_template; i++) { (gdb) 2433 templ = prebuilt->mysql_template + i; (gdb) 2435 data = rec_get_nth_field(rec, offsets, (gdb) 2438 if (rec_offs_nth_extern(offsets, templ->rec_field_no)) { (gdb) 2458 if (len != UNIV_SQL_NULL) { (gdb) 2459 if (templ->type == DATA_BLOB) { (gdb) 2502 row_sel_field_store_in_mysql_format( (gdb) 2507 if (extern_field_heap) { (gdb) 2512 if (templ->mysql_null_bit_mask) { (gdb) 2431 for (i = 0; i < prebuilt->n_template; i++) { (gdb) 2433 templ = prebuilt->mysql_template + i; (gdb) 2435 data = rec_get_nth_field(rec, offsets, (gdb) 2438 if (rec_offs_nth_extern(offsets, templ->rec_field_no)) { (gdb) 2458 if (len != UNIV_SQL_NULL) { (gdb) 2459 if (templ->type == DATA_BLOB) { (gdb) 2502 row_sel_field_store_in_mysql_format( (gdb) step row_sel_field_store_in_mysql_format (dest=0x8c5d480 '¥' <repeats 64 times>, templ=0x40ace2d0, data=0x40dfceb0 "\200\002\207B", len=3) at row0sel.c:2279 2279 if (templ->type == DATA_INT) { (gdb) next 2299 } else if (templ->type == DATA_VARCHAR (gdb) 2303 field_end = dest + templ->mysql_col_len; (gdb) 2305 if (templ->mysql_type == DATA_MYSQL_TRUE_VARCHAR) { (gdb) 2315 ut_memcpy(dest, data, len); (gdb) list 2310 dest = row_mysql_store_true_var_len(dest, len, 2311 templ->mysql_length_byte s); 2312 } 2313 2314 /* Copy the actual data */ 2315 ut_memcpy(dest, data, len); 2316 2317 /* Pad with trailing spaces. We pad with spaces also the 2318 unused end of a >= 5.0.3 true VARCHAR column, just in ca se 2319 MySQL expects its contents to be deterministic. */ (gdb) print len $1 = 3 (gdb) list 2280 2275 byte* pad_ptr; 2276 2277 ut_ad(len != UNIV_SQL_NULL); 2278 2279 if (templ->type == DATA_INT) { 2280 /* Convert integer data from Innobase to a little-endian 2281 format, sign bit restored to normal */ 2282 2283 ptr = dest + len; 2284 (gdb) 2285 for (;;) { 2286 ptr--; 2287 *ptr = *data; 2288 if (ptr == dest) { 2289 break; 2290 } 2291 data++; 2292 } 2293 2294 if (!templ->is_unsigned) { (gdb) 2295 dest[len - 1] = (byte) (dest[len - 1] ^ 128); 2296 } 2297 2298 ut_ad(templ->mysql_col_len == len); 2299 } else if (templ->type == DATA_VARCHAR 2300 || templ->type == DATA_VARMYSQL 2301 || templ->type == DATA_BINARY) { 2302 2303 field_end = dest + templ->mysql_col_len; 2304 (gdb) 2305 if (templ->mysql_type == DATA_MYSQL_TRUE_VARCHAR) { 2306 /* This is a >= 5.0.3 type true VARCHAR. Store t he 2307 length of the data to the first byte or the firs t 2308 two bytes of dest. */ 2309 2310 dest = row_mysql_store_true_var_len(dest, len, 2311 templ->mysql_length_byte s); 2312 } 2313 2314 /* Copy the actual data */ (gdb) 2315 ut_memcpy(dest, data, len); 2316 2317 /* Pad with trailing spaces. We pad with spaces also the 2318 unused end of a >= 5.0.3 true VARCHAR column, just in ca se 2319 MySQL expects its contents to be deterministic. */ 2320 2321 pad_ptr = dest + len; 2322 2323 ut_ad(templ->mbminlen <= templ->mbmaxlen); 2324 (gdb) 2325 /* We handle UCS2 charset strings differently. */ 2326 if (templ->mbminlen == 2) { 2327 /* A space char is two bytes, 0x0020 in UCS2 */ 2328 2329 if (len & 1) { 2330 /* A 0x20 has been stripped from the col umn. 2331 Pad it back. */ 2332 2333 if (pad_ptr < field_end) { 2334 *pad_ptr = 0x20; (gdb) 2335 pad_ptr++; 2336 } 2337 } 2338 2339 /* Pad the rest of the string with 0x0020 */ 2340 2341 while (pad_ptr < field_end) { 2342 *pad_ptr = 0x00; 2343 pad_ptr++; 2344 *pad_ptr = 0x20; (gdb) 2345 pad_ptr++; 2346 } 2347 } else { 2348 ut_ad(templ->mbminlen == 1); 2349 /* space=0x20 */ 2350 2351 memset(pad_ptr, 0x20, field_end - pad_ptr); 2352 } 2353 } else if (templ->type == DATA_BLOB) { 2354 /* Store a pointer to the BLOB buffer to dest: the BLOB was (gdb) next 2321 pad_ptr = dest + len; (gdb) next 2326 if (templ->mbminlen == 2) { (gdb) 2351 memset(pad_ptr, 0x20, field_end - pad_ptr); (gdb) print *templ $2 = {col_no = 2, rec_field_no = 4, mysql_col_offset = 32, mysql_col_len = 2, mysql_null_byte_offset = 0, mysql_null_bit_mask = 0, type = 4, mysql_type = 246, mysql_length_bytes = 0, charset = 63, mbminlen = 1, mbmaxlen = 1, is_unsigned = 0} (gdb) print field_end $3 = (unsigned char *) 0x8c5d482 "\207", '¥' <repeats 61 times> (gdb) print pad_ptr $4 = (unsigned char *) 0x8c5d483 '¥' <repeats 61 times> (gdb)
[25 May 2005 23:20]
Paul DuBois
Noted in 5.0.6 changelog, and in the "upgrading to 5.0" section.