Bug #64144 rename table a to b ignores failures in fil_rename_tablespace
Submitted: 27 Jan 2012 4:21 Modified: 27 Jan 2012 6:32
Reporter: Mark Callaghan Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: InnoDB Plugin storage engine Severity:S2 (Serious)
Version:5.1.52 OS:Any
Assigned to: CPU Architecture:Any
Tags: DDL, innodb, rename

[27 Jan 2012 4:21] Mark Callaghan
Description:
I noticed this after an InnoDB table was lost during a RENAME statement that failed

The db error log had this:

120120 18:15:39 InnoDB: Warning: problems renaming 'db/__osc_new_zz' to 'db/zz', 24998 iterations
InnoDB: Warning: tablespace './db/__osc_new_zz.ibd' has i/o ops stopped for a long time 25105 
120120 18:15:39 InnoDB: Warning: problems renaming 'db/__osc_new_zz' to 'db/zz', 24999 iterations
InnoDB: Warning: tablespace './db/__osc_new_zz.ibd' has i/o ops stopped for a long time 25106 
120120 18:15:39 InnoDB: Warning: problems renaming 'db/__osc_new_zz' to 'db/zz', 25000 iterations
InnoDB: Warning: tablespace './db/__osc_new_zz.ibd' has i/o ops stopped for a long time 25107 
120120 18:15:39 InnoDB: Warning: problems renaming 'db/__osc_new_zz' to 'db/zz', 25001 iterations

I am still trying to figure out what the SQL statement was.

How to repeat:
Change fil_rename_tablespace to always return an error

        if (1 || count > 25000) {
                space->stop_ios = FALSE;
                mutex_exit(&fil_system->mutex);
                return(FALSE);

Then run this as i.test
-- source include/have_innodb_plugin.inc

create table t(i int primary key) engine=innodb;

rename table t to s;
show tables;
show table status;
select * from s;

with this as i-master.opt

--innodb_file_format=barracuda --innodb_file_per_table=1

The output shows that:
* no error is reported to the user on the rename
* mysql thinks the table s exists
* SHOW TABLE STATUS has not output for the table
* innodb doesn't think the table 's' exists

create table t(i int primary key) engine=innodb;
rename table t to s;
show tables;
Tables_in_test
s
show table status;
Name	Engine	Version	Row_format	Rows	Avg_row_length	Data_length	Max_data_length	Index_length	Data_free	Auto_increment	Create_time	Update_time	Check_time	Collation	Checksum	Create_options	Comment

mysqltest: At line 8: query 'select * from s' failed: 1146: Table 'test.s' doesn't exist

---

when the test ends these files exist:
s.frm  t.ibd

Suggested fix:
This is the second or third bug I have had where InnoDB lost a table on a DDL failure. Error injection tests would catch many of these problems. Yet there are no such tests for InnoDB.

The problem is in row_rename_table_for_mysql. It doesn't set 'err' on errors in this case and 'err' is returned (returning 0 in this case)

                if (!dict_table_rename_in_cache(table, new_name,
                                                !new_is_tmp)) {
                        trx->error_state = DB_SUCCESS;
                        trx_general_rollback_for_mysql(trx, NULL);
                        trx->error_state = DB_SUCCESS;
                        goto funct_exit;
                }
[27 Jan 2012 5:03] Mark Callaghan
It looks like this was the root cause as the production server has:
* FRM file renamed
* iibd file not renamed
[27 Jan 2012 5:39] MySQL Verification Team
see also bug #62100 and
[27 Jan 2012 5:52] Mark Callaghan
There are two reasons why retry can be done in fil_rename_tablespace. It would help to know which one forces is the cause in the error message used by it. It would also help to know the values of:
* node->n_pending
* node->n_pending_flushes
* node->modification_counter
* node->flush_counter
[27 Jan 2012 5:55] MySQL Verification Team
http://lists.mysql.com/mysql/225516
[27 Jan 2012 6:31] MySQL Verification Team
In mysql-5.1 bzr code now do set the err:

if (!dict_table_rename_in_cache(table, new_name,
						!new_is_tmp)) {
			trx->error_state = DB_SUCCESS;
			trx_general_rollback_for_mysql(trx, NULL);
			trx->error_state = DB_SUCCESS;
			err = DB_ERROR;
			goto funct_exit;
		}

This was fixed by:
revno: 3634
committer: Marko M„kel„ <marko.makela@oracle.com>
branch nick: mysql-5.1
timestamp: Thu 2011-10-27 14:58:12 +0300
message:
  Bug #12884631 62146: TABLES ARE LOST FOR DDL
  
  row_rename_table_for_mysql(): Return DB_ERROR instead of DB_SUCCESS
  when fil_rename_tablespace() returns an error. This bug was introduced
  in the InnoDB Plugin.