MySQL Bugs: #53539: Ndb : MySQLD default values in frm embedded in backup not endian-converted

Bug #53539	Ndb : MySQLD default values in frm embedded in backup not endian-converted
Submitted:	10 May 2010 13:43	Modified:	2 Jul 2010 11:59
Reporter:	Frazer Clement	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	mysql-5.1-telco-6.2	OS:	Any
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
1) Create a table with default values on a SPARC machine.
  e.g. 

create table jag (a int primary key, b int default 6, c float default 5.5, d double default 10.12, e varchar(20) default 'abcdefgh') engine=ndb;

2) Run show create table on SPARC :

mysql> show create table jag;
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                                          |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| jag   | CREATE TABLE `jag` (
  `a` int(11) NOT NULL,
  `b` int(11) DEFAULT '6',
  `c` float DEFAULT '5.5',
  `d` double DEFAULT '10.12',
  `e` varchar(20) DEFAULT 'abcdefgh',
  PRIMARY KEY (`a`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)

mysql>    

3) Perform Ndb backup 
4) Copy backup files to little-endian arch machine (eg. x86)
5) Restore backup, including metadata (-m)
6) Perform show create table on restored table

mysql> show create table jag;
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                                                                         |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| jag   | CREATE TABLE `jag` (
  `a` int(11) NOT NULL,
  `b` int(11) DEFAULT '100663296',
  `c` float DEFAULT '6.32266e-41',
  `d` double DEFAULT '1.19203925053371e-14',
  `e` varchar(20) DEFAULT 'abcdefgh',
  PRIMARY KEY (`a`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1 |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> 

It seems that the endianness dependent values (e.g. int, float, double) have missed conversion somewhere.  The endianness independent values (char strings) are ok.

The defaults information used by MySQLD is embedded in the copy of the table's .frm file that is stored as a 'black box' by the Ndb dictionary.
.frm files are supposed to be endian-independent, but it seems like some step is missed (or performed twice) when reading the defaults from the frm.

How to repeat:
See steps above.
Same issue should apply when going little -> big endian.

Suggested fix:
1) Determine what is happening currently
2) Modify to ensure the correct steps are taken.

The problem appears to be that :
 - The frm file contains a 'record' of default values, read and written in the handler's preferred byte order, as defined by ha->low_byte_first()
 - Most handlers have this function hard-coded to return 1.
 - Ndb returns 1 when the host is little-endian, 0 when it's big endian.
 - In cases where the frm file is written on one platform, and read on another, the endian-affected defaults appear to be trashed.

The assumption seems to be that a given handler *is* either big or little endian, rather than changing when re-loading a table from frm.

A workaround could be to first restore the metadata, then use ALTER TABLE to correct the defaults, then proceed with the table data and log.

A proper fix could be to change the SQL layer to 
  - *always* write defaults in little-endian format, making frm properly platform independent
  - convert defaults to big-endian when loading if ha->low_byte_first() is false on the host

This would break compatibility with existing big-endian frm files.  This could be worked-around if there is some indicator in the frm file about the endianness of the defaults in the record.

Another approach is to extend ndb_restore (which is 'aware' of an endian-conversion taking place) to reformat the defaults in the embedded frm file.  This would require knowledge of the format and contents of the frm file which ndb_restore doesn't currently have.

A third approach is to have ndb_restore 'mark' the table as being 'opposite endian' when restoring it, and have the 'first' MySQLD to discover the table effectively perform an online alter to change the endianness of the embedded frm embedded defaults.  Ideally other discovering MySQLDs would be serialised by the distributed schema lock mechanism, and could use the new frm file after the alter completes.  The table could be marked as 'opposite endian' by setting the packed-frm-version to some out-of-range value.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/108696

3540 Frazer Clement	2010-05-19
      Bug#53539 Ndb : MySQLD default values in frm embedded in backup not endian-converted
      
      Workaround to fix testcase breaking on big-endian.

Fix requires fix at Server level of bug#53818.

Workaround test has been pushed to 7.0, 7.1 etc in ndb_native_default_support testcase.

Leaving bug in state 'Verified'.