Bug #22881 CREATE LOGFILE GROUP ADD UNDOFILE incompatible with local datanodes
Submitted: 2 Oct 2006 8:43 Modified: 24 Oct 2006 9:05
Reporter: Roland Bouman Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Disk Data Severity:S3 (Non-critical)
Version:5.1.11 OS:Linux (ubuntu dapper drake linux)
Assigned to: CPU Architecture:Any
Tags: CREATE LOGFILE GROUP, disk-based storage, local cluster, UNDO file

[2 Oct 2006 8:43] Roland Bouman
Description:
The use disk-based storage, a logifle group needs to be created first:

http://dev.mysql.com/doc/refman/5.1/en/create-logfile-group.html

Undo files maybe added to the logfile group. Addition of a logfile involves a specification of the path of the logfile.

According to the documentation, the undofile is then created for each datanode:

"When used with ENGINE = NDB, a logfile group and associated UNDO logfile are created on each Cluster data node. You can verify that the UNDO files were created and obtain information about them by querying the INFORMATION_SCHEMA.FILES table."

For datanodes that are running on the same physical host, the path specified for the undofile identifies the same actual file. I have not encountered any obvious problems due to this, but it does seem like something is likely to go horribly wrong in such a setup.

How to repeat:
Setup a cluster on one physical host:

[NDB_MGMD DEFAULT]
DataDir=/opt/mysql/cluster

[NDB_MGMD]
Id=1
HostName=localhost
LogDestination=FILE:filename=/opt/mysql/cluster/cluster-log

[NDBD DEFAULT]
NoOfReplicas=2
DataDir=/opt/mysql/cluster

[NDBD]
Id=2
HostName=localhost

[NDBD]
Id=3
HostName=localhost

[MYSQLD]
Id=4
HostName=localhost

[MySQLD]
Id=5
HostName=localhost

Then, startup the cluster, and use mysql command line tool to connect to one of the SQL Nodes. 

Create a logfile group + undofile:

create logfile group logfilegroup1 
add undofile '/opt/mysql/cluster/logfilegroup1_undofile1.dat' 
initial_size 20M 
undo_buffer_size 10M 
engine ndb;
 
Use the information schema to get the cluster's view on this logfile group:

mysql> select file_name, tablespace_name, table_name, logfile_group_name, extra from information_schema.files;
+------------------------------------------------+-----------------+------------+--------------------+----------------+
| file_name                                      | tablespace_name | table_name | logfile_group_name | extra          |
+------------------------------------------------+-----------------+------------+--------------------+----------------+
| /opt/mysql/cluster/logfilegroup1_undofile1.dat |                 |            | logfilegroup1      | CLUSTER_NODE=2 |
| /opt/mysql/cluster/logfilegroup1_undofile1.dat |                 |            | logfilegroup1      | CLUSTER_NODE=3 |
+------------------------------------------------+-----------------+------------+--------------------+----------------+
2 rows in set (0.00 sec)

verify that there is only one such file using a command line shell:

roland@Rolenove:/opt/mysql/cluster$ ls logfilegroup*
logfilegroup1_undofile1.dat

The behaviour as described above has not been entirely reproducable. Dropping the logfile group, shutting down the cluster (management server + data nodes, leaving the sql node alive) and retrying yielded this very strange result:

mysql> create logfile group logfilegroup1 add undofile '/opt/mysql/cluster/logfilegroup1_undofile1.dat' initial_size 20M undo_buffer_size 10M engine ndb;
ERROR 1515 (HY000): Failed to create UNDOFILE
mysql> select file_name, tablespace_name, table_name, logfile_group_name, extra from information_schema.files;
Empty set (0.00 sec)

(so far so good, we want it to fail, but now:)
repeating the statement:

mysql> create logfile group logfilegroup1 add undofile '/opt/mysql/cluster/logfilegroup1_undofile1.dat' initial_size 20M undo_buffer_size 10M engine ndb;
ERROR 1515 (HY000): Failed to create LOGFILE GROUP

mmm, so first creating only the undo file failed, now, the 2nd time, the logfile group itself fails to be created, implying that we did create a logfile group the first time, only the undo file failed to be created. Strange.

mysql> drop logfile group logfilegroup1 engine = ndb;
Query OK, 0 rows affected (0.29 sec)

so, is it there or not? Drop logfile group seems to have done something...

Suggested fix:
I can't judge really. Maybe disk-based storage is not supported at all in this case? If so, the cluster should not silently accept the creation of the log group.

Another solution would be to change the semantics of the path in the UNDOFILE clause - not letting it specify the actual filename, but rather a basename to which the cluster appends a node id to make the files unique.

Anyhow, the attempt to reproduce it seems to give a very strange result. I mean the 
ERROR 1515 (HY000): Failed to create UNDOFILE
followed by the 
ERROR 1515 (HY000): Failed to create LOGFILE GROUP

If an 'empty' logfile group would be created, it would be very strange for 2 reasons:
1) the statement specifies creating the logfile group WITH the undofile. Either both or none should be created - not half
2) the CREATE LOGFILE GROUP syntax must include the ADD UNDOFILE clause so it seems strange to be able to create an empty logfilegroup this way.
[2 Oct 2006 8:52] Roland Bouman
Shutting down the entire cluster (management server, data nodes and sql nodes), leaving the empty logfile group behind makes it entirely impossible to drop the logfile group
[2 Oct 2006 9:25] Roland Bouman
I had some trouble reproducing the original situation - i.e. successfully creating a logfile group on a local cluster.

I found a way now: 

1) Startup the cluster (2 datanodes).
2) Stop one of the nodes (2 STOP)
3) create the logfilegroup.
4) start the ndbd process. It rejoins successfully.
[24 Oct 2006 9:05] Roland Bouman
Solution: don't use absolute paths (Thanks, Stewart!).

By using relative paths, files will be created in the ndb_<id>_fs directory which resides in the DataDirectory configured for the datanode.