MySQL Bugs: #39219: Falcon silently fails to start if tablespace is locked by another process.

Bug #39219	Falcon silently fails to start if tablespace is locked by another process.
Submitted:	3 Sep 2008 16:28	Modified:	5 Sep 2008 8:37
Reporter:	Philip Stoev	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S1 (Critical)
Version:	6.0-falcon-team	OS:	Any
Assigned to:	Vladislav Vaintroub	CPU Architecture:	Any

Description:
If another process is holding a lock on the tablespace file, Falcon will fail to initialize. The server will then continue running without a Falcon storage engine. Falcon tables created before will not be accessible. Falcon tables created afterwards will be silently converted into MyISAM, unless the NO_ENGINE_SUBSTITUTION SQL mode is in effect, which is *not* the default.

This is in contrast with the Innodb behavior, which loops again and again trying to obtain a lock on its tablespace files. It only allows the server to start after it has obtained a lock on those files and initialized successfully.

Please note that a situation where two mysqld processes are competing for the same tablespace may be possible when the safe_mysqld script is used. I believe the Innodb behavior is a result of having to accomodate that, and any other weird server restart scenarios.

How to repeat:
1. In one console, start mysqld

./mysqld --language=/build/bzr/6.0-falcon-team/sql/share/english/ --datadir=/build/datadir --falcon --innodb --socket="/tmp/sock" --skip-grant-tables

2. In another console, start another mysqld

./mysqld --language=/build/bzr/6.0-falcon-team/sql/share/english/ --datadir=/build/datadir --falcon --innodb --socket="/tmp/mysql.sock" --skip-grant-tables

The second mysqld will report Falcon can not start. Then, it will proceed to start Innodb, and Innodb will loop forever waiting for the tablespace to be unlocked by the first mysqld.

3. Kill the first mysqld.

At this point, the Innodb of the second mysqld will detect that the tablespace is now available and will initialize. The second server will start accepting connections without having Falcon present.

4. Create a Falcon table. It will be silently converted to MyISAM. Any existing Falcon tables will not be accessible.

Suggested fix:
* Either Falcon should loop like Innodb does, or failure to initialize a storage engine must cause the entire server to terminate before attempting to accept any connections.

Vlad,  I will let you determine the best way to fix this.  You may want to ask some others for their opinion.  Here is what Peter G says;

>>Kevin Lewis wrote:
>>Peter, What is your opinion of this behavior?  
>>Should Falcon loop endlessly, fail helplessly, or continue to fail quietly?

> Peter Gulutzan wrote;
>Since people might be used to the InnoDB precedent, I believe
>"loop endlessly" is best. Since the InnoDB looped message is
>"
>InnoDB: Unable to lock ./ibdata1, error: 11
>InnoDB: Check that you do not already have another mysqld process
>InnoDB: using the same InnoDB data or log files.
>080904 16:44:03  InnoDB: Retrying to lock the first data file
>"

Falcon does not fail silently. It complains in the error log and returns error code to the storage engine API. That storage engine API does _not_ crash is something that can be discussed, but it should be handled on the correct layer.
Not with Falcons local fix, but in the same way for all engines. Looping endlessly while filling the disk with the same message in the error log like Innodb does is not the smartest way to use the disk.