Description:
Info:
- We are running OpenBSD 3.4 w/ the packaged MySQL client/server software.
- Our mail application (atmail) creates 800+ tables for its operation.
Last week we were setting up NFS to use mysqlhotcopy to make daily backups of our mysql data. On our test of the Mysql datase itself, the copy was successful. We then went on to run the mysqlhotcopy on our atmail database and were given error messages similar to :
/backup/test//atmail_old' DBD::mysql::db do failed: Can't find file: './atmail/EmailMessage_k.frm'
(errno: 9) at /usr/local/bin/mysqlhotcopy line 438.
We didn't think much of this error until our online applications started to fail w/ a "can't connect to Mysql server" error on all languages (perl, php etc).
The database was then shutdown to check its integrity. mysqlcheck returned no issues, and every table within the "atmail" database was confirmed and in tact.
MySQL was restarted, no issues on startup. Tested our online apps, and they were still having issues connected to the Mysql server. apache was then restarted and everything was fine.
This was then attempted:
bash-2.05b# mysqldump -c -l -u root -p atmail > testbackup.sql
Enter password:
mysqldump: Got error: 1105: File './atmail/Calander_other.MYD' not found (Errcode: 9) when using LOCK TABLES
bash-2.05b# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 305 to server version: 3.23.57
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use atmail;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Didn't find any fields in table 'Calander_other'
Didn't find any fields in table 'Calander_p'
Didn't find any fields in table 'Calander_q'
Didn't find any fields in table 'Calander_r'
Didn't find any fields in table 'Calander_s'
Didn't find any fields in table 'Calander_t'
Didn't find any fields in table 'Calander_u'
Didn't find any fields in table 'Calander_v'
Didn't find any fields in table 'Calander_w'
Didn't find any fields in table 'Calander_x'
Didn't find any fields in table 'Calander_y'
Didn't find any fields in table 'Calander_z'
Didn't find any fields in table 'ClientBilling'
Didn't find any fields in table 'ClientPayment'
Didn't find any fields in table 'EmailDatabase_f'
Didn't find any fields in table 'EmailDatabase_g'
Didn't find any fields in table 'EmailDatabase_h'
Didn't find any fields in table 'EmailDatabase_i'
Didn't find any fields in table 'EmailDatabase_l'
Didn't find any fields in table 'EmailDatabase_n'
Didn't find any fields in table 'EmailDatabase_o'
Didn't find any fields in table 'EmailDatabase_other'
Didn't find any fields in table 'EmailDatabase_q'
Didn't find any fields in table 'EmailDatabase_r'
Didn't find any fields in table 'EmailDatabase_u'
Didn't find any fields in table 'EmailDatabase_v'
Didn't find any fields in table 'EmailDatabase_w'
Didn't find any fields in table 'EmailDatabase_x'
Didn't find any fields in table 'EmailDatabase_y'
Didn't find any fields in table 'EmailDatabase_z'
Didn't find any fields in table 'EmailMessage_Reply'
Didn't find any fields in table 'EmailMessage_e'
Didn't find any fields in table 'EmailMessage_f'
Didn't find any fields in table 'EmailMessage_g'
Didn't find any fields in table 'EmailMessage_h'
Didn't find any fields in table 'EmailMessage_i'
Didn't find any fields in table 'EmailMessage_j'
Didn't find any fields in table 'EmailMessage_l'
Didn't find any fields in table 'EmailMessage_m'
Didn't find any fields in table 'EmailMessage_n'
Didn't find any fields in table 'EmailMessage_o'
Didn't find any fields in table 'EmailMessage_other'
Didn't find any fields in table 'EmailMessage_q'
Didn't find any fields in table 'EmailMessage_r'
Didn't find any fields in table 'EmailMessage_s'
Didn't find any fields in table 'EmailMessage_u'
Didn't find any fields in table 'EmailMessage_v'
Didn't find any fields in table 'EmailMessage_w'
Didn't find any fields in table 'EmailMessage_x'
Didn't find any fields in table 'EmailMessage_y'
Didn't find any fields in table 'EmailMessage_z'
Didn't find any fields in table 'EmailUIDL_a'
Didn't find any fields in table 'EmailUIDL_b'
Didn't find any fields in table 'EmailUIDL_c'
Didn't find any fields in table 'EmailUIDL_e'
Didn't find any fields in table 'EmailUIDL_f'
Didn't find any fields in table 'EmailUIDL_g'
Didn't find any fields in table 'EmailUIDL_h'
Didn't find any fields in table 'EmailUIDL_i'
Didn't find any fields in table 'EmailUIDL_j'
Didn't find any fields in table 'EmailUIDL_l'
Didn't find any fields in table 'EmailUIDL_m'
Didn't find any fields in table 'EmailUIDL_n'
Didn't find any fields in table 'EmailUIDL_o'
Didn't find any fields in table 'EmailUIDL_other'
Didn't find any fields in table 'EmailUIDL_p'
Didn't find any fields in table 'EmailUIDL_q'
Didn't find any fields in table 'EmailUIDL_r'
Didn't find any fields in table 'EmailUIDL_s'
Didn't find any fields in table 'EmailUIDL_t'
Didn't find any fields in table 'EmailUIDL_u'
Didn't find any fields in table 'EmailUIDL_v'
Didn't find any fields in table 'EmailUIDL_w'
Didn't find any fields in table 'EmailUIDL_x'
Didn't find any fields in table 'EmailUIDL_y'
Didn't find any fields in table 'EmailUIDL_z'
Didn't find any fields in table 'Log_SMS'
Didn't find any fields in table 'Log_Virus'
Didn't find any fields in table 'MailSort_a'
Didn't find any fields in table 'MailSort_b'
Didn't find any fields in table 'MailSort_c'
Didn't find any fields in table 'MailSort_e'
Didn't find any fields in table 'MailSort_f'
Didn't find any fields in table 'MailSort_g'
Didn't find any fields in table 'MailSort_h'
Didn't find any fields in table 'MailSort_i'
Didn't find any fields in table 'MailSort_j'
Didn't find any fields in table 'MailSort_l'
Didn't find any fields in table 'MailSort_m'
Didn't find any fields in table 'MailSort_n'
Didn't find any fields in table 'MailSort_o'
Didn't find any fields in table 'MailSort_other'
Didn't find any fields in table 'MailSort_p'
Didn't find any fields in table 'MailSort_q'
Didn't find any fields in table 'MailSort_r'
Didn't find any fields in table 'MailSort_s'
Didn't find any fields in table 'MailSort_t'
Didn't find any fields in table 'MailSort_u'
Didn't find any fields in table 'MailSort_v'
Didn't find any fields in table 'MailSort_w'
Didn't find any fields in table 'MailSort_x'
Didn't find any fields in table 'MailSort_y'
Didn't find any fields in table 'MailSort_z'
Didn't find any fields in table 'SMSCredits'
Didn't find any fields in table 'SMSqueue'
Didn't find any fields in table 'SMSsent'
Didn't find any fields in table 'SpamDB_a'
Didn't find any fields in table 'SpamDB_b'
Didn't find any fields in table 'SpamDB_c'
Didn't find any fields in table 'SpamDB_e'
Didn't find any fields in table 'SpamDB_f'
Didn't find any fields in table 'SpamDB_g'
Didn't find any fields in table 'SpamDB_h'
Didn't find any fields in table 'SpamDB_i'
Didn't find any fields in table 'SpamDB_j'
Didn't find any fields in table 'SpamDB_l'
Didn't find any fields in table 'SpamDB_m'
Didn't find any fields in table 'SpamDB_n'
Didn't find any fields in table 'SpamDB_o'
Didn't find any fields in table 'SpamDB_other'
Didn't find any fields in table 'SpamDB_p'
Didn't find any fields in table 'SpamDB_q'
Didn't find any fields in table 'SpamDB_r'
Didn't find any fields in table 'SpamDB_s'
Didn't find any fields in table 'SpamDB_t'
Didn't find any fields in table 'SpamDB_u'
Didn't find any fields in table 'SpamDB_v'
Didn't find any fields in table 'SpamDB_w'
Didn't find any fields in table 'SpamDB_x'
Didn't find any fields in table 'SpamDB_y'
Didn't find any fields in table 'SpamDB_z'
Didn't find any fields in table 'SpellCheck'
Didn't find any fields in table 'UserPgp_a'
Didn't find any fields in table 'UserPgp_b'
Didn't find any fields in table 'UserPgp_c'
Didn't find any fields in table 'UserPgp_d'
Didn't find any fields in table 'UserPgp_e'
Didn't find any fields in table 'UserPgp_f'
Didn't find any fields in table 'UserPgp_g'
Didn't find any fields in table 'UserPgp_h'
Didn't find any fields in table 'UserPgp_i'
Didn't find any fields in table 'UserPgp_j'
Didn't find any fields in table 'UserPgp_k'
Didn't find any fields in table 'UserPgp_l'
Didn't find any fields in table 'UserPgp_m'
Didn't find any fields in table 'UserPgp_n'
Didn't find any fields in table 'UserPgp_o'
Didn't find any fields in table 'UserPgp_other'
Didn't find any fields in table 'UserPgp_p'
Didn't find any fields in table 'UserPgp_q'
Didn't find any fields in table 'UserPgp_r'
Didn't find any fields in table 'UserPgp_s'
Didn't find any fields in table 'UserPgp_t'
Didn't find any fields in table 'UserPgp_u'
Didn't find any fields in table 'UserPgp_v'
Didn't find any fields in table 'UserPgp_w'
Didn't find any fields in table 'UserPgp_x'
Didn't find any fields in table 'UserPgp_y'
Didn't find any fields in table 'UserPgp_z'
Didn't find any fields in table 'UserSettings_f'
Didn't find any fields in table 'UserSettings_g'
Didn't find any fields in table 'UserSettings_h'
Didn't find any fields in table 'UserSettings_i'
Didn't find any fields in table 'UserSettings_l'
Didn't find any fields in table 'UserSettings_n'
Didn't find any fields in table 'UserSettings_o'
Didn't find any fields in table 'UserSettings_other'
Didn't find any fields in table 'UserSettings_q'
Didn't find any fields in table 'UserSettings_r'
Didn't find any fields in table 'UserSettings_u'
Didn't find any fields in table 'UserSettings_v'
Didn't find any fields in table 'UserSettings_w'
Didn't find any fields in table 'UserSettings_x'
Didn't find any fields in table 'UserSettings_y'
Didn't find any fields in table 'UserSettings_z'
Didn't find any fields in table 'titan_packages'
Database changed
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Didn't find any fields in table 'columns_priv'
Didn't find any fields in table 'db'
Didn't find any fields in table 'func'
Didn't find any fields in table 'host'
Didn't find any fields in table 'tables_priv'
Didn't find any fields in table 'user'
AT THIS POINT ... none of our perl or php scripts were able to contact the MySQL Server. It was shutdown again and restarted. Nothing. Apache was then restarted again, and it was allright.
We cannot access our data through the client, or run mysqldump or mysqlhotcopy at this point.
It also appears that Mysql MUST be started before Apache in order to cooperate w/ one another.
How to repeat:
mysqlhotcopy or mysqldump our database.
mysqldump: Got error: 1105: File './atmail/Calander_other.MYD' not found (Errcode: 9) when using LOCK TABLES
OR
DBD::mysql::db do failed: Can't find file: './atmail/EmailDatabase_g.frm'
(errno: 9) at /usr/local/bin/mysqlhotcopyline 438.
bash-2.05b# mysqladmin -u root -p shutdown
Enter password:
mysqladmin: connect to server at 'localhost' failed
error: 'Lost connection to MySQL server during query'
bash-2.05b# kill -9 PID
bash-2.05b# safe_mysqld &
bash-2.05b# apachctl stop
bash-2.05b# apachctl start
bash-2.05b# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 305 to server version: 3.23.57
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use atmail;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Didn't find any fields in table 'Calander_other'
Didn't find any fields in table 'Calander_p'
Didn't find any fields in table 'Calander_q'
..........
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Didn't find any fields in table 'columns_priv'
Didn't find any fields in table 'db'
Didn't find any fields in table 'func'
Didn't find any fields in table 'host'
Didn't find any fields in table 'tables_priv'
Didn't find any fields in table 'user'
Suggested fix:
Could the mysqldump or mysqlhotcopy have a buffering issue with the number of tables we are attempting to use? unlikely ... but suggestion?
We have recompiled mysql and the same issues exist w/ our database.
Interestingly, if I relocate our database to another machine and run mysqldump ... it will work. However, mysqlhotcopy will fail with the : DBD::mysql::db do failed: Can't find file: './atmail/EmailDatabase_g.frm'
(errno: 9) at /usr/local/bin/mysqlhotcopyline 438.
It should also be noted, that these files listed are not always the files in the error messages. Its be _k, _g, _i, etc ...
We really need a solution to this, we're flying on autopilot until we can get back into our database through the client.
