Bug #27700 MySQL Server Fails To Start After RPM Install (InnoDB Error)
Submitted: 7 Apr 2007 18:38 Modified: 19 May 2007 22:21
Reporter: Nickolas Daskalou Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.0.37 OS:Other (CentOS 3)
Assigned to: Assigned Account CPU Architecture:Any
Tags: error, fails, hangs, innodb, install, installation, mysql_install_db, rpm, server

[7 Apr 2007 18:38] Nickolas Daskalou
Description:
When installing the MySQL 5.0.37 server using the Red Hat Enterprise Linux 3 RPM (x86), the RPM installation fails and prints out:

=====================================
warning: MySQL-server-community-5.0.37-0.rhel3.i386.rpm: V3 DSA signature: NOKEY, key ID 5072e1f5
[[*** NOTE: RPM INSTALLATION HANGS HERE FOR A FEW MINUTES!! ***]]]
/usr/bin/mysql_install_db: line 1: 30386 Killed                  /usr/sbin/mysqld --bootstrap --skip-grant-tables --basedir=/ --datadir=/var/lib/mysql --skip-innodb --skip-bdb --skip-ndbcluster --user=mysql --max_allowed_packet=8M --net_buffer_length=16K
/usr/bin/mysql_create_system_tables: line 778: 30391 Broken pipe             cat  <<END_OF_DATA
use mysql;
set table_type=myisam;
$c_d
$i_d

$c_h
$i_h

$c_u
$i_u

$c_f
$i_f

$c_t
$c_c

$c_ht
$c_hc
$c_hr
$c_hk

$c_tzn
$i_tzn
$c_tz
$i_tz
$c_tzt
$i_tzt
$c_tztt
$i_tztt
$c_tzls
$i_tzls

$c_p
$c_pp

END_OF_DATA

Installation of system tables failed!

Examine the logs in /var/lib/mysql for more information.
You can also try to start the mysqld daemon with:
/usr/sbin/mysqld --skip-grant &
You can use the command line tool
/usr/bin/mysql to connect to the mysql
database and look at the grant tables:

shell> /usr/bin/mysql -u root mysql
mysql> show tables

Try 'mysqld --help' if you have problems with paths. Using --log
gives you a log in /var/lib/mysql that may be helpful.

The latest information about MySQL is available on the web at
http://www.mysql.com
Please consult the MySQL manual section: 'Problems running mysql_install_db',
and the manual section that describes problems on your OS.
Another information source is the MySQL email archive.
Please check all of the above before mailing us!
And if you do mail us, you MUST use the /usr/bin/mysqlbug script!
Starting MySQL/etc/init.d/mysql: line 369:  9744 Killed                  $bindir/mysqld_safe --datadir=$datadir --pid-file=$server_pid_file $other_args >/dev/null 2>&1
/etc/init.d/mysql: line 151: kill: (9744) - No such process
 ERROR!
======================================

The contents of /var/lib/mysql/hostname.err is:

======================================
070408 04:08:05  mysqld started
InnoDB: The first specified data file /var/lib/mysql/ibdata1 did not exist:
InnoDB: a new database to be created!
070408  4:08:05  InnoDB: Setting file /var/lib/mysql/ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
070408  4:08:05  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 12 MB
InnoDB: Database physically writes the file full: wait...
070408  4:08:06  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 12 MB
InnoDB: Database physically writes the file full: wait...
======================================

In fact I've been trying to install MySQL 5.0.37 with InnoDB support (as set in my /etc/my.cnf files) for the past 7 hours with no luck at all. I can manage to get the server up and running but without InnoDB support. From what I can see from the error log file, it hangs straight after these lines:

======================================
InnoDB: Setting log file ./ib_logfile1 size to 12 MB
InnoDB: Database physically writes the file full: wait...
======================================

My /etc/my.cnf file:

======================================
[mysqld]
pid-file=/var/lib/mysql/mysql.pid
set-variable = max_connections=500
safe-show-database
default-character-set=utf8
collation-server=utf8_unicode_ci
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

#InnoDB Stuff

# The following line just enables the InnoDB Storage Engine
innodb

# The following line makes MySQL create a separate file
# for each table of Storage Engine type InnoDB
innodb_file_per_table

innodb_data_home_dir=/var/lib/mysql

# Data files must be able to hold your data and indexes.
# Make sure that you have enough free disk space.
innodb_data_file_path = ibdata1:10M:autoextend
#
# Set buffer pool size to 50-80% of your computer's memory
innodb_buffer_pool_size=40M
innodb_additional_mem_pool_size=10M
#
# Set the log file size to about 25% of the buffer pool size
innodb_log_file_size=12M
innodb_log_buffer_size=8M
#
innodb_flush_log_at_trx_commit=1

[client]
socket=/var/lib/mysql/mysql.sock
======================================

I can't believe it's this hard to get InnoDB working! I had MySQL 5.0.27 installed previously and it ran "out of the box" without ever hanging or failing to start up (Although I never checked to see if InnoDB was enabled).

How to repeat:
1. Shutdown or uninstall any previous versions of the MySQL server.

2. At the command prompt:

> wget http://dev.mysql.com/get/Downloads/MySQL-5.0/MySQL-server-community-5.0.37-0.rhel3.i386.rp...
> rpm -Uvh MySQL-server-community-5.0.37-0.rhel3.i386.rpm
[7 Apr 2007 18:53] Nickolas Daskalou
I tried installing the 5.0.27 RPM and it worked perfectly - no hanging and InnoDB worked out of the box!

I'll see if 5.0.33 works...
[7 Apr 2007 19:09] Nickolas Daskalou
5.0.33 works using the Generic Linux RPMs for i386 found from here:

http://server.macvicar.net/downloads/mysql/generic/archive/

I'll see if the 5.0.36 RPMs from their site work too...
[8 Apr 2007 16:40] Nickolas Daskalou
Ok, MySQL 5.0.36 doesn't seem to work either:

============================================
> rpm -i MySQL-server-5.0.36-0.glibc23.i386.rpm
Error in my_thread_global_end(): 1 threads didn't exit
Error in my_thread_global_end(): 1 threads didn't exit
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h host.mymy.com.au password 'new-password'
See the manual for more instructions.

NOTE:  If you are upgrading from a MySQL <= 3.22.10 you should run
the /usr/bin/mysql_fix_privilege_tables. Otherwise you will not be
able to use the new GRANT command!

Please report any problems with the /usr/bin/mysqlbug script!

The latest information about MySQL is available on the web at
http://www.mysql.com
Support MySQL by buying support/licenses at http://shop.mysql.com
Starting MySQL....................................................................................................................................................................................../var/tmp/rpm-tmp.41581: line 43: 27200 Terminated              /etc/init.d/mysql start
============================================
[10 Apr 2007 13:30] Heikki Tuuri
Nickolas,

if you delete the InnoDB files ibdata1, ib_logfile0, ib_logfile1, does the .err log show that InnoDB fails to complete the write to ib_logfile1? What is the size of that file in the MySQL datadir?

Why did the script in the first printout print that the installation of the MySQL system tables failed? Does the script time out if InnoDB hangs in the file write?

Maybe the .err log is not flushed completely so that we did not see all that mysqld wrote there? Do you get a different printout if you delete the InnoDB files and start mysqld manually in a shell:

heikki@ws38:~/mysql-5.0/sql$ ./mysqld
070410 16:30:07 [Warning] Changed limits: max_open_files: 1024  max_connections: 886  table_cache: 64
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
070410 16:30:07  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
070410 16:30:07  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 20 MB
InnoDB: Database physically writes the file full: wait...
070410 16:30:08  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 20 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
070410 16:30:09  InnoDB: Started; log sequence number 0 0
070410 16:30:09 [Note] /home/heikki/mysql-5.0/sql/.libs/lt-mysqld: ready for connections.
Version: '5.0.38-debug-log'  socket: '/home/heikki/bugsocket'  port: 3307  Source distribution
...

Regards,

Heikki
[12 Apr 2007 0:28] Nickolas Daskalou
Hi Heikki,

> if you delete the InnoDB files ibdata1, ib_logfile0, ib_logfile1, does the .err log show that InnoDB fails to complete the write to ib_logfile1?

I'm not sure what happens when I delete those files. I've got a stable server running now (5.0.33) so I don't want to try this right now, but I might give it a go again on the weekend.

> What is the size of that file in the MySQL datadir?

The size of each of those files in the MySQL datadir was 10MB for the ibdata1 file and 12MB for each log file (as was set in the configuration file).

> Why did the script in the first printout print that the installation of the
MySQL system tables failed? Does the script time out if InnoDB hangs in the file
write?

Yes, the script timed out on its own. It's weird because when the script hangs for a few minutes, the ibdata1 and 2 x log files have already been created.

> Maybe the .err log is not flushed completely so that we did not see all that
mysqld wrote there? Do you get a different printout if you delete the InnoDB
files and start mysqld manually in a shell

Nope, I got the same printout. However, when I installed a working version (5.0.27 or 5.0.33, the following was also printed in the .err file:

============================================
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
070408  4:41:28  InnoDB: Started; log sequence number 0 0
============================================

Cheers,

Nick
[12 Apr 2007 12:36] Heikki Tuuri
I am putting this to the 'Need Feedback' status. If there is something wrong with the .rpm's, we will get feedback from other MySQL installers.

Regards,

Heikki
[18 Apr 2007 19:43] Wayne Seguin
We too have encountered this issue. 
We compiled 5.0.37 from source on RHEL 4.
[19 Apr 2007 3:54] Nickolas Daskalou
@ Wayne Seguin:

So you guys didn't install using RPMs but you did get the same InnoDB error?
[19 May 2007 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[30 May 2008 2:04] Phil Taylor
If you need to move the datadir location in mysql then do ln -s /location/here /var/lib/mysql

and dont bother setting the datadir in the my.cnf file - worked for me anyway.