Bug #44930 Live upgrade to 5.4 fails as configuration of innodb logfiles has changed.
Submitted: 18 May 2009 11:39 Modified: 19 May 2009 20:25
Reporter: Vemund Østgaard Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:5.4.0 OS:Linux (2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux)
Assigned to: CPU Architecture:Any

[18 May 2009 11:39] Vemund Østgaard
Description:
I'm seeing a problem when doing live upgrade from 5.1.34 to 5.4.0.

It seems that innodb tries to create a third logfile when the 5.4.0 version is started against an existing database and this causes a failure during initialization of innodb and it does not get registered as a storage engine. After this mysqld segfaults when the mysqlcheck command is run, apparently as part of doing something related to performance_schema:

Running 'mysqlcheck'...
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqlcheck: Got error: 2013: Lost connection to MySQL server at 'reading authorization packet', system error: 0 when trying to connect
FATAL ERROR: Upgrade failed

Stacktrace from master.err:

090414 19:33:16 InnoDB: Log file ./ib_logfile2 did not exist: new to be created
090414 19:33:16 [ERROR] Plugin 'InnoDB' init function returned error.
090414 19:33:16 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
090414 19:33:16 [Warning] /home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld: unknown option '--loose-skip-ndbcluster'
090414 19:33:16 [Note] Event Scheduler: Loaded 0 events
090414 19:33:16 [Note] /home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld: ready for connections.
Version: '5.4.0-beta-log' socket: '/tmp/systest/var/master/tmp/master.sock' port: 9306 MySQL Community Server (GPL)
090414 19:34:35 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60541 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x13e3200
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4b4931a0 thread_stack 0x40000
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(my_print_stacktrace+0x24) [0x89fb34]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(handle_segfault+0x322) [0x5e0b12]
/lib64/tls/libpthread.so.0 [0x360d30c420]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(plugin_lock(THD*, st_plugin_int**)+0x21) [0x74edf1]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(ha_checktype(THD*, legacy_db_type, bool, bool)+0x10c) [0x6cff9c]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(open_table_def(THD*, st_table_share*, unsigned int)+0x14ad) [0x63517d]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(get_table_share(THD*, TABLE_LIST*, char*, unsigned int, unsigned int, int*)+0x201) [0x623d21]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld [0x629cd2]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(open_table(THD*, TABLE_LIST*, st_mem_root*, bool*, unsigned int)+0x5b7) [0x62ab57]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(open_ltable(THD*, TABLE_LIST*, thr_lock_type, unsigned int)+0x65) [0x62b1a5]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(open_performance_schema_table(THD*, TABLE_LIST*, Open_tables_state*)+0x46) [0x62b376]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(Log_to_csv_event_handler::log_general(THD*, long, char const*, unsigned int, int, char const*, unsigned int, char const*, unsigned int, charset_info_st*)+0xfb) [0x679dbb]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(LOGGER::general_log_write(THD*, enum_server_command, char const*, unsigned int)+0x157) [0x677357]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(LOGGER::general_log_print(THD*, enum_server_command, char const*, __va_list_tag*)+0x51) [0x677461]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(general_log_print(THD*, enum_server_command, char const*, ...)+0xdb) [0x67757b]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(check_user(THD*, enum_server_command, char const*, unsigned int, char const*, bool)+0x24b) [0x5e992b]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld [0x5ea25c]
/home/vo136787/mysql/mysql-5.4.0-beta-linux-x86_64-glibc23/bin/mysqld(handle_one_connection+0x141) [0x5ea6c1]
/lib64/tls/libpthread.so.0 [0x360d30610a]
/lib64/tls/libc.so.6(__clone+0x73) [0x360cac6003]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil) is an invalid pointer
thd->thread_id=1
thd->killed=NOT_KILLED 

Some testing shows that not only the default number of logfiles for innodb has changed in 5.4.0, but also the default size. The default for innodb_log_file_size has changed (from 5MB to 128MB), the *minumum* has also increased from 1MB to 32MB.  In particular, the new *minimum* is greater than the old *default*, which means users cannot explicitly set identical log file size when upgrading unless their existing InnoDB log files are at least as large as the new minimum.

How to repeat:
1. Create a database using 5.1.34 (or any 5.1 version)
2. Stop mysqld
3. Start a 5.4.0 version of mysqld with the existing database files.
[19 May 2009 20:25] Omer Barnir
This issue is covered in the 5.4 feature summary, the 5.4.0 changelog, and the 5.4 upgrade notes:

http://dev.mysql.com/doc/mysql-5.4-features/en/availability-installation.html
http://dev.mysql.com/doc/refman/5.4/en/news-5-4-0.html
http://dev.mysql.com/doc/refman/5.4/en/upgrading-from-5-1.html

There is also an upgrade incompatibility related to innodb_file_io_threads.

As such closing the bug
[19 May 2009 20:29] Omer Barnir
To clarify, the issue is an 'existing' issue with changing the number of innodb files when upgrading - nothing specific to 5.4 - it was just 'exposed' in 5.4 as the default settings were changed