MySQL Bugs: #27752: Upgrade of Mysql-server doesn't work, MySQL is completely stopped

Bug #27752	Upgrade of Mysql-server doesn't work, MySQL is completely stopped
Submitted:	11 Apr 2007 10:12	Modified:	10 Dec 2007 4:47
Reporter:	media forest	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	5.0.38-1	OS:	Linux
Assigned to:		CPU Architecture:	Any

Description:
When trying to upgrade mysql-server through apt-get on a debian system, Mysql wente completely out of order.
I got these messages in syslog : 

Apr 11 10:59:32 ns1 mysqld_safe[10539]: Installation of system tables failed!
Apr 11 10:59:32 ns1 mysqld_safe[10539]: 
Apr 11 10:59:32 ns1 mysqld_safe[10539]: Examine the logs in /var/lib/mysql for more information.
Apr 11 10:59:32 ns1 mysqld_safe[10539]: You can try to start the mysqld daemon with:
Apr 11 10:59:32 ns1 mysqld_safe[10539]: /usr/sbin/mysqld --skip-grant &
Apr 11 10:59:32 ns1 mysqld_safe[10539]: and use the command line tool
Apr 11 10:59:32 ns1 mysqld_safe[10539]: /usr/bin/mysql to connect to the mysql
Apr 11 10:59:32 ns1 mysqld_safe[10539]: database and look at the grant tables:
Apr 11 10:59:32 ns1 mysqld_safe[10539]: 
Apr 11 10:59:32 ns1 mysqld_safe[10539]: shell> /usr/bin/mysql -u root mysql
Apr 11 10:59:32 ns1 mysqld_safe[10539]: mysql> show tables
Apr 11 10:59:32 ns1 mysqld_safe[10539]: 
Apr 11 10:59:32 ns1 mysqld_safe[10539]: Try 'mysqld --help' if you have problems with paths. Using --log
Apr 11 10:59:32 ns1 mysqld_safe[10539]: gives you a log in /var/lib/mysql that may be helpful.
Apr 11 10:59:32 ns1 mysqld_safe[10539]: 
Apr 11 10:59:32 ns1 mysqld_safe[10539]: The latest information about MySQL is available on the web at
Apr 11 10:59:32 ns1 mysqld_safe[10539]: http://www.mysql.com
Apr 11 10:59:32 ns1 mysqld_safe[10539]: Please consult the MySQL manual section: 'Problems running mysql_install_db',
Apr 11 10:59:32 ns1 mysqld_safe[10539]: and the manual section that describes problems on your OS.
Apr 11 10:59:32 ns1 mysqld_safe[10539]: Another information source is the MySQL email archive.
Apr 11 10:59:32 ns1 mysqld_safe[10539]: Please check all of the above before mailing us!
Apr 11 10:59:32 ns1 mysqld_safe[10539]: And if you do mail us, you MUST use the /usr/bin/mysqlbug script!
Apr 11 11:10:17 ns1 mysqld_safe[12889]: Installation of system tables failed!

How to repeat:
The messages in syslog doesn't help at all because :
"Apr 11 10:59:32 ns1 mysqld_safe[10539]: Examine the logs in /var/lib/mysql for more information."

I haven't any logs in the /var/lib/mysq directory

/usr/bin/mysql -u root mysql doesn't work

/usr/bin/mysqlbug script doesn't work

Thank you for the report.

Do you upgrade or do clean installation? Can you start server with `/usr/sbin/mysqld --skip-grant &` command?

I was upgrading from 5.0.32 to 5.0.38-1
doing apt-get upgrade

I manage to start mysqd with '/usr/sbin/mysqld --skip-grant &' but mysql server does'nt answer though I find mysqld in running processes.

I tried on another computer and it hangs exactly the same way while throwing "Stopping Mysql database server: mysqld."

Here is the mysql errors log :

Error in my_thread_global_end(): 1 threads didn't exit
070411 10:41:04 - mysqld got signal 4;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 233983 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbffff118, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81c5000
0x4005f806
0x4005cb64
0x4005b728
0x400590eb
0x81c78a2
0x402133be
0x813c951
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
ERROR: 1062  Duplicata du champ 'localhost-root' pour la clef 1
070411 10:59:27 [ERROR] Aborting

070411 10:59:27 [Note] /usr/sbin/mysqld: Arrêt du serveur terminé

Error in my_thread_global_end(): 1 threads didn't exit
ERROR: 1062  Duplicata du champ 'localhost-root' pour la clef 1

070411 12:35:09 [ERROR] /usr/sbin/mysqld: Ne peut trouver le fichier: './mysql/host.frm' (Errcode: 13)
070411 12:35:09 [ERROR] /usr/sbin/mysqld: Ne peut trouver le fichier: './mysql/host.frm' (Errcode: 13)
070411 12:35:09 [ERROR] Fatal error: Can't open and lock privilege tables: Ne peut trouver le fichier: './mysql/host.frm' (Errcode: 13)

Please, send the resolved stack trace.

Excuse me, but I don't understand exactly what you mean...
How can I "send the resolved stack trace" ?

You have to take it from the error log:

0x81c5000
0x4005f806
0x4005cb64
0x4005b728
0x400590eb
0x81c78a2
0x402133be
0x813c951

put it into a separate file and then perform actions described in the manual, 
http://dev.mysql.com/doc/internals/en/using-stack-trace.html. Then send the results of that actions - stack trace resolved over your (Debian!) mysqld binary.

Ok,
Here are the stack traces :
first one
0x81c5000 handle_segfault + 784
0x4005f806 _end + 932620758
0x4005cb64 _end + 932609332
0x4005b728 _end + 932604152
0x400590eb _end + 932594363
0x81c78a2 main + 1442
0x402133be _end + 934405518
0x813c951 _start + 33

second one
0x81c5000 handle_segfault + 784
0x4005f806 _end + 932620758
0x4005cb64 _end + 932609332
0x4005b728 _end + 932604152
0x4005e188 _end + 932615000
0x4005b1ea _end + 932602810
0x81c3673 signal_hand + 355
0x40059c51 _end + 932597281
0x402b838a _end + 935081306

My problem is exactly same.

I made some investigation. There is a variable in the Debian post installation script:
    password_column_fix_query=`/bin/echo -e \
        "USE mysql\n" \
        "ALTER TABLE user CHANGE password Password varchar(41) collate utf8_bin NOT NULL default ''"`;

...A few lines later...

The script hangs exactly at this line:
    echo "$password_column_fix_query"                             | $MYSQL_BOOTSTRAP 2>&1 | $ERR_LOGGER

At this point there are some mysqld process running:

ps aux|grep mysql
root      7989  0.1  0.5   2720  1364 pts/0    S    14:47   0:00 /bin/bash -e /var/lib/dpkg/info/mysql-server-5.0.postinst configure
mysql     8085  0.1  1.8  19688  4740 pts/0    S    14:47   0:00 /usr/sbin/mysqld --bootstrap --user=mysql --skip-grant-tables --skip-bdb --skip-innodb
root      8087  0.0  1.8  19688  4740 pts/0    S    14:47   0:00 /usr/sbin/mysqld --bootstrap --user=mysql --skip-grant-tables --skip-bdb --skip-innodb
root      8089  0.0  1.8  19688  4740 pts/0    S    14:47   0:00 /usr/sbin/mysqld --bootstrap --user=mysql --skip-grant-tables --skip-bdb --skip-innodb

I hope these informations were useful for developers.

Here's the answer I received from debian MySQL maintainer :
>MySQL-5.0.36 and above are not compatible with kernel 2.4.x. Please either
>upgrade to kernel 2.6 (something else isn't supported in Debian 4.0 etch
>anyway) or downgrade MySQL.

So I upgraded the kernel then the installation worked !

So, looks like we can close this report as not a result of bug in MySQL's code.

After upgrade my kernel I get the same error

linux-image-2.6.18-4-686_2.6.18.dfsg.1-12etch1
linux-image-2.6.18-4-686_2.6.18.dfsg.1-12_i386

 2.6.18-4-686 
http://www.us.debian.org/security/2007/dsa-1286

All reporters:

Please, send the results of

getconf GNU_LIBC_VERSION
getconf GNU_LIBPTHREAD_VERSION

from the systems affected.

$ getconf GNU_LIBC_VERSION
  glibc 2.3.6

$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.3.6

When the option skip-innodb is used, the mysql server start without errors

All reporters:

Please, upload your entire error logs to this report.

May  7 11:40:51 dimension mysqld[32554]:                                                                       ;InnoDB: End of page dump
May  7 11:40:51 dimension mysqld[32554]: 070507 11:40:51  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
May  7 11:40:51 dimension mysqld[32554]: InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
May  7 11:40:51 dimension mysqld[32554]: InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
May  7 11:40:51 dimension mysqld[32554]: InnoDB: Page number (if stored to page already) 0,
May  7 11:40:51 dimension mysqld[32554]: InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
May  7 11:40:51 dimension mysqld[32554]: 070507 11:40:51InnoDB: Error: trying to access a stray pointer 0x3626fff8
May  7 11:40:51 dimension mysqld[32554]: InnoDB: buf pool start is at 0xb6260000, end at 0xb6a60000
May  7 11:40:51 dimension mysqld[32554]: InnoDB: Probable reason is database corruption or memory
May  7 11:40:51 dimension mysqld[32554]: InnoDB: corruption. If this happens in an InnoDB database recovery, see
May  7 11:40:51 dimension mysqld[32554]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
May  7 11:40:51 dimension mysqld[32554]: InnoDB: how to force recovery.
May  7 11:40:51 dimension mysqld[32554]: 070507 11:40:51InnoDB: Assertion failure in thread 3082991296 in file ./../include/buf0buf.ic line 259
May  7 11:40:51 dimension mysqld[32554]: InnoDB: We intentionally generate a memory trap.
May  7 11:40:51 dimension mysqld[32554]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
May  7 11:40:51 dimension mysqld[32554]: InnoDB: If you get repeated assertion failures or crashes, even
May  7 11:40:51 dimension mysqld[32554]: InnoDB: immediately after the mysqld startup, there may be
May  7 11:40:51 dimension mysqld[32554]: InnoDB: corruption in the InnoDB tablespace. Please refer to
May  7 11:40:51 dimension mysqld[32554]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
May  7 11:40:51 dimension mysqld[32554]: InnoDB: about forcing recovery.
May  7 11:40:51 dimension mysqld[32554]: mysqld got signal 11;
May  7 11:40:51 dimension mysqld[32554]: This could be because you hit a bug. It is also possible that this binary
May  7 11:40:51 dimension mysqld[32554]: or one of the libraries it was linked against is corrupt, improperly built,
May  7 11:40:51 dimension mysqld[32554]: or misconfigured. This error can also be caused by malfunctioning hardware.
May  7 11:40:51 dimension mysqld[32554]: We will try our best to scrape up some info that will hopefully help diagnose
May  7 11:40:51 dimension mysqld[32554]: the problem, but since we have already crashed, something is definitely wrong
May  7 11:40:51 dimension mysqld[32554]: and this may fail.
May  7 11:40:51 dimension mysqld[32554]:
May  7 11:40:51 dimension mysqld[32554]: key_buffer_size=0
May  7 11:40:51 dimension mysqld[32554]: read_buffer_size=131072
May  7 11:40:51 dimension mysqld[32554]: max_used_connections=0
May  7 11:40:51 dimension mysqld[32554]: max_connections=100
May  7 11:40:51 dimension mysqld[32554]: threads_connected=0
May  7 11:40:51 dimension mysqld[32554]: It is possible that mysqld could use up to
May  7 11:40:51 dimension mysqld[32554]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 217599 K
May  7 11:40:51 dimension mysqld[32554]: bytes of memory
May  7 11:40:51 dimension mysqld[32554]: Hope that's ok; if not, decrease some variables in the equation.
May  7 11:40:51 dimension mysqld[32554]:
May  7 11:40:51 dimension mysqld[32554]: thd=(nil)
May  7 11:40:51 dimension mysqld[32554]: Attempting backtrace. You can use the following information to find out
May  7 11:40:51 dimension mysqld[32554]: where mysqld died. If you see no messages after this, something went
May  7 11:40:51 dimension mysqld[32554]: terribly wrong...
May  7 11:40:51 dimension mysqld[32554]: Cannot determine thread, fp=0xbff9ed78, backtrace may not be correct.
May  7 11:40:51 dimension mysqld[32554]: Stack range sanity check OK, backtrace follows:
May  7 11:40:51 dimension mysqld[32554]: 0x81c0659
May  7 11:40:51 dimension mysqld[32554]: 0x83cd2b3
May  7 11:40:51 dimension mysqld[32554]: 0x834d25e
May  7 11:40:51 dimension mysqld[32554]: 0x83578d5
May  7 11:40:51 dimension mysqld[32554]: 0x8305830
May  7 11:40:51 dimension mysqld[32554]: Stack trace seems successful - bottom reached
May  7 11:40:51 dimension mysqld[32554]: Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
May  7 11:40:51 dimension mysqld[32554]: stack trace is much more helpful in diagnosing the problem, so please do
May  7 11:40:51 dimension mysqld[32554]: resolve it
May  7 11:40:51 dimension mysqld[32554]: The manual page at http://www.mysql.com/doc/en/Crashing.html contains
May  7 11:40:51 dimension mysqld[32554]: information that should help you find out what is causing the crash.
May  7 11:40:51 dimension mysqld_safe[32562]: ended

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Closing as "Not a bug" because initial problem was caused by changes in Debian packaging. 

Other reporters with upgrade problems:

please, open a separate bug reports for them.