Bug #68783 Mysql crashes and continues to respawn and crash again
Submitted: 26 Mar 2013 17:39 Modified: 15 Apr 2013 5:46
Reporter: Aram Mirzadeh Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Installing Severity:S1 (Critical)
Version:5.6.10 OS:Linux (CentOS 6.4 64bit)
Assigned to: CPU Architecture:Any

[26 Mar 2013 17:39] Aram Mirzadeh
Description:
2013-03-26 13:22:19 3796 [Note] Server socket created on IP: '::'.
17:22:19 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=134217728
read_buffer_size=4194304
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1988600 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x374bc50
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fff01a21a48 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8ff755]
/usr/sbin/mysqld(handle_fatal_signal+0x3e8)[0x66e9a8]
/lib64/libpthread.so.0[0x31c500f500]
/usr/sbin/mysqld(_Z9get_fieldP11st_mem_rootP5Field+0x5a)[0x77dd9a]
/usr/sbin/mysqld[0x692713]
/usr/sbin/mysqld(_Z10acl_reloadP3THD+0x439)[0x696129]
/usr/sbin/mysqld(_Z8acl_initb+0x117)[0x69aee7]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x54e)[0x581ede]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x31c4c1ecdd]
/usr/sbin/mysqld(__gxx_personality_v0+0x2f1)[0x5764c9]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
130326 13:22:19 mysqld_safe Number of processes running now: 0
130326 13:22:19 mysqld_safe mysqld restarted

How to repeat:
Simply starting mysqld
[27 Mar 2013 6:53] MySQL Verification Team
Hello Aram,

Thank you for the report.

Can you start server with "skip-grant-tables" in the my.cnf and show us the structure of the mysql.user table?

Regards,
Umesh
[27 Mar 2013 8:43] Aram Mirzadeh
mysql> desc user;
+-----------------------+-----------------------------------+------+-----+---------+-------+
| Field                 | Type                              | Null | Key | Default | Extra |
+-----------------------+-----------------------------------+------+-----+---------+-------+
| Host                  | char(60)                          | NO   | PRI |         |       |
| User                  | char(16)                          | NO   | PRI |         |       |
| Password              | char(41)                          | NO   |     |         |       |
| Select_priv           | enum('N','Y')                     | NO   |     | N       |       |
| Insert_priv           | enum('N','Y')                     | NO   |     | N       |       |
| Update_priv           | enum('N','Y')                     | NO   |     | N       |       |
| Delete_priv           | enum('N','Y')                     | NO   |     | N       |       |
| Create_priv           | enum('N','Y')                     | NO   |     | N       |       |
| Drop_priv             | enum('N','Y')                     | NO   |     | N       |       |
| Reload_priv           | enum('N','Y')                     | NO   |     | N       |       |
| Shutdown_priv         | enum('N','Y')                     | NO   |     | N       |       |
| Process_priv          | enum('N','Y')                     | NO   |     | N       |       |
| File_priv             | enum('N','Y')                     | NO   |     | N       |       |
| Grant_priv            | enum('N','Y')                     | NO   |     | N       |       |
| References_priv       | enum('N','Y')                     | NO   |     | N       |       |
| Index_priv            | enum('N','Y')                     | NO   |     | N       |       |
| Alter_priv            | enum('N','Y')                     | NO   |     | N       |       |
| Show_db_priv          | enum('N','Y')                     | NO   |     | N       |       |
| Super_priv            | enum('N','Y')                     | NO   |     | N       |       |
| Create_tmp_table_priv | enum('N','Y')                     | NO   |     | N       |       |
| Lock_tables_priv      | enum('N','Y')                     | NO   |     | N       |       |
| Execute_priv          | enum('N','Y')                     | NO   |     | N       |       |
| Repl_slave_priv       | enum('N','Y')                     | NO   |     | N       |       |
| Repl_client_priv      | enum('N','Y')                     | NO   |     | N       |       |
| Create_view_priv      | enum('N','Y')                     | NO   |     | N       |       |
| Show_view_priv        | enum('N','Y')                     | NO   |     | N       |       |
| Create_routine_priv   | enum('N','Y')                     | NO   |     | N       |       |
| Alter_routine_priv    | enum('N','Y')                     | NO   |     | N       |       |
| Create_user_priv      | enum('N','Y')                     | NO   |     | N       |       |
| ssl_type              | enum('','ANY','X509','SPECIFIED') | NO   |     |         |       |
| ssl_cipher            | blob                              | NO   |     | NULL    |       |
| x509_issuer           | blob                              | NO   |     | NULL    |       |
| x509_subject          | blob                              | NO   |     | NULL    |       |
| max_questions         | int(11) unsigned                  | NO   |     | 0       |       |
| max_updates           | int(11) unsigned                  | NO   |     | 0       |       |
| max_connections       | int(11) unsigned                  | NO   |     | 0       |       |
| max_user_connections  | int(11) unsigned                  | NO   |     | 0       |       |
+-----------------------+-----------------------------------+------+-----+---------+-------+
37 rows in set (0.00 sec)
[29 Mar 2013 12:41] MySQL Verification Team
Thank you for the details.

Looks like grant tables are messed up, probably a upgrade from 5.0 or something.The user table column counts matches with that of 5.0 version, 
How did you upgrade? mysql_upgrade should be executed each time you upgrade MySQL.. 

5.0 user table - http://dev.mysql.com/doc/refman/5.0/en/grant-table-structure.html

5.6 user table - http://dev.mysql.com/doc/refman/5.6/en/grant-table-structure.html
[29 Mar 2013 20:30] Aram Mirzadeh
mysqldump each database (including mysql) from source 
mysql < sql on new server 
mysql_upgrade
[15 Apr 2013 5:46] MySQL Verification Team
Hello Aram,

It seems that you are trying to upgrade from 5.0 to 5.6 and official documentation says (http://dev.mysql.com/doc/refman/5.6/en/upgrading.html) :

"As a general rule, to upgrade from one release series to another, go to the
next series rather than skipping a series. To upgrade from a release series
previous to MySQL 5.5, upgrade to each successive release series in turn
until you have reached MySQL 5.5, and then proceed with the upgrade to MySQL
5.6.".

You are skipping couple of versions here in between. This issue has been identified by my senior colleague and reported in internal BugDB 16343542
This issue is observed if you start up 5.6.x and point it to a datadir containing system tables from 5.0.x, it just crashes.