Bug #16470 crash on grant if old grant tables
Submitted: 13 Jan 2006 0:28 Modified: 19 Jun 2008 1:42
Reporter: Elan Ruusamäe (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.0.18 OS:Any (PLD Linux)
Assigned to: Kristofer Pettersson
Triage: D3 (Medium)

[13 Jan 2006 0:28] Elan Ruusamäe
Description:
running GRANT command causes server crash.
mysql> GRANT SELECT ON horde_histories TO lamp@localhost;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Version: '5.0.18'  socket: '/var/lib/mysql/mysqldb/mysql.sock'  port: 0  PLD Linux Distribution MySQL RPM
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8b01600
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
frame pointer (ebp) is NULL, did you compile with
-fomit-frame-pointer? Aborting backtrace!
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8b162a8 = GRANT SELECT ON horde_histories TO lamp@localhost
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
insert into mysql database: http://haarber.dyn.ee/pld/db-mysql.sql
insert into horde database: http://haarber.dyn.ee/pld/db-horde_histories.sql

run on horde database: GRANT SELECT ON horde_histories TO lamp@localhost;
[13 Jan 2006 7:55] Elan Ruusamäe
here's stack dump. i hope it's useful.
# mysql_resolve_stack_dump -s mysqld.sym -n mysqld.stack
0x81c68af handle_segfault + 437
0xffffe420 _end + -141867536
(nil)
0xb7d7b7f8 _end + -1352459832
0x84f46f5 alloc_root + 162
0x818d004 _Z9sql_allocj + 42
0x812fd07 _ZN9Sql_allocnwEj + 17
0x82747f6 _Z17mysql_table_grantP3THDP13st_table_listR4ListI11st_lex_userERS3_I10LEX_COLUMNEmb + 2370
0x81e2f48 _Z21mysql_execute_commandP3THD + 17976
0x81e7eba _Z11mysql_parseP3THDPcj + 460
0x81dcf7d _Z16dispatch_command19enum_server_commandP3THDPcj + 1953
0x81dc7d1 _Z10do_commandP3THD + 577
0x81db866 handle_one_connection + 836
0xb7f60c7b _end + -1350472117
0xb7e11bbe _end + -1351844466
# mysql_resolve_stack_dump -s mysqld.sym -n mysqld.stack2
0x81c68af handle_segfault + 437
0xffffe420 _end + -141867536
(nil)
0xb7d107f8 _end + -1352898104
0x84f46f5 alloc_root + 162
0x818d004 _Z9sql_allocj + 42
0x812fd07 _ZN9Sql_allocnwEj + 17
0x82747f6 _Z17mysql_table_grantP3THDP13st_table_listR4ListI11st_lex_userERS3_I10LEX_COLUMNEmb + 2370
0x81e2f48 _Z21mysql_execute_commandP3THD + 17976
0x81e7eba _Z11mysql_parseP3THDPcj + 460
0x81dcf7d _Z16dispatch_command19enum_server_commandP3THDPcj + 1953
0x81dc7d1 _Z10do_commandP3THD + 577
0x81db866 handle_one_connection + 836
0xb7ef5c7b _end + -1350910389
0xb7da6bbe _end + -1352282738
mysql_resolve_stack_dump -s mysqld.sym -n mysqld.stack3
0x81c68af handle_segfault + 437
0xffffe420 _end + -141867536
(nil)
0xb7d097f8 _end + -1352926776
0x84f46f5 alloc_root + 162
0x818d004 _Z9sql_allocj + 42
0x812fd07 _ZN9Sql_allocnwEj + 17
0x82747f6 _Z17mysql_table_grantP3THDP13st_table_listR4ListI11st_lex_userERS3_I10LEX_COLUMNEmb + 2370
0x81e2f48 _Z21mysql_execute_commandP3THD + 17976
0x81e7eba _Z11mysql_parseP3THDPcj + 460
0x81dcf7d _Z16dispatch_command19enum_server_commandP3THDPcj + 1953
0x81dc7d1 _Z10do_commandP3THD + 577
0x81db866 handle_one_connection + 836
0xb7eeec7b _end + -1350939061
0xb7d9fbbe _end + -1352311410
[13 Jan 2006 9:57] Miguel Solorzano
I was unable to repeat the behavior reported:

mysql> select version();
+--------------+
| version()    |
+--------------+
| 5.0.19-debug |
+--------------+
1 row in set (0.00 sec)

mysql> GRANT SELECT ON horde_histories TO lamp@localhost;
Query OK, 0 rows affected (0.00 sec)

However I wasn't applied the script provide for the mysql db,
so could you please inform: is necessary to apply that script
for the crash to happen? Which server version is the source
for that script?

Thanks in advance.
[13 Jan 2006 10:01] Elan Ruusamäe
version 5.0.18 (as marked in the bug) and yes. that structure of mysql db is needed to crash.
[13 Jan 2006 10:32] Miguel Solorzano
I was unable to repeat with current source server. Anyway I think isn't
a good practice to modify the grant tables. There are modifications done
manually in the mysql_db.sql script?

miguel@hegel:~/dbs/5.0> bin/mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 5.0.19-debug

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use mysql
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> source /home/miguel/s/db-mysql.sql
Query OK, 0 rows affected (0.00 sec)

<cut>

mysql> create database horde;
Query OK, 1 row affected (0.00 sec)

mysql> use horde
Database changed
mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)

mysql> source /home/miguel/s/db-horde_histories.sql
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)
<cut>

mysql> GRANT SELECT ON horde_histories TO lamp@localhost;
Query OK, 0 rows affected (0.01 sec)
[13 Jan 2006 11:41] Elan Ruusamäe
it's dump of `mysql' and `horde' databases, with entries removed which weren't needed to crash. i didn't modify `mysql' table manually.

925  mysqldump -umysql mysql --extended-insert=false> db-mysql.sql
933  mysqldump -umysql horde --extended-insert=false horde_histories> db-horde_histories.sql
[13 Jan 2006 12:01] Elan Ruusamäe
perhaps you try inserting that db-mysql.sql to empty database?
anyway, here's with recent snapshot and -debug build.

# mysql -umysql mysql
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 5.0.19-nightly-20060111-debug

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> GRANT SELECT ON horde_histories TO lamp@localhost;
ERROR 2013 (HY000): Lost connection to MySQL server during query

mysqld: my_alloc.c:173: alloc_root: Assertion `((mem_root)->min_malloc != 0)' failed.
mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8d97ff0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb50b2d28, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81c6d81
0xffffe420
(nil)
0xb7d977f8
0x84f4c65
0x818d044
0x812fe17
0x8274d04
0x81e3436
0x81e83c4
0x81dd459
0x81dccad
0x81dbd36
0xb7f7cc7b
0xb7e2dbbe
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8dacd80 = GRANT SELECT ON horde_histories TO lamp@localhost
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

# mysql_resolve_stack_dump -s /usr/share/mysql/mysqld.sym -n mysqld.stack4
0x81c6d81 handle_segfault + 437
0xffffe420 _end + -141871984
(nil)
0xb7d977f8 _end + -1352349592
0x84f4c65 alloc_root + 162
0x818d044 _Z9sql_allocj + 42
0x812fe17 _ZN9Sql_allocnwEj + 17
0x8274d04 _Z17mysql_table_grantP3THDP13st_table_listR4ListI11st_lex_userERS3_I10LEX_COLUMNEmb + 2370
0x81e3436 _Z21mysql_execute_commandP3THD + 17994
0x81e83c4 _Z11mysql_parseP3THDPcj + 460
0x81dd459 _Z16dispatch_command19enum_server_commandP3THDPcj + 1953
0x81dccad _Z10do_commandP3THD + 577
0x81dbd36 handle_one_connection + 836
0xb7f7cc7b _end + -1350361877
0xb7e2dbbe _end + -1351734226

and in case you can't reproduce, try executing these commands:
--------------
wget http://haarber.dyn.ee/pld/db-mysql.sql
wget http://haarber.dyn.ee/pld/db-horde_histories.sql
rm -rf mysql horde
env -i /usr/sbin/mysqld --defaults-file=/var/lib/mysql/mysqld.conf --skip-grant-tables &
mysqladmin create mysql
mysql mysql <db-mysql.sql
mysqladmin create horde
mysql horde <db-horde_histories.sql
mysqladmin shutdown
env -i /usr/sbin/mysqld --defaults-file=/var/lib/mysql/mysqld.conf  &
echo 'GRANT SELECT ON horde_histories TO lamp@localhost;' | mysql -umysql horde
--------------
also i've put mysqld.conf used online: http://haarber.dyn.ee/pld/mysqld.conf
[21 Jan 2006 13:48] Arkadiusz Miskiewicz
I'm also able to repeat the problem with grant:

grant select on track.User to 'vacation'@'1.1.1.1' IDENTIFIED BY 'xxx.8';

and...
Version: '5.0.18-debug-log'  socket: '/var/lib/mysql/mysqldb/mysql.sock'  port: 3307  PLD Linux Distribution MySQL RPM
mysqld: my_alloc.c:173: alloc_root: Assertion `((mem_root)->min_malloc != 0)' failed.
mysqld got signal 6;

when doing --debug --single-thread debug:
Version: '5.0.18-debug-log'  socket: '/var/lib/mysql/mysqldb/mysql.sock'  port: 3307  PLD Linux Distribution MySQL RPM
mysqld: my_alloc.c:173: alloc_root: Assertion `((mem_root)->min_malloc != 0)' failed.

Program received signal SIGABRT, Aborted.
0x00002aaaaba4c901 in raise () from /lib64/tls/libc.so.6
(gdb) bt
#0  0x00002aaaaba4c901 in raise () from /lib64/tls/libc.so.6
#1  0x00002aaaaba4e06c in abort () from /lib64/tls/libc.so.6
#2  0x00002aaaaba465a2 in __assert_fail () from /lib64/tls/libc.so.6
#3  0x00000000008e239f in alloc_root (mem_root=0xd39020, Size=168) at my_alloc.c:173
#4  0x000000000056fce3 in sql_alloc (Size=168) at thr_malloc.cc:42
#5  0x000000000050b6d8 in Sql_alloc::operator new (size=168) at sql_list.h:29
#6  0x000000000066219a in mysql_table_grant (thd=0x16244b0, table_list=0x1644428, user_list=@0x1624cd8,
    columns=@0x1624cf0, rights=1, revoke_grant=false) at sql_acl.cc:2898
#7  0x00000000005ca69e in mysql_execute_command (thd=0x16244b0) at sql_parse.cc:3851
#8  0x00000000005cf1dd in mysql_parse (thd=0x16244b0,
    inBuf=0x1644340 "grant select on track.User to 'vacation'@'195.117.14.1' IDENTIFIED BY 'dupa.8'", length=78)
    at sql_parse.cc:5628
#9  0x00000000005c4eef in dispatch_command (command=COM_QUERY, thd=0x16244b0,
    packet=0x163c311 "grant select on track.User to 'vacation'@'195.117.14.1' IDENTIFIED BY 'dupa.8'", packet_length=79)
    at sql_parse.cc:1713
#10 0x00000000005c46af in do_command (thd=0x16244b0) at sql_parse.cc:1514
#11 0x00000000005c3800 in handle_one_connection (arg=0x16244b0) at sql_parse.cc:1158
#12 0x00000000005b0551 in create_new_thread (thd=0x16244b0) at mysqld.cc:3791
#13 0x00000000005b0e82 in handle_connections_sockets (arg=0x0) at mysqld.cc:4062
#14 0x00000000005b0027 in main (argc=6, argv=0x7fffffcd3788) at mysqld.cc:3463

(gdb) l
168       gptr point;
169       reg1 USED_MEM *next= 0;
170       reg2 USED_MEM **prev;
171       DBUG_ENTER("alloc_root");
172       DBUG_PRINT("enter",("root: 0x%lx", mem_root));
173       DBUG_ASSERT(alloc_root_inited(mem_root));
174
175       Size= ALIGN_SIZE(Size);
176       if ((*(prev= &mem_root->free)) != NULL)
177       {
(gdb) print mem_root
$1 = (MEM_ROOT *) 0xd39020
(gdb) print *mem_root
$2 = {free = 0x0, used = 0x0, pre_alloc = 0x0, min_malloc = 0, block_size = 0, block_num = 0, first_block_usage = 0,
  error_handler = 0}

(gdb) frame 4
#4  0x000000000056fce3 in sql_alloc (Size=168) at thr_malloc.cc:42
42        char *ptr= (char*) alloc_root(root,Size);
Current language:  auto; currently c++
(gdb) print root
$3 = (MEM_ROOT *) 0xd39020
(gdb) print *root
$4 = {free = 0x0, used = 0x0, pre_alloc = 0x0, min_malloc = 0, block_size = 0, block_num = 0, first_block_usage = 0,
  error_handler = 0}
(gdb) print Size
$5 = 168
(gdb) frame 5
#5  0x000000000050b6d8 in Sql_alloc::operator new (size=168) at sql_list.h:29
29          return (void*) sql_alloc((uint) size);
(gdb) print size
$6 = 168

How to repeat,

I'm going to attach binary for database (tested on i686 and amd64 Linux 2.6), login is mysql, pass mysql, then:
create database track;

CREATE TABLE `User` (
  `id` int(11) NOT NULL auto_increment,
  `login` varchar(20) default NULL,
  `password` varchar(16) NOT NULL default '',
  `firstname` varchar(100) default NULL,
  `lastname` varchar(100) default NULL,
  `email` varchar(100) default NULL,
  `team` int(11) default NULL,
  `position` int(11) default NULL,
  `security` tinyint(4) default NULL,
  `lastlogin` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
  `currentlogin` timestamp NOT NULL default '0000-00-00 00:00:00',
  `alert` char(1) default 'y',
  `employment` varchar(8) NOT NULL default '',
  `office` varchar(8) NOT NULL default '',
  `activeaccount` char(1) default '1',
  `employto` date default NULL,
  `employfrom` date NOT NULL default '2002-07-01',
  `vacation` char(1) default NULL,
  `vacationtext` text,
  `vacationfrom` timestamp NOT NULL default '0000-00-00 00:00:00',
  `vacationto` timestamp NOT NULL default '0000-00-00 00:00:00',
  `vacationforward` int(11) default NULL,
  `mentor` int(11) default NULL,
  `issbu` int(11) default NULL,
  `ispbu` int(11) default NULL,
  `lastuserupdate` date default NULL,
  `validfrom` date default NULL,
  `skype` varchar(100) default NULL,
  `gg` int(11) default NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `login` (`login`)
) ENGINE=MyISAM DEFAULT CHARSET=latin2

and 
grant select on track.User to 'vacation'@'1.1.1.1' IDENTIFIED BY 'xxx.8';
[21 Jan 2006 13:58] Elan Ruusamäe
raw database files which allow to reproduce problem (by arekm)

Attachment: mysql.tar.gz (application/x-tgz, text), 8.19 KiB.

[21 Jan 2006 14:00] Arkadiusz Miskiewicz
Note that the mysql db stucture comes from 4.0 or 4.1 and was just dumped and restored without running mysql_fix_privilege_tables. Seems that updating db structure with mysql_fix_privilege_tables fixes the problem.
[21 Jan 2006 14:10] Elan Ruusamäe
diffs between before and after fix_privileges_table script run, which in fact also fixed crash for me

Attachment: bug-16470.tar.bz2 (application/x-tbz, text), 3.91 KiB.

[15 May 2006 13:08] Martin Devera
Just for completness. I migrated from 4.1 to 5.0 and hit the same bug with 5.0.20. mysql_upgrade fixed it.
[31 Jan 2007 6:35] Zivago Lee
Same behavior on a fresh install of Mysql 5.0.22 from centosplus repo on CentOS 4.4.  Had to run mysql_upgrade before running the GRANT command.
[24 Oct 2007 8:27] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/36237

ChangeSet@1.2537, 2007-10-24 10:29:24+02:00, thek@adventure.(none) +1 -0
  Bug #16470 crash on grant if old grant tables
  
  If a user upgraded the server without running mysql_upgrade, and later tried
  to run a GRANT command on grant tables like tables_priv, the server would
  crash.
  
  This patch fixes this problem by checking if the grant tables were properly
  initialized before attempt to store any new grants.
[22 Nov 2007 14:11] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/38280

ChangeSet@1.2580, 2007-11-22 15:14:47+01:00, thek@adventure.(none) +4 -0
  Bug#16470 crash on grant if old grant tables
  
  Loading 4.1 into 5.0 or 5.1 failed silently because procs_priv table missing.
  This caused the server to crash on any attempt to store new grants because
  of uninitialized structures.
  
  This patch breaks up the grant loading function into two phases to allow
  for procs_priv table to fail with an warning instead of crashing the server.
[26 Nov 2007 18:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/38547

ChangeSet@1.2580, 2007-11-26 19:09:40+01:00, thek@adventure.(none) +4 -0
  Bug#16470 crash on grant if old grant tables
  
  Loading 4.1 into 5.0 or 5.1 failed silently because procs_priv table missing.
  This caused the server to crash on any attempt to store new grants because
  of uninitialized structures.
  
  This patch breaks up the grant loading function into two phases to allow
  for procs_priv table to fail with an warning instead of crashing the server.
[6 Dec 2007 10:00] Bugs System
Pushed into 5.1.23-rc
[6 Dec 2007 10:01] Bugs System
Pushed into 6.0.5-alpha
[14 Dec 2007 0:07] Konstantin Osipov
Kristofer, please get in touch with Serg for instructions how to convert
the error produced by FUSH PRIVILEGES when procs_priv is gone to a warning.
I pushed the change that produces the error into the team tree.
[14 Dec 2007 0:08] Konstantin Osipov
(5.1 only)
[6 Mar 2008 14:35] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/43522

ChangeSet@1.2517, 2008-03-06 15:40:54+01:00, thek@adventure.(none) +6 -0
  Bug#16470 crash on grant if old grant tables
  
  Iterative patch to convert an error message to warnings based on 
  originating code (ie error handling strategies).
[18 Jun 2008 10:58] Konstantin Osipov
Pushed into the main trees (please see the version numbers in the progress log of the bug).
[19 Jun 2008 1:42] Paul Dubois
Noted in 5.1.23, 6.0.5 changelogs.

Use of GRANT statements with versions of the grant tables from an old
version of MySQL could cause a server crash.