| Bug #86298 | Temporary sever hang on GRANT | ||
|---|---|---|---|
| Submitted: | 12 May 2017 8:32 | Modified: | 12 Jun 2017 16:39 |
| Reporter: | Roel Van de Paar | Email Updates: | |
| Status: | No Feedback | Impact on me: | |
| Category: | MySQL Server: Security: Privileges | Severity: | S1 (Critical) |
| Version: | 8.0.1 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
[12 May 2017 9:45]
Roel Van de Paar
This testcase can also produce this crash;
Core was generated by `/sda/MS130916-mysql-8.0.0-dmr-linux-x86_64-debug/bin/mysqld --no-defaults --max'.
Program terminated with signal 6, Aborted.
#0 0x00007f907b5d8741 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61 val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0 0x00007f907b5d8741 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x00000000023fc1b9 in my_write_core (sig=6) at /git/MS8.0_dbg/mysys/stacktrace.cc:275
#2 0x0000000001b517ed in handle_fatal_signal (sig=6) at /git/MS8.0_dbg/sql/signal_handler.cc:219
#3 <signal handler called>
#4 0x00007f907996c1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5 0x00007f907996d8c8 in __GI_abort () at abort.c:90
#6 0x00007f9079965146 in __assert_fail_base (fmt=0x7f9079ab63a8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x2feac60 "table->file->ht->db_type == DB_TYPE_NDBCLUSTER || error != 146",
file=file@entry=0x2fea808 "/git/MS8.0_dbg/sql/auth/sql_user_table.cc", line=line@entry=2406,
function=function@entry=0x2febb80 <handle_grant_table(THD*, TABLE_LIST*, ACL_TABLES, bool, st_lex_user*, st_lex_user*)::__PRETTY_FUNCTION__> "int handle_grant_table(THD*, TABLE_LIST*, ACL_TABLES, bool, LEX_USER*, LEX_USER*)") at assert.c:92
#7 0x00007f90799651f2 in __GI___assert_fail (assertion=0x2feac60 "table->file->ht->db_type == DB_TYPE_NDBCLUSTER || error != 146",
file=0x2fea808 "/git/MS8.0_dbg/sql/auth/sql_user_table.cc", line=2406,
function=0x2febb80 <handle_grant_table(THD*, TABLE_LIST*, ACL_TABLES, bool, st_lex_user*, st_lex_user*)::__PRETTY_FUNCTION__> "int handle_grant_table(THD*, TABLE_LIST*, ACL_TABLES, bool, LEX_USER*, LEX_USER*)") at assert.c:101
#8 0x0000000001c0a4ea in handle_grant_table (thd=0x7f8fd8019a40, tables=0x7f906c415720, table_no=TABLE_DB, drop=true,
user_from=0x7f8fd90bc778, user_to=0x0) at /git/MS8.0_dbg/sql/auth/sql_user_table.cc:2405
#9 0x0000000001c0d4ac in handle_grant_data (thd=0x7f8fd8019a40, tables=0x7f906c415720, drop=true, user_from=0x7f8fd90bc778,
user_to=0x0) at /git/MS8.0_dbg/sql/auth/sql_user.cc:1096
#10 0x0000000001c0e2ac in mysql_drop_user (thd=0x7f8fd8019a40, list=..., if_exists=false) at /git/MS8.0_dbg/sql/auth/sql_user.cc:1434
#11 0x0000000001871a31 in mysql_execute_command (thd=0x7f8fd8019a40, first_level=true) at /git/MS8.0_dbg/sql/sql_parse.cc:3697
#12 0x0000000001875c14 in mysql_parse (thd=0x7f8fd8019a40, parser_state=0x7f906c419520) at /git/MS8.0_dbg/sql/sql_parse.cc:5233
#13 0x000000000186c3b8 in dispatch_command (thd=0x7f8fd8019a40, com_data=0x7f906c419cb0, command=COM_QUERY)
at /git/MS8.0_dbg/sql/sql_parse.cc:1481
#14 0x000000000186b244 in do_command (thd=0x7f8fd8019a40) at /git/MS8.0_dbg/sql/sql_parse.cc:1043
#15 0x0000000001b43af1 in handle_connection (arg=0x5f5d430) at /git/MS8.0_dbg/sql/conn_handler/connection_handler_per_thread.cc:301
#16 0x000000000242bf29 in pfs_spawn_thread (arg=0x5d3c640) at /git/MS8.0_dbg/storage/perfschema/pfs.cc:2282
#17 0x00007f907b5d3dc5 in start_thread (arg=0x7f906c41a700) at pthread_create.c:308
#18 0x00007f9079a2e73d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
mysqld: /git/MS8.0_dbg/sql/auth/sql_user_table.cc:2406: int handle_grant_table(THD*, TABLE_LIST*, ACL_TABLES, bool, LEX_USER*, LEX_USER*): Assertion `table->file->ht->db_type == DB_TYPE_NDBCLUSTER || error != 146' failed.
14:55:48 UTC - mysqld got signal 6 ;
[12 May 2017 9:52]
Roel Van de Paar
Small correction. The stack mentioned above was from the original trial where we saw this issue. It was on this query it actually crashed; DROP USER 'Tanjotuser2'@'127.0.0.1'. Then, reducer was able to reduce the testcase to the one shown above (i.e. producing the same crash). However, when executing the testcase at the CLI, it produces the semi/temporary hang. It is possible and likely that given the right circumstances (pquery binary executed, SOURCE'd or similar, or perhaps something more obscure like machine loading etc.) this testcase produces the same crash (it does so inside reducer, but 8.0 handling is still quite limited in scripts). I hope the above is plenty to find the bug(s) here.
[12 May 2017 16:39]
MySQL Verification Team
Hi Roel, I am starting to work on reproducing this bug ..... After your last note , I do have to ask you something. Do I understand you correctly or the final test case now looks like this: DROP DATABASE test; SET @@session.pseudo_slave_mode=ON; BINLOG ' SOgWTg0BAAAAbgAAAHIAAAAAAAQANS0LjMtbTUtZGVidWctbG0nAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAABI0BZOEzgNAAgAEgAEBAQEEgAAVgAEGggAAAAICAgCAAAAAAVAYI0='; xa start''; UPDATE mysql.user SET authentication_string='$0$asd'; xa end''; XA PREPARE''; grant usage,execute on test.* to _0@localhost; DROP USER 'Tanjotuser2'@'127.0.0.1' Also, since you have dropped user 'Tanjotuser2', how did you create it ??? What was its CREATE USER and GRANT commands like ???? Next, you use XA. Is that crucial in reproducing the crash ??? If yes then why do you do PREPARE after END ??? Last, but not least, is that BINLOG command necessary and why ??? What does it contain , in the plain SQL ???? Thanks in advance for your replies ...
[12 May 2017 17:45]
MySQL Verification Team
Hi Roel, Couple of more questions ... First , you have this command: UPDATE mysql.user SET authentication_string='$0$asd'; This is direct DML on privileges tables, which is not allowed. Furthermore, it changes authentication string for all users, without flushing privileges. Doing this would also ruin my installation, but I will try it if you explain to me what is the purpose of this exercise .... Next, you have this command: grant usage,execute on test.* to _0@localhost; which is a strange user name. We do not see how is that user created. Last, but not least, what has it all to do with the user whose DROPing crashed the server ???
[13 Jun 2017 1:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[14 Jun 2018 6:28]
MySQL Verification Team
the hang is innodb_lock_wait_timeout because the mysql.user table is now innodb and the grant statement tries to refresh acl's i guess it conflicts somehow with xa ongoing.
Description: On the GRANT in the below testcase, the server hangs for about 52 seconds. It is instantly repeatable by re-executing the GRANT. The commands before it are fast to complete. Other commands after are fast to complete also; mysql> grant usage,execute on test.* to _0@localhost; ........52 second delay & open line shown........ ERROR 1133 (42000): Can't find any matching row in the user table mysql> CREATE DATABASE test2; USE test2; CREATE TABLE t1 (id int); Query OK, 1 row affected (0.04 sec) Database changed Query OK, 0 rows affected (0.10 sec) How to repeat: DROP DATABASE test; SET @@session.pseudo_slave_mode=ON; BINLOG ' SOgWTg0BAAAAbgAAAHIAAAAAAAQANS0LjMtbTUtZGVidWctbG0nAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAABI0BZOEzgNAAgAEgAEBAQEEgAAVgAEGggAAAAICAgCAAAAAAVAYI0='; xa start''; UPDATE mysql.user SET authentication_string='$0$asd'; xa end''; XA PREPARE''; grant usage,execute on test.* to _0@localhost;