Bug #50308 Test federated_debug causes server crash
Submitted: 13 Jan 2010 13:43 Modified: 10 Feb 2010 18:44
Reporter: Georgi Kodinov Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Security: Privileges Severity:S3 (Non-critical)
Version:mysql-pe OS:Linux (Fedora 12 x86_64)
Assigned to: Georgi Kodinov CPU Architecture:Any

[13 Jan 2010 13:43] Georgi Kodinov
Description:
When running federated_debug.test I'm getting : 
federated.federated_debug                [ fail ]
        Test ended at 2010-01-13 15:38:28

CURRENT_TEST: federated.federated_debug
mysqltest: At line 22: command "$MYSQLADMIN --no-defaults --default-character-set=latin1 -S $MASTER_MYSOCK -P $MASTER_MYPORT  -u root --password= refresh 2>&1" failed

Output from before failure:
/home/kgeorge/mysql/work/merge-pe/client/.libs/lt-mysqladmin: refresh failed; error: 'Lost connection to MySQL server during query'
exec of '/home/kgeorge/mysql/work/merge-pe/client/mysqladmin --no-defaults --default-character-set=latin1 -S /home/kgeorge/mysql/work/merge-pe/mysql-test/var/tmp/mysqld.1.sock -P 13010  -u root --password= refresh 2>&1' failed, error: 256, status: 1, errno: 0

The result from queries just before the failure was:
< snip >
#
# Bug#47525: MySQL crashed (Federated)
#
# Switch to slave
CREATE TABLE t1(a INT);
INSERT INTO t1 VALUES (1);
# Switch to master
CREATE TABLE t1(a INT) ENGINE=FEDERATED
CONNECTION='mysql://root@127.0.0.1:SLAVE_PORT/test/t1';
SELECT * FROM t1;
a
1
# Start a asynchronous reload

More results from queries before failure can be found in /home/kgeorge/mysql/work/merge-pe/mysql-test/var/log/federated_debug.log

Server [mysqld.1 - pid: 17505, winpid: 17505, exit: 256] failed during test run
Server log from this test:
100113 16:38:28 [Note] Plugin 'InnoDB' is disabled.
100113 16:38:28 [Note] Plugin 'ndbcluster' is disabled.
100113 16:38:28 [Note] Event Scheduler: Loaded 0 events
100113 16:38:28 [Note] /home/kgeorge/mysql/work/merge-pe/sql/mysqld: ready for connections.
Version: '6.0.14-alpha-debug-log'  socket: '/home/kgeorge/mysql/work/merge-pe/mysql-test/var/tmp/mysqld.1.sock'  port: 13010  Source distribution
100113 16:38:28 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=3
max_threads=151
thread_count=3
connection_count=3
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60733 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
/home/kgeorge/mysql/work/merge-pe/sql/mysqld(my_print_stacktrace+0x35) [0xb5191c]
/home/kgeorge/mysql/work/merge-pe/sql/mysqld(handle_segfault+0x2a7) [0x6e8e8c]
/lib64/libpthread.so.0() [0x39b240efa0]
/home/kgeorge/mysql/work/merge-pe/sql/mysqld(reload_acl_and_cache(THD*, unsigned long, TABLE_LIST*, bool*)+0x5f6) [0x707644]
/home/kgeorge/mysql/work/merge-pe/sql/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x14e9) [0x6f8fd6]
/home/kgeorge/mysql/work/merge-pe/sql/mysqld(do_command(THD*)+0x252) [0x6f78d8]
/home/kgeorge/mysql/work/merge-pe/sql/mysqld(handle_one_connection+0x14c) [0x6f609a]
/lib64/libpthread.so.0() [0x39b2406a3a]
/lib64/libc.so.6(clone+0x6d) [0x39b1cddf3d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

How to repeat:
run the test with a server compiled with BUILD/compile-pentium-debug-max

Suggested fix:
stop it from crashing
[13 Jan 2010 13:43] Georgi Kodinov
The gdb stacktrace looks like this : 
Program terminated with signal 11, Segmentation fault.
#0  0x00000039b240c2ac in pthread_kill () from /lib64/libpthread.so.0
#0  0x00000039b240c2ac in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000b519e4 in my_write_core (sig=11) at stacktrace.c:328
#2  0x00000000006e908c in handle_segfault (sig=11) at mysqld.cc:2764
#3  <signal handler called>
#4  0x0000000000707644 in reload_acl_and_cache (thd=0x0, options=32814, 
    tables=0x0, write_to_binlog=0x7f2b12267c9f) at sql_parse.cc:6999
#5  0x00000000006f8fd6 in dispatch_command (command=COM_REFRESH, 
    thd=0x1b3e3c8, packet=0x1b48f39 ".", packet_length=1) at sql_parse.cc:1264
#6  0x00000000006f78d8 in do_command (thd=0x1b3e3c8) at sql_parse.cc:760
#7  0x00000000006f609a in handle_one_connection (arg=0x1b3e3c8)
    at sql_connect.cc:1164
#8  0x00000039b2406a3a in start_thread () from /lib64/libpthread.so.0
#9  0x00000039b1cddf3d in clone () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 5 (Thread 17569):
#0  0x00000039b240e11d in read () from /lib64/libpthread.so.0
#1  0x0000000000be213b in vio_read (vio=0x1b03738, buf=0x1b3a388 "\001", 
    size=4) at viosocket.c:44
#2  0x00000000006daf1a in my_real_read (net=0x1b2f930, complen=0x7f2b122a8d98)
    at net_serv.cc:833
#3  0x00000000006db72c in my_net_read (net=0x1b2f930) at net_serv.cc:1023
#4  0x00000000006f772a in do_command (thd=0x1b2f818) at sql_parse.cc:706
#5  0x00000000006f609a in handle_one_connection (arg=0x1b2f818)
    at sql_connect.cc:1164
#6  0x00000039b2406a3a in start_thread () from /lib64/libpthread.so.0
#7  0x00000039b1cddf3d in clone () from /lib64/libc.so.6
#8  0x0000000000000000 in ?? ()

Thread 4 (Thread 17568):
#0  0x00000039b240e11d in read () from /lib64/libpthread.so.0
#1  0x0000000000be213b in vio_read (vio=0x1b030c8, buf=0x1b2b7d8 "\001", 
    size=4) at viosocket.c:44
#2  0x00000000006daf1a in my_real_read (net=0x1ae0380, complen=0x7f2b122e9d98)
    at net_serv.cc:833
#3  0x00000000006db72c in my_net_read (net=0x1ae0380) at net_serv.cc:1023
#4  0x00000000006f772a in do_command (thd=0x1ae0268) at sql_parse.cc:706
#5  0x00000000006f609a in handle_one_connection (arg=0x1ae0268)
    at sql_connect.cc:1164
#6  0x00000039b2406a3a in start_thread () from /lib64/libpthread.so.0
#7  0x00000039b1cddf3d in clone () from /lib64/libc.so.6
#8  0x0000000000000000 in ?? ()

Thread 3 (Thread 17509):
#0  0x00000039b240ed28 in sigwait () from /lib64/libpthread.so.0
#1  0x00000000006e97e4 in signal_hand (arg=0x0) at mysqld.cc:2966
#2  0x00000039b2406a3a in start_thread () from /lib64/libpthread.so.0
#3  0x00000039b1cddf3d in clone () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()

Thread 2 (Thread 17506):
#0  0x00000039b1cd6ca3 in select () from /lib64/libc.so.6
#1  0x00000000006ecfc5 in handle_connections_sockets () at mysqld.cc:5383
#2  0x00000000006ec559 in main (argc=10, argv=0x7fff2fe49918) at mysqld.cc:4876

Thread 1 (Thread 17588):
#0  0x00000039b240c2ac in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000b519e4 in my_write_core (sig=11) at stacktrace.c:328
#2  0x00000000006e908c in handle_segfault (sig=11) at mysqld.cc:2764
#3  <signal handler called>
#4  0x0000000000707644 in reload_acl_and_cache (thd=0x0, options=32814, 
    tables=0x0, write_to_binlog=0x7f2b12267c9f) at sql_parse.cc:6999
#5  0x00000000006f8fd6 in dispatch_command (command=COM_REFRESH, 
    thd=0x1b3e3c8, packet=0x1b48f39 ".", packet_length=1) at sql_parse.cc:1264
#6  0x00000000006f78d8 in do_command (thd=0x1b3e3c8) at sql_parse.cc:760
#7  0x00000000006f609a in handle_one_connection (arg=0x1b3e3c8)
    at sql_connect.cc:1164
#8  0x00000039b2406a3a in start_thread () from /lib64/libpthread.so.0
#9  0x00000039b1cddf3d in clone () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()
[13 Jan 2010 14:42] Valeriy Kravchuk
Test passes without any problems on both 5.1.43-bzr and 6.0-codebase trees on Mac OS X.
[13 Jan 2010 15:06] Sveta Smirnova
Thank you for the report.

Verified as described.
[14 Jan 2010 8:53] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/96844

3827 Georgi Kodinov	2010-01-14
      Bug #50308: Test federated_debug causes server crash
      
      In the special debug mode used by federated_debug
      reload_acl_and_cache() is called with a NULL THD pointer.
      And since the control path in this function doesn't 
      expect that the THD pointer may be missing the server 
      crashes on NULL pointer dereference.
      Fixed by correctly checking for the presence of THD
      before dereferencing it.
[14 Jan 2010 15:11] Georgi Kodinov
Pushed to mysql-pe
[5 Feb 2010 11:53] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100205113942-oqovjy0eoqbarn7i) (version source revid:alik@sun.com-20100204064210-ljwanqvrjs83s1gq) (merge vers: 6.0.14-alpha) (pib:16)
[10 Feb 2010 18:44] Paul DuBois
Noted in 6.0.14 changelog.

A NULL pointer was dereferenced in a special debug mode used by a
FEDERATED test case.