Bug #15869 Cannot shutdown the server - it restarts
Submitted: 19 Dec 2005 21:04 Modified: 22 May 2006 18:37
Reporter: Joerg Behrens Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:4.1.16 OS:Other (IRIX 6.5.28)
Assigned to: Magnus Blåudd CPU Architecture:Any

[19 Dec 2005 21:04] Joerg Behrens
Description:
When trying to shutdown a running mysqld with the init script mysql.server or with 'mysqladmin shutdown' the mysqd just restarts.

uname -Ra
IRIX64 o2k 6.5 6.5.28m 07010238 IP27
$ cc -version
MIPSpro Compilers: Version 7.4.4m

./configure --with-extra-charsets=complex --enable-thread-safe-client --with-unix-socket-path=/usr/nekoware/var/run/l4/mysql.sock   --disable-dependency-tracking  --with-big-tables --without-readline  --with-openssl=/usr/nekoware  --prefix=/usr/nekoware/mysql4_test --wiebug=full

The backtrace

[o2k]:/usr/nekoware/mysql4_test/var $ file core
core: IRIX N32 core dump of 'mysqld'
[o2k]:/usr/nekoware/mysql4_test/var $ dbx ../libexec/mysqld core
dbx version 7.3.7 (96228_Jun17 patchSG0005844) Jun 17 2005 02:44:36
Debugger Server version Jun 17 2005 02:47:22
Core from signal SIGSEGV: Segmentation violation
(dbx) where

Thread 0x1000a
>  0 _prctl(0x15, 0x4, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/irix/lib/libc/libc_n32_M4/proc/prctl.s
":15, 0xfa4ab98]
   1 pthread_kill(0x15, 0xb, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/eoe/lib/libpthread/libpthread_n3
2_M3/sig.c":150, 0xc06ffa8]
   2 write_core(sig = 11) ["/raids/strip2/MIPS/mysql-4.1.16/sql/stacktrace.c":220, 0x10548e78]
   3 ::handle_segfault(sig = 11) ["/raids/strip2/MIPS/mysql-4.1.16/sql/mysqld.cc":1981, 0x10228e1c]
   4 sig_fixup_mask(0x15, 0x4, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/eoe/lib/libpthread/libpthread_
n32_M3/sig.c":480, 0xc070894]
   5 _sigtramp(0xb, 0x4, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/irix/lib/libc/libc_n32_M4/signal/sig
tramp.s":71, 0xfaefcec]
   6 _SGIPT_libc_sigaction(0x0, 0x1388ff48, 0x0, 0x0, 0xc078928, 0xffffffff, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/eoe/lib
/libpthread/libpthread_n32_M3/sig.c":447, 0xc070828]
   7 _sigaction(0x0, 0x1388ff48, 0x80000000, 0x0, 0xc078928, 0xffffffff, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/irix/lib/li
bc/libc_n32_M4/signal/sigaction.c":26, 0xfa4f778]
   8 ::kill_server(void*)(sig_ptr = (nil)) ["/raids/strip2/MIPS/mysql-4.1.16/sql/mysqld.cc":863, 0x10226880]
   9 ::kill_server_thread(arg = 0xf) ["/raids/strip2/MIPS/mysql-4.1.16/sql/mysqld.cc":905, 0x102269d4]
   10 _SGIPT_pt_start() ["/xlv44/6.5.28m/work/eoe/lib/libpthread/libpthread_n32_M3/pt.c":803, 0xc06d31c]
(dbx) where

Thread 0x1000a
>  0 _prctl(0x15, 0x4, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/irix/lib/libc/libc_n32_M4/proc/prctl.s
":15, 0xfa4ab98]
   1 pthread_kill(0x15, 0xb, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/eoe/lib/libpthread/libpthread_n3
2_M3/sig.c":150, 0xc06ffa8]
   2 write_core(sig = 11) ["/raids/strip2/MIPS/mysql-4.1.16/sql/stacktrace.c":220, 0x10548e78]
   3 ::handle_segfault(sig = 11) ["/raids/strip2/MIPS/mysql-4.1.16/sql/mysqld.cc":1981, 0x10228e1c]
   4 sig_fixup_mask(0x15, 0x4, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/eoe/lib/libpthread/libpthread_
n32_M3/sig.c":480, 0xc070894]
   5 _sigtramp(0xb, 0x4, 0xffff, 0x0, 0x0, 0x0, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/irix/lib/libc/libc_n32_M4/signal/sig
tramp.s":71, 0xfaefcec]
   6 _SGIPT_libc_sigaction(0x0, 0x1388ff48, 0x0, 0x0, 0xc078928, 0xffffffff, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/eoe/lib
/libpthread/libpthread_n32_M3/sig.c":447, 0xc070828]
   7 _sigaction(0x0, 0x1388ff48, 0x80000000, 0x0, 0xc078928, 0xffffffff, 0xc080000, 0x2c078924) ["/xlv44/6.5.28m/work/irix/lib/li
bc/libc_n32_M4/signal/sigaction.c":26, 0xfa4f778]
   8 ::kill_server(void*)(sig_ptr = (nil)) ["/raids/strip2/MIPS/mysql-4.1.16/sql/mysqld.cc":863, 0x10226880]
   9 ::kill_server_thread(arg = 0xf) ["/raids/strip2/MIPS/mysql-4.1.16/sql/mysqld.cc":905, 0x102269d4]
   10 _SGIPT_pt_start() ["/xlv44/6.5.28m/work/eoe/lib/libpthread/libpthread_n32_M3/pt.c":803, 0xc06d31c]
(dbx) quit
[o2k]:/usr/nekoware/mysql4_test/var $ dbx
dbx version 7.3.7 (96228_Jun17 patchSG0005844) Jun 17 2005 02:44:36
Debugger Server version Jun 17 2005 02:47:22
(dbx) exit

The <hostname>.err file

cat o2k.err
051219 21:22:36  mysqld started
InnoDB: !!!!!!!!!!!!!! UNIV_DEBUG switched on !!!!!!!!!!!!!!!
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
051219 21:22:37  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
051219 21:22:38  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
051219 21:22:38  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
051219 21:23:28  InnoDB: Started; log sequence number 0 0
/usr/nekoware/mysql4_test/libexec/mysqld: ready for connections.
Version: '4.1.16-debug'  socket: '/usr/nekoware/var/run/mysql4/mysql.sock'  port: 3300  Source distribution
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=258048
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 92780 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Writing a core file
051219 21:23:29  mysqld restarted
InnoDB: !!!!!!!!!!!!!! UNIV_DEBUG switched on !!!!!!!!!!!!!!!
051219 21:23:30  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
051219 21:23:30  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 36808.
InnoDB: Doing recovery: scanned up to log sequence number 0 43634
051219 21:23:30  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
051219 21:23:30  InnoDB: Flushing modified pages from the buffer pool...
051219 21:23:33  InnoDB: Started; log sequence number 0 43634
/usr/nekoware/mysql4_test/libexec/mysqld: ready for connections.
Version: '4.1.16-debug'  socket: '/usr/nekoware/var/run/mysql4/mysql.sock'  port: 3300  Source distribution

regards
Joerg Behrens

How to repeat:
'mysqladmin shutdown'
[19 Dec 2005 21:06] Joerg Behrens
The correct configure was
./configure --with-extra-charsets=complex --enable-thread-safe-client --with-unix-socket-path=/usr/nekoware/var/run/mysql4/mysql.sock   --disable-dependency-tracking  --with-big-tables --without-readline  --with-openssl=/usr/nekoware  --prefix=/usr/nekoware/mysql4_test --with-debug=full
[20 Dec 2005 16:36] Valeriy Kravchuk
Thank you for a bug report. Verified on our octane2 server.

1. Build 4.1.16 with the following commands:

export PATH=/usr/people/mysqldev/octane2-64bit/bin:/usr/freeware/bin:/usr/sbin:/usr/bsd:/sbin:/usr/bin:/usr/bin/X11

./configure --prefix=/users/vkravchuk/dbs/4.1-octane --with-extra-chars
ets=complex --enable-thread-safe-client --disable-dependency-tracking --with-bin-tables --with-readline --with-debug=full 'CC=cc' 'CFLAGS=-O3 -c99 -OPT:Olimit=0 -64' 'CXXFLAGS=-O3  -c99 -OPT:Olimit=0 -LANG:exceptions=OFF -LANG:std=ON -LANG:
libc_in_namespace_std=OFF -64' 'CXX=CC' 'LDFLAGS=-64'

(note: on our octane2 it does not want to build successfully --without-readline)

make
make install
cd ~dbs/4.1-octane/
bin/mysql_install_db

2. Then start the server, connect, exit and then try to stop it with mysqladmin:

-bash-2.05b$ bin/mysqld_safe &
[1] 32331406
-bash-2.05b$ Starting mysqld daemon with databases from /users/vkravchuk/dbs/4.1-octane/var

-bash-2.05b$ bin/mysql -uroot mysql

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 4.1.16-debug

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| func                      |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| host                      |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
15 rows in set (0.00 sec)

mysql> exit
Bye
-bash-2.05b$ bin/mysqladmin -uroot shutdown
051220 17:23:14  mysqld restarted

In the error log we get:

-bash-2.05b$ cat var/octane2.err
051220 17:22:04  mysqld started
InnoDB: !!!!!!!!!!!!!! UNIV_DEBUG switched on !!!!!!!!!!!!!!!
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
051220 17:22:04  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
051220 17:22:05  InnoDB: Log file ./ib_logfile0 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
051220 17:22:06  InnoDB: Log file ./ib_logfile1 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
051220 17:22:47  InnoDB: Started; log sequence number 0 0
/users/vkravchuk/dbs/4.1-octane/libexec/mysqld: ready for connections.
Version: '4.1.16-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
mysqld got signal 10;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388572
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225788 Kbytes of memory
Hope that's ok; if not, decrease some variables in the equation.

051220 17:23:14  mysqld restarted
InnoDB: !!!!!!!!!!!!!! UNIV_DEBUG switched on !!!!!!!!!!!!!!!
051220 17:23:15  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
051220 17:23:15  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 43634.
InnoDB: Doing recovery: scanned up to log sequence number 0 43634
051220 17:23:15  InnoDB: Flushing modified pages from the buffer pool...
051220 17:23:18  InnoDB: Started; log sequence number 0 43634
/users/vkravchuk/dbs/4.1-octane/libexec/mysqld: ready for connections.
Version: '4.1.16-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribut
ion

-bash-2.05b$ uname -a
IRIX64 octane2 6.5 07080050 IP30

So, there is a problem, surely.
[25 Dec 2005 19:06] Joerg Behrens
I have made a step back and check the 4.1.15 release. This version runs fine without the shutdown problem.

[o2k]:/usr/nekoware-build/mysql4 $ bin/mysqld_safe &
[1] 3366063
[o2k]:/usr/nekoware-build/mysql4 $ Starting mysqld daemon with databases from /usr/nekoware-build/mysql4/var

[o2k]:/usr/nekoware-build/mysql4 $ bin/mysql -u root -p
Enter password:
mysql> select version();
+-----------+
| version() |
+-----------+
| 4.1.15    |
+-----------+
1 row in set (0.01 sec)

mysql> exit
Bye
[o2k]:/usr/nekoware-build/mysql4 $ bin/mysqladmin -u root shutdown
STOPPING server from pid file /usr/nekoware-build/mysql4/var/o2k.pid
051225 19:30:27  mysqld ended

[1]+  Done                    bin/mysqld_safe

051225 19:30:24  InnoDB: Starting shutdown...
051225 19:30:27  InnoDB: Shutdown completed; log sequence number 0 43634
051225 19:30:27 [Note] /usr/nekoware-build/mysql4/libexec/mysqld: Shutdown complete

051225 19:30:27  mysqld ended

So the problem comes in after 4.1.15.

regards
Joerg
[22 Apr 2006 20:32] Joerg Behrens
Today i compiled the current 5.0.20a version and it shows the same problem.

After starting the server it cant be shutdown

[o2k]:/usr2/MIPS/mysql-5.0.20a $  /usr/nekoware-build/mysql5/bin/mysqld_safe &
[1] 5202990
[o2k]:/usr2/MIPS/mysql-5.0.20a $ Starting mysqld daemon with databases from /usr/nekoware-build/mysql5/var

[o2k]:/usr2/MIPS/mysql-5.0.20a $ /usr/nekoware-build/mysql5/bin/mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2 to server version: 5.0.20a

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>  select version();
+-----------+
| version() |
+-----------+
| 5.0.20a   |
+-----------+
1 row in set (0.00 sec)

[o2k]:/usr2/MIPS/mysql-5.0.20a $ /usr/nekoware-build/mysql5/bin/mysqladmin shutdown
060422 22:24:55  mysqld restarted
[o2k]:/usr2/MIPS/mysql-5.0.20a $ tail -f /usr/nekoware-build/mysql5/var/o2k.err
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
060422 22:24:55  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 43655.
InnoDB: Doing recovery: scanned up to log sequence number 0 43655
060422 22:24:55  InnoDB: Started; log sequence number 0 43655
060422 22:24:55 [Note] /usr/nekoware-build/mysql5/libexec/mysqld: ready for connections.
Version: '5.0.20a'  socket: '/usr/nekoware/var/run/mysql5/mysql.sock'  port: 3300  Source distribution

regards
Joerg
[27 Apr 2006 7:03] Magnus Blåudd
Compiled and tested at our build host nocona.
octane2> uname -a
IRIX64 octane2 6.5 07080050 IP30

Both with the configure arguments provided here as well as the ones we used for building mysql-4.1.18. The mysqld will crash in "select()" so I haven't managed to reproduce the problem described. 
dbx version 7.3.5 (92898_Aug07 MR) Aug  7 2003 14:15:31
Elf 32 Program header 5 in core file does not match program header 5 in object /nfstmp1/bug15869/mysql-4.1.18/mysql-test/../sql/mysqld, vmap index 26.(vmaps described in <core.out.h>) (use of the core file may be misleading!)
Program Header p_p_memsz 0x474fa8, core p_memsz 0x474ea8
Program Header p_p_filesz 0x474fa8, core p_filesz 0x474ea8
Program Header p_p_filesz 0x474fa8, core p_filesz 0x474ea8
Elf 32 Program header 5 in core file does not match program header 5 in object /nfstmp1/bug15869/mysql-4.1.18/mysql-test/../sql/mysqld, vmap index 26.(vmaps described in <core.out.h>) (use of the core file may be misleading!)
Program Header p_p_memsz 0x474fa8, core p_memsz 0x474ea8
Program Header p_p_filesz 0x474fa8, core p_filesz 0x474ea8
Program Header p_p_filesz 0x474fa8, core p_filesz 0x474ea8
Core from signal SIGSEGV: Segmentation violation
(dbx) where

Thread 0x10000
>  0 __select(0x5, 0x7fff2d20, 0x0, 0x0, 0x0, 0x75, 0x61, 0x1077d028) ["/xlv46/6.5.25m/work/irix/lib/libc/libc_n32_M4/sys/select.s":17, 0xfa44ba8]
   1 _select(0x5, 0x7fff2d20, 0x0, 0x0, 0x0, 0x75, 0x61, 0x1077d028) ["/xlv46/6.5.25m/work/irix/lib/libc/libc_n32_M4/sys/selectSCI.c":30, 0xfa44c7c]
   2 ::handle_connections_sockets(0x4, 0x7fff2d20, 0x0, 0x0, 0x0, 0x75, 0x61, 0x1077d028) ["/nfstmp1/bug15869/mysql-4.1.18/sql/mysqld.cc":3666, 0x10187c70]
   3 ::main(0xfa4a984, 0xfb4fc50, 0x0, 0x0, 0x0, 0x75, 0x61, 0x1077d028) ["/nfstmp1/bug15869/mysql-4.1.18/sql/mysqld.cc":3237, 0x10187440]
   4 __start() ["/xlv55/kudzu-apr12/work/irix/lib/libc/libc_n32_M4/csu/crt1text.s":177, 0x1007c5a8]
(dbx)

I have looked at our "Build status" page for 4.1.18 and it looks like it's built and tested fine with the following configure:
CC=cc CXX=CC CFLAGS="-O3 -c99 -OPT:Olimit=0" CXXFLAGS="-O3  -c99 -OPT:Olimit=0 -LANG:exceptions=OFF -LANG:std=ON -LANG:libc_in_namespace_std=OFF"  ./configure --prefix=/usr/local/mysql --localstatedir=/usr/local/mysql/data --libexecdir=/usr/local/mysql/bin --with-comment="MySQL Community Edition - Standard (GPL)" --with-extra-charsets=complex --with-server-suffix="-standard" --enable-thread-safe-client --enable-local-infile  --disable-shared --with-zlib-dir=bundled --with-readline --with-embedded-server --with-archive-storage-engine --with-innodb 

There is also a page in our manual that describes building on Irix - http://dev.mysql.com/doc/refman/4.1/en/sgi-irix.html

Will dig some more.
[27 Apr 2006 7:08] Magnus Blåudd
I mean build host "octane2" :)
[10 May 2006 12:01] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/6188
[18 May 2006 16:45] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/6574
[19 May 2006 11:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/6621
[22 May 2006 8:37] Magnus Blåudd
Pushed to 4.1.20
[22 May 2006 8:38] Magnus Blåudd
Pushed to 5.0.22
[22 May 2006 8:39] Magnus Blåudd
Pushed to 5.1.11 a patch that refuses to set a signal handler for signal 0 as that is not a valid signal number
[22 May 2006 18:37] Paul DuBois
Noted in 4.1.20, 5.0.22, 5.1.11 changelogs.

The server no longer uses a signal handler for signal 0
because it could cause a crash on some platforms.