Bug #20851 mysqld segfaults when can not init innodb with fs is full
Submitted: 4 Jul 2006 17:56 Modified: 23 Aug 2006 11:18
Reporter: Andrei Elkin Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.1.12 OS:Linux (Ubuntu 5.10)
Assigned to: CPU Architecture:Any

[4 Jul 2006 17:56] Andrei Elkin
Description:
060704 20:51:08  InnoDB: Error: Write to file ./ibdata1 failed at offset 0 52428
060704 20:51:08  InnoDB: Error: Write to file ./ibdata1 failed at offset 0 5242880.
InnoDB: 1048576 bytes should have been written, only 167936 were written.
InnoDB: Operating system error number 22.

060704 20:51:08 [ERROR] Plugin 'InnoDB' init function returned error.
060704 20:51:08 [ERROR] Failed to init plugins.
060704 20:51:08 [ERROR] Aborting

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210304832 (LWP 22026)]
0x0833ff16 in binlog_func_list (thd=0x0, plugin=0x89bcea0, arg=0xbffa724c)
    at handler.cc:2671
2671      if (hton->state == SHOW_OPTION_YES && hton->binlog_func)
(gdb) bt
#0  0x0833ff16 in binlog_func_list (thd=0x0, plugin=0x89bcea0, arg=0xbffa724c)
    at handler.cc:2671
#1  0x08428e32 in plugin_foreach (thd=0x0, func=0x833fefe <binlog_func_list>, 
    type=1, arg=0xbffa724c) at sql_plugin.cc:951
#2  0x0833ff9b in binlog_func_foreach (thd=0x0, bfn=0xbffa7370)
    at handler.cc:2691
#3  0x08340079 in ha_binlog_end (thd=0x0) at handler.cc:2721
#4  0x0824897e in clean_up (print_message=true) at mysqld.cc:1184
#5  0x08248dd4 in unireg_abort (exit_code=1) at mysqld.cc:1162
#6  0x0824e8e7 in main (argc=33, argv=0xbffa74b4) at mysqld.cc:3579

How to repeat:
make full your mysqld data dir, start the server.
[23 Jul 2006 11:18] Valeriy Kravchuk
More detailed steps on how to repeat are needed. I tried to repeat in the following way:

openxs@suse:~/dbs/5.1> df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              7871560   7851572     19988 100% /
tmpfs                   128148         0    128148   0% /dev/shm
openxs@suse:~/dbs/5.1> bin/mysqld_safe &
[1] 32480
openxs@suse:~/dbs/5.1> Starting mysqld daemon with databases from /home/openxs/d
bs/5.1/var
STOPPING server from pid file /home/openxs/dbs/5.1/var/suse.pid
060723 11:46:41  mysqld ended
tee: /home/openxs/dbs/5.1/var/suse.err: No space left on device

tee: /home/openxs/dbs/5.1/var/suse.err: No space left on device

[1]+  Exit 1                  bin/mysqld_safe
openxs@suse:~/dbs/5.1> tail var/suse.err
060723 11:45:51 [Note] SCHEDULER: Waiting for worker threads to finish
060723 11:45:51 [Note] SCHEDULER: Emptying the queue
060723 11:45:51 [Note] SCHEDULER: Stopped
060723 11:45:51  InnoDB: Starting shutdown...
060723 11:45:53  InnoDB: Shutdown completed; log sequence number 0 46403
060723 11:45:53 [Note] /home/openxs/dbs/5.1/libexec/mysqld: Shutdown complete

060723 11:45:53  mysqld ended

060723 11:46:40  mysqld started
openxs@suse:~/dbs/5.1> ls -l var
total 19998
drwx------  2 openxs users      168 2006-07-23 11:42 cluster
-rw-rw----  1 openxs users 10485760 2006-07-23 11:46 ibdata1
-rw-rw----  1 openxs users  5242880 2006-07-23 11:46 ib_logfile0
-rw-rw----  1 openxs users  4718592 2006-07-23 11:46 ib_logfile1
drwx------  2 openxs users     2136 2006-07-23 11:42 mysql
-rw-rw----  1 openxs users     4488 2006-07-23 11:46 suse.err
drwx------  2 openxs users       48 2006-07-23 11:42 test

So, if there is not enough space to create log files, there is no segfault. Same when there is no space to create ibdata1 completely. 

Segfault happens only when you have incomplete (!) ibdata1 file, and then free some space and try to restart the server:

openxs@suse:~/dbs/5.1> cd /tmp
openxs@suse:/tmp> rm big*.txt
openxs@suse:/tmp> df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              7871560   7563256    308304  97% /
tmpfs                   128148         0    128148   0% /dev/shm
openxs@suse:/tmp> cd -
/home/openxs/dbs/5.1
openxs@suse:~/dbs/5.1> bin/mysqld_safe &
[1] 32586
openxs@suse:~/dbs/5.1> Starting mysqld daemon with databases from /home/openxs/d
bs/5.1/var
STOPPING server from pid file /home/openxs/dbs/5.1/var/suse.pid
060723 11:48:45  mysqld ended

[1]+  Done                    bin/mysqld_safe
openxs@suse:~/dbs/5.1> tail -50 var/suse.err
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
060723 11:48:45 [ERROR] Plugin 'InnoDB' init function returned error.
060723 11:48:45 [ERROR] Failed to init plugins.
060723 11:48:45 [ERROR] Aborting

mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 217596
 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbffff0a4, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x821679a
0xffffe420
0x82fe1ca
0x82fe1ca
0x82fe2c9
0x821509d
0x8214fd0
0x8218d12
0x40199e80
0x8166bc1
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
060723 11:48:45  mysqld ended

Is it the same situation you encountered?
[23 Aug 2006 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".