Bug #13404 INNODB memory allocation error during startup
Submitted: 22 Sep 2005 14:04 Modified: 22 Sep 2005 20:58
Reporter: Yariv Graf Email Updates:
Status: Not a Bug Impact on me:
Category:MySQL Server Severity:S1 (Critical)
Version:4.0.22 ICC OS:Cent OS 2.6.12-5
Assigned to: CPU Architecture:Any

[22 Sep 2005 14:04] Yariv Graf
Hello kind sirs,

I have a DL380 G4 box, Dual Xeon 3.0Ghz with 6GB of DDR2 Ram,
I'm Linux CentOS 4 with a vanilla kernel, and a remote storage 
which I access via iSCSI (linux-iscsi-4.0.2 complied as a module) 
connected with two Gigabit interfaces. the 'balancing' between the two 
links is maintained by multipathd (mutlipath-tools 0.4.4).

The situation is like this:

I had a DL380 G4 box with 4GB of DDR2 Ram that was running RedHat 
ES3-Update4 with stock redhat kernel and mysql 4.0.22 Intel-ICC 
precomplied binaries (downloaded from mysql.com - 
mysql-standard-4.0.22-pc-linux-gnu-i686-icc-glibc23.tar.gz ) the storage 
  was a local Raid5 of 4 10K RPM 72Gb SCSI disks.

On the old box, everything worked fine, other than IO problems to the 
local raid, so I've moved it to the new box:

I've moved the data 'as-is' to the new iSCSI box, upgraded the MySQL 
binary to 4.0.25 (still, precomplied Intel-ICC binaries), and kept the 
my.cnf as it was on the previous box:

-- my.cnf --

innodb_additional_mem_pool_size = 200M
innodb_buffer_pool_size = 2G
innodb_data_file_path = 
innodb_file_io_threads = 4
innodb_thread_concurrency = 16
innodb_log_buffer_size = 8M
innodb_log_file_size = 500M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90

-- my.cnf --

The problem is that when I start MySQL I get the following error:

-- error log --

050902 16:06:12  mysqld started
InnoDB: Fatal error: cannot allocate 2097168384 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 214666988 bytes. Operating system errno: 12
InnoDB: Cannot continue operation!
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We now intentionally generate a seg fault so that
InnoDB: on Linux we get a stack trace.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help 
the problem, but since we have already crashed, something is definitely 
and this may fail.

It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections 
= 1858782 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Stack range sanity check OK, backtrace follows:
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and 
follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa2a17e7d  is invalid pointer
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
050902 16:06:12  mysqld ended

-- error log --
the dB has 42GB of data,
I've tried playing with 'innodb_additional_mem_pool_size' and 
'innodb_log_file_size' and the only thing that works is when I lower 
'innodb_additional_mem_pool_size' to 1500M, anything higher than that 
doesn't work.

I've checked ulimits on the machine and it looks fine:

-- ulimit -a --
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 52735
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 52735
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

-- ulimit -a --

Also, before I start MySQL I see the box has about 5900M of free memory 
(free -m).

Any help would be greatly appritiated.

Thanks in advanced,

How to repeat:
same configuration. just fire up MySQL
[22 Sep 2005 20:58] Heikki Tuuri
Please check that your Linux installation is configured to allow a process size > 2 GB. 32-bit operating systems typically have a maximum of 2 GB, but certain Linux kernels allow up to 3.5 GB process size.