Description:
Hello kind sirs,
I have a DL380 G4 box, Dual Xeon 3.0Ghz with 6GB of DDR2 Ram,
I'm Linux CentOS 4 with a 2.6.12.5 vanilla kernel, and a remote storage
which I access via iSCSI (linux-iscsi-4.0.2 complied as a module)
connected with two Gigabit interfaces. the 'balancing' between the two
links is maintained by multipathd (mutlipath-tools 0.4.4).
The situation is like this:
I had a DL380 G4 box with 4GB of DDR2 Ram that was running RedHat
ES3-Update4 with stock redhat kernel and mysql 4.0.22 Intel-ICC
precomplied binaries (downloaded from mysql.com -
mysql-standard-4.0.22-pc-linux-gnu-i686-icc-glibc23.tar.gz ) the storage
was a local Raid5 of 4 10K RPM 72Gb SCSI disks.
On the old box, everything worked fine, other than IO problems to the
local raid, so I've moved it to the new box:
I've moved the data 'as-is' to the new iSCSI box, upgraded the MySQL
binary to 4.0.25 (still, precomplied Intel-ICC binaries), and kept the
my.cnf as it was on the previous box:
-- my.cnf --
innodb_additional_mem_pool_size = 200M
innodb_buffer_pool_size = 2G
innodb_data_file_path =
ibdata1:10G;ibdata2:10G;ibdata3:10G;ibdata4:10G;ibdata5:10G;ibdata6:10G
innodb_file_io_threads = 4
innodb_thread_concurrency = 16
innodb_log_buffer_size = 8M
innodb_log_file_size = 500M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
-- my.cnf --
The problem is that when I start MySQL I get the following error:
-- error log --
050902 16:06:12 mysqld started
InnoDB: Fatal error: cannot allocate 2097168384 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 214666988 bytes. Operating system errno: 12
InnoDB: Cannot continue operation!
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We now intentionally generate a seg fault so that
InnoDB: on Linux we get a stack trace.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.
key_buffer_size=16777216
read_buffer_size=1044480
max_used_connections=0
max_connections=200
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
= 1858782 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x83a3008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Stack range sanity check OK, backtrace follows:
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and
follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa2a17e7d is invalid pointer
thd->thread_id=1701522733
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
050902 16:06:12 mysqld ended
-- error log --
the dB has 42GB of data,
I've tried playing with 'innodb_additional_mem_pool_size' and
'innodb_log_file_size' and the only thing that works is when I lower
'innodb_additional_mem_pool_size' to 1500M, anything higher than that
doesn't work.
I've checked ulimits on the machine and it looks fine:
-- ulimit -a --
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 52735
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 52735
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
-- ulimit -a --
Also, before I start MySQL I see the box has about 5900M of free memory
(free -m).
Any help would be greatly appritiated.
Thanks in advanced,
Yariv
How to repeat:
same configuration. just fire up MySQL
