MySQL Bugs: #8834: MySQL server dies with a signal 11 error

Bug #8834	MySQL server dies with a signal 11 error
Submitted:	27 Feb 2005 20:44	Modified:	7 Jun 2005 14:18
Reporter:	Igal Rubinstein	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	4.1.9	OS:	Linux (RedHat Enterprise Linux 3)
Assigned to:		CPU Architecture:	Any

Description:
Hey,

Maybe, as a result of an attack, the number of threads_connected climbed suddenly to 1206, max_used_connections jumped to 1207 and the server died with the following message:

"mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=2093056
max_used_connections=1207
max_connections=1500
threads_connected=1206
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2336900 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

You seem to be running 32-bit Linux and have 1206 concurrent connections.
If you have not changed STACK_SIZE in LinuxThreads and built the binary
yourself, LinuxThreads is quite likely to steal a part of the global heap for
the thread stack. Please read http://www.mysql.com/doc/en/Linux.html

thd=0x97013d08
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x97dff628, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80b2cf1
0x82ca188
0x82ba2c4
0x80acaaf
0x80c3b30
0x82c5a81
0x82f7fca
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil) is invalid pointer
thd->thread_id=2103286
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
pure virtual method called
pure virtual method called
Fatal signal 6 while backtracing"

I tried to execute the following:

[root@sql root]# resolve_stack_dump -s ./mysqld.sym -n ./sql.err
0x80b2cf1 handle_segfault + 397
0x82ca188 __pthread_sighandler + 116
0x82ba2c4 my_vsnprintf + 48
0x80acaaf _Z10net_printfP3THDjz + 139
0x80c3b30 handle_one_connection + 220
0x82c5a81 pthread_start_thread + 193
0x82f7fca __clone + 106
[root@sql root]#

I use RedHat Enterprise Solution Linux 3 & MySQL 4.1.9
The hardware is: CPU: 4x2.7Ghz & 8Gig of RAM.
Kernel info: 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:59:02 EST 2004 i686 i686 i386 GNU/Linux

Thanks!
Igal.

How to repeat:
---

Suggested fix:
---

Hi

Would it be possible for you to provide us with a test case that reproduces this behaviour ?

Thanks a lot

Thank you for your answer.

Actually, it seems to happen when the system receives many new connections
in a very short period of time. My assumption is that the crash happens when 
the number of threads (and connections) climbs beyond 600-700.

Igal.

P.S As you can see from the above text there is no specific query that causes
such behavior.

P.S As you can see from the above text there is no specific query that causes
such behavior.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

reopened

Could you be possibly just suffering from the following known problem that is documented on http://dev.mysql.com/doc/mysql/en/innodb-configuration.html ?

  Warning: On 32-bit GNU/Linux x86, you must be careful not to set memory usage
  too high. glibc may allow the process heap to grow over thread stacks, which 
  crashes your server.  It is a risk if the value of the following expression is close 
  to or exceeds 2GB: [...]

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".