MySQL Bugs: #12225: MySQL crash on Linux embedded-server [mysqld got signal 11; ]

Bug #12225	MySQL crash on Linux embedded-server [mysqld got signal 11; ]
Submitted:	27 Jul 2005 20:49	Modified:	5 Sep 2005 22:59
Reporter:	Jimmy Muliawan	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: Embedded Library ( libmysqld )	Severity:	S1 (Critical)
Version:	mysql Ver 11.15 Distrib 3.23.41	OS:	Linux (RH Linux (ker: vmlinuz-2.4.7-10))
Assigned to:		CPU Architecture:	Any

Description:
MySQL crashed after ~7 days of load test (query, edit) while searching for an entity.

Number of processes running now: 0
050725 10:32:35  mysqld restarted
/usr/libexec/mysqld: ready for connections
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked agaist is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail

key_buffer_size=8388600
record_buffer=131072
sort_buffer=2097144
max_used_connections=0
max_connections=100
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (record_buffer + sort_buffer)*max_connections = 225791 K
bytes of memory
Hope that's ok, if not, decrease some variables in the equation

Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Stack range sanity check OK, backtrace follows:
0x80b25bf
0x4002a567
0x4017038b
0x80cd0a7
0x81c2e8d
0x80cdc64
0x80caff9
0x80ca415
0x80cb24e
0x80cb4f5
0x80b867b
0x80bb9a5
0x80b7ac3
0x80b7099
Stack trace seems successful - bottom reached
Please read http://www.mysql.com/doc/U/s/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x826af80 = SELECT keyinfo_id FROM KEYINFO WHERE owner_tag=15255305 AND entity_tag=15255305 
thd->thread_id=1

Successfully dumped variables, if you ran with --log, take a look at the
details of what thread 1 did to cause the crash.  In some cases of really
bad corruption, the values shown above may be invalid

The manual page at http://www.mysql.com/doc/C/r/Crashing.html contains
information that should help you find out what is causing the crash
----------------------------------

Here's some info on the backtrace from the symbol file:
0x80b25bf flush_thread_cache__Fv + 559
0x4002a567 _end + 937289887
0x4017038b _end + 938624707
0x80cd0a7 setup_ftfuncs__FP3THDP13st_table_listRt4List1Z15Item_func_match + 247
0x81c2e8d hash_insert + 509
0x80cdc64 openfrm__FPCcT0UiUiUiP8st_table + 2980
0x80caff9 abort_locked_tables__FP3THDPCcT1 + 185
0x80ca415 open_table__FP3THDPCcN21Pb + 869
0x80cb24e open_tables__FP3THDP13st_table_list + 94
0x80cb4f5 open_and_lock_tables__FP3THDP13st_table_list + 21
0x80b867b mysql_execute_command__Fv + 667
0x80bb9a5 mysql_parse__FP3THDPcUi + 69
0x80b7ac3 do_command__FP3THD + 1187
0x80b7099 handle_one_connection__FPv + 617

How to repeat:
Put MySQL under load for a long period of time and try to search for an entity.

We need a repeatable test case. If we create it on our own we can't get
the behavior reported.

Well, we reproduced it on one of our embedded voice mail product (in Linux). Basically, we use mysql as the database for the voicemail and some web interfaces. The load test procedure involves leaving and retrieving fax messages via phone interfaces (requires some special hardwares). So, it'll be a little tough to reproduce it without the hardwares.

Does the backtrace info provide any useful information? I have a tarball of /var/lib/mysql, I'll try to attach it somewhere in this bug note.

partial log from mysqld.log

Attachment: mysql-log.txt (text/plain), 10.20 KiB.

Thank you very much for your error log.

This, unfortunately, does not help at all.

What we need is a repeatable test case.

A repeatable test case is a series of SQL commands that always lead to the problem, in your case, a crash.

You can read more about how to make one in our manual.

We're trying to determine the SQL command sequences before mysql crashed. Is there any logging that we can turn on (for mysql) to log each SQL command being issued? Let us know if there's any other useful logging that we can turn on. We will try to reproduce the problem (with loggings turned on).

You can use the general query log, using i.e:

static char *server_args[] = {"--log=path_file_name"}

or to use it in your embedded server group in my.cnf

I checked my.cnf file on the server:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

So, we already have the general err-log in /var/log/mysqld.log (the log that I gave to you before). Is there any more specific log to turn on?

Jimmy, it's completely different log (log for error messages).  Miguel asked for setup the log for queries.

BTW, why the subject of this bug report contains embedded server, if the error log and stack trace refers to mysqld (dedicated server)?

Hi, could you show me the step-by-step instruction on how to turn on this log (log for queries) in my.cnf file?

Let me clarify, it's actually a dedicated server on a voice board (containing small PC section), plugged to one of the slot in a PBX. So it's basically a PC inside a switch box.

Could you please verify if the bug below can apply to your case:

http://bugs.mysql.com/bug.php?id=11871

Thanks in advance.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".