Bug #39229 MySQLd signal 11
Submitted: 3 Sep 2008 21:21 Modified: 8 Oct 2008 10:59
Reporter: Alvin Chang Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.0.67-log OS:Linux (Centos 5.2 x86-64)
Assigned to: CPU Architecture:Any
Tags: crash, stored procedures

[3 Sep 2008 21:21] Alvin Chang
Description:
We've recently tried to upgrade to 5.0.67 from 5.0.19, and we receive a crash every time a certain query or stored procedure is requested from a web page. When this even happened, We tried reverting back to 5.0.51a and that gives us a crash as well, but with a different error. So far I've gone all the way back to 5.0.22 and the problem has disappeared. 

However one of the reasons why we went to 5.0.67 was because we had an issue on 5.0.19 replication, when a stored procedure insert would not appropriately write an insert statement to the binary log, thus breaking replication. Here is the error log we receive:

5.0.67:

----------------- Start ----------------

080902 13:19:02 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.67-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
080902 13:20:52 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=2097152
max_used_connections=17
max_connections=1000
threads_connected=11
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 18694144 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x10acdb20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x10f127c8, backtrace may not be correct.
Bogus stack limit or frame pointer, fp=0x10f127c8, stack_bottom=0x45390000, thread_stack=262144, aborting backtrace.
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x10aab710 = SELECT  COUNT(t.trackDetailID) as 'TotalViews',
              (SELECT  COUNT(tu.trackDetailID)
                FROM iqtrack.postedjobs pj
                JOIN iqtrack.trackurlapplyattempt tu
                ON tu.reqEmediaID=pj.reqEmediaID
                WHERE
                  pj.requisitionID =  NAME_CONST('reqID',1312934)
                AND
                  pj.postedDateTime is not null
                AND
                  tu.createdDateTime BETWEEN  NAME_CONST('dateFrom',_binary'2000-01-01') and  NAME_CONST('dateTo',_binary'2008-09-02')
                AND
                pj.emediaID = p.emediaID) as 'TotalApplies',
              p.emediaName,
              (SELECT
                tu.referrer  as 'referrer'
                FROM
                trackurlapplyattempt tu
                INNER JOIN postedjobs ptj on ptj.reqemediaid = tu.reqemediaid
                WHERE
                ptj.requisitionID =  NAME_CONST('reqID',1312934)
                and
                tu.emediaID=p.emediaID
              
thd->thread_id=65
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
080902 13:20:52  mysqld restarted
InnoDB: Log scan progressed past the checkpoint lsn 19 4254751035
080902 13:20:53  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 19 4257398680
080902 13:20:53  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 163, file name mysql-bin.000010
InnoDB: Last MySQL binlog file position 0 7955072, file name ./mysql-bin.000059
080902 13:20:56  InnoDB: Started; log sequence number 19 4257398680
080902 13:20:56 [Note] Recovering after a crash using mysql-bin
080902 13:20:56 [Note] Starting crash recovery...
080902 13:20:56 [Note] Crash recovery finished.
080902 13:20:56 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.67-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)

-------------- END -----------------

At First I thought the error was due to my over assignment of connections, but after changing those parameters in my.cnf, the issue remains so I have determined it doesn't have anything to do with those parameters. as well as it being repeated on 5.0.51a

Here is my current my.cnf

--------------- START ------------

[client]
#password       = your_password
port            = 3306
socket          = /tmp/mysql.sock

[mysqld]
basedir=/usr/local/mysql
datadir=/var/mysql/data
innodb_log_group_home_dir=/var/mysql/data
port            = 3306
socket          = /tmp/mysql.sock
skip-locking
key_buffer = 256M
max_allowed_packet = 128M
table_cache = 256
sort_buffer_size = 16M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 8M
thread_cache = 24
query_cache_size = 128M
query_cache_type = 1
max_connections = 200
thread_concurrency = 8
tmp_table_size=256M
max_heap_table_size=32M
innodb_buffer_pool_size = 1200M
innodb_flush_log_at_trx_commit = 2
innodb_log_files_in_group=3
innodb_log_file_size=50M
innodb_log_buffer_size=8M
innodb_file_io_threads=4
#sync-binlog=1
#innodb_safe_binlog

innodb_file_per_table

log-bin=mysql-bin
relay-log=iq-mysql-1-relay-bin

server-id       = 1 

log_bin_trust_routine_creators = 1

old-passwords

#skip-slave-start

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 8M
sort_buffer_size = 8M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 8M
sort_buffer_size = 8M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

-------------------- END --------------------

Please let me know if there are any additional questions

How to repeat:
Error is repeated when running a query generated from a stored procedure call from a web site.
[4 Sep 2008 3:22] Valeriy Kravchuk
Thank you for a problem report. had you tried to run mysql_upgrade after upgrading to 5.0.67? See http://dev.mysql.com/doc/refman/5.0/en/mysql-upgrade.html for the details.
[4 Sep 2008 13:18] Alvin Chang
Thanks for your reply,

I forgot to run that script, however I have a replicated machine that I have currently pulled out of production with 5.0.67 and the same databases on it, and I just ran the upgrade script. everything came back with an OK. I will try late this evening by running the web app against it. And I will update this ticket if the issue remains or goes away. 
Thanks

Alvin
[4 Sep 2008 14:45] Valeriy Kravchuk
Please, inform about any results.
[6 Sep 2008 19:05] Alvin Chang
I have tried the same process after upgrading the database with the upgrade script. the error still persists. the server crashes. And reloads. 

any ideas?
[8 Sep 2008 10:59] Valeriy Kravchuk
Please, send the results of EXPLAIN for the crashing query on your system. If possible, both from 5.0.19 (where it does not crash) and 5.0.67.
[8 Oct 2008 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".