MySQL Bugs: #46683: Unknown exception at 0xc0000005

Bug #46683	Unknown exception at 0xc0000005
Submitted:	12 Aug 2009 22:26	Modified:	14 Dec 2009 0:45
Reporter:	L U	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: General	Severity:	S1 (Critical)
Version:	5.1.39, 5.4.2-beta	OS:	Windows (2003 x64)
Assigned to:		CPU Architecture:	Any

Description:
http://bugs.php.net/bug.php?id=49191

How to repeat:
n/a

Suggested fix:
n/a

Thank you for the bug report. If I understood this isn't a server bug but a libmysql.dll issue right?. Are you able to provide a test case (step by step intructions how to repeat the dll crash?). Thanks in advance.

Well, you would run IIS6 in ISAPI:

<?php
$db = new mysqli($mysql_host,$mysql_user,$mysql_pass,$mysql_db) or
die("DB failed to connect.");
$db->set_charset('utf8');
?>

Run this script a few times from the browser, and the debugger picks up first chance exceptions (I don't think this is good? I don't know how to debug, just supplying what the program gave me.). I haven't gotten an access error for a long while now since upgrading to the 5.2.11 version of PHP.

Many thanks for writing a bug report.

The test works fine without web server and also with open source web server.

So this is neither a MySQL nor a PHP bug.

Your problem is related to IIS.

Thank you for the feedback.

Please add option log-warnings=2 into my.ini file, restart server, then repeat the problem and send us error log file.

090910 16:39:04 [Note] C:\Program Files\MySQL\MySQL Server 5.1\bin\mysqld: Normal shutdown

090910 16:39:04 [Note] Event Scheduler: Purging the queue. 0 events
090910 16:39:06 [Warning] C:\Program Files\MySQL\MySQL Server 5.1\bin\mysqld: Forcing close of thread 211816  user: '[redacted]'

090910 16:39:06 [Warning] C:\Program Files\MySQL\MySQL Server 5.1\bin\mysqld: Forcing close of thread 12084  user: '[redacted]'

090910 16:39:06  InnoDB: Starting shutdown...
090910 16:39:09  InnoDB: Shutdown completed; log sequence number 0 1489449111
090910 16:39:09 [Warning] Forcing shutdown of 1 plugins
090910 16:39:09 [Note] C:\Program Files\MySQL\MySQL Server 5.1\bin\mysqld: Shutdown complete

090910 16:39:13 [Note] Plugin 'FEDERATED' is disabled.
090910 16:39:16  InnoDB: Started; log sequence number 0 1489449111
090910 16:39:17 [Note] Event Scheduler: Loaded 0 events
090910 16:39:17 [Note] C:\Program Files\MySQL\MySQL Server 5.1\bin\mysqld: ready for connections.
Version: '5.1.38-community'  socket: ''  port: 3306  MySQL Community Server (GPL)

There was nothing new in this file, the only err file. It was located in the data directory. I shut the server down, and started it up again thinking maybe it was withholding anything. It wasn't.

[10-Sep-2009 21:17:55] PHP Warning:  mysqli::mysqli() [<a href='mysqli.mysqli'>mysqli.mysqli</a>]: (HY000/2003): Can't connect to MySQL server on 'localhost' (10061)

090910 17:52:24 [Warning] Aborted connection 198200 to db: 'db1' user: 'uname1' host: 'localhost' (Got an error reading communication packets)
090910 17:57:08 [Warning] Aborted connection 210862 to db: 'db1' user: 'uname1' host: 'localhost' (Got an error reading communication packets)
090910 18:07:09 [Warning] Aborted connection 237564 to db: 'db2' user: 'uname1' host: 'localhost' (Got an error reading communication packets)
090910 18:12:18 [Warning] Aborted connection 229100 to db: 'db2' user: 'uname1' host: 'localhost' (Got timeout reading communication packets)

Not much more than this. DB2 is a large database of around 450MB of uncompressed records, db1 is small, 250KB, at most, uncompressed.

I've been getting a lot of 'MySQL server has gone away' type messages appearing on my PHP error log. I do not know the root cause of this. The problem seems to happen less when I use 127.0.0.1 than using localhost. Makes no sense, but that's what is happening.

Got another one, and performance is still suffering. Any suggestion on what I can do to help this ticket get solved? I do not know how to use a debugger on a Windows 2003 x64, so if I need to do that, you would have to walk me through it.

090916 18:57:35 - mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=55574528
read_buffer_size=65536
max_used_connections=254
max_threads=800
threads_connected=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 317422 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x37049490
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
0000000140007EE1    mysqld.exe!memcpy()[memcpy.asm:278]
000000014011B79A    mysqld.exe!String::append()[sql_string.cc:459]
00000001400578B9    mysqld.exe!thd_security_context()[sql_class.cc:387]
0000000140208F17    mysqld.exe!innobase_mysql_print_thd()[ha_innodb.cc:794]
0000000140216A2F    mysqld.exe!trx_print()[trx0trx.c:1764]
0000000140237C9D    mysqld.exe!lock_print_info_all_transactions()[lock0lock.c:4297]
000000014021378F    mysqld.exe!srv_printf_innodb_monitor()[srv0srv.c:1695]
000000014020F5C9    mysqld.exe!innodb_show_status()[ha_innodb.cc:7223]
000000014009BED2    mysqld.exe!ha_show_status()[handler.cc:4417]
000000014006A70D    mysqld.exe!mysql_execute_command()[sql_parse.cc:2426]
000000014006E736    mysqld.exe!mysql_parse()[sql_parse.cc:5935]
000000014006F2CA    mysqld.exe!dispatch_command()[sql_parse.cc:1215]
000000014006FF37    mysqld.exe!do_command()[sql_parse.cc:854]
00000001400963C7    mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
0000000140315565    mysqld.exe!pthread_start()[my_winthread.c:85]
00000001402DF7D7    mysqld.exe!_callthreadstart()[thread.c:295]
00000001402DF8A5    mysqld.exe!_threadstart()[thread.c:275]
0000000077D6B71A    kernel32.dll!BaseThreadStart()
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 000000003708C560=SHOW INNODB STATUS
thd->thread_id=689878
thd->killed=NOT_KILLED

An error that was output when refreshing process list. Server died before or during this process.

Attachment: error.bmp (image/bmp, text), 270.05 KiB.

Thank you for the feedback.

Crash when issueing SHOW INNODB STATUS is not related to the original problem. Anyway, please, provide information about your OS and hardware: is it 32-bit or 64-bit? How many physical RAM do you have?

Windows Server 2003 X64 Enterprise
Asus DSBV-DX Server Board
20GB RAM
Dual Quad Core E5405

I have moved MySQL to another server running the same hardware, but less of it, and have experienced no problems so far.

That server is the same board, 2GB ram and one Quad core E5405.

I would say bad memory however the RAM checked out fine for me.

The new server has been experiencing the same type of issues it seems. There have been no memory dumps, but for 1-10 seconds every 4-10 hours it doesn't respond to new connections or ones that have already been made.

Thank you for the feedback. I tried to run the script you have provide without luck to repeat the crash, are you tested the release 5.1.39?. Thanks in advance.

It takes 4-10 hours for an error to occur in the logs. That and the traffic is larger than just one machine making queries. It averages 1,400 queries per second, 100-200 connections per second, and consists of updates, deletes, and selects all day. So there are probably contention issues somewhere.

The .39 version of MySQL has not been installed, but I will later today or tomorrow.

Thank you for the update.

Setting to "Need feedback" as we waiting results from your testing.

There have been no crash dumps, however MySQL stops responding still for about 10 seconds, very often now.

There is no more information I am able to provide unless you guys are able to tell me what you need. The MySQL server just locks up from serving responses and within 10 seconds it comes back online after rejecting tons of connections during that period. Nothing is logged in the error file, but I am unable to log in using the admin tools when this is occuring. It complains that the server is not running and offers to try a ping.

Heavy Selects and Updates are done. Inserts average five every minute.

091004  9:51:26 - mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=25165824
read_buffer_size=65536
max_used_connections=801
max_threads=800
threads_connected=6
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 287726 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x28035240
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
0000000140007ED0    mysqld.exe!memcpy()[memcpy.asm:272]
000000014011BBEA    mysqld.exe!String::append()[sql_string.cc:459]
0000000140057A29    mysqld.exe!thd_security_context()[sql_class.cc:387]
0000000140209537    mysqld.exe!innobase_mysql_print_thd()[ha_innodb.cc:794]
000000014021704F    mysqld.exe!trx_print()[trx0trx.c:1764]
00000001402382BD    mysqld.exe!lock_print_info_all_transactions()[lock0lock.c:4297]
0000000140213DAF    mysqld.exe!srv_printf_innodb_monitor()[srv0srv.c:1695]
000000014020FBE9    mysqld.exe!innodb_show_status()[ha_innodb.cc:7223]
000000014009C092    mysqld.exe!ha_show_status()[handler.cc:4417]
000000014006A87D    mysqld.exe!mysql_execute_command()[sql_parse.cc:2426]
000000014006E8A6    mysqld.exe!mysql_parse()[sql_parse.cc:5935]
000000014006F43A    mysqld.exe!dispatch_command()[sql_parse.cc:1215]
00000001400700A7    mysqld.exe!do_command()[sql_parse.cc:854]
00000001400965B7    mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
0000000140315B25    mysqld.exe!pthread_start()[my_winthread.c:85]
00000001402DFDA7    mysqld.exe!_callthreadstart()[thread.c:295]
00000001402DFE75    mysqld.exe!_threadstart()[thread.c:275]
0000000077D6B71A    kernel32.dll!BaseThreadStart()
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0000000028022E40=SHOW INNODB STATUS
thd->thread_id=67051
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

This is a new crash just recently appeared. This is the dedicated server. It locked up the whole box this time and took about 4 minutes to come back online.

I understand you guys are busy, but can I get an idea if someone is at least reading this ticket? I'd hate to find that this is not going to be fixed, not enough information is provided or the trace is not correct or it is not a bug and I'm imagining the MySQL ceasing to respond to connections for short periods of time until the eventual PHP access violations kick in.

091008  8:26:31 - mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=25165824
read_buffer_size=65536
max_used_connections=1239
max_threads=2000
threads_connected=34
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 675640 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x37c849f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
0000000140005755    memcpy()[memcpy.asm:284]
000000014010642A    String::append()[sql_string.cc:459]
000000014004EBEF    thd_security_context()[sql_class.cc:387]
00000001401EADE7    innobase_mysql_print_thd()[ha_innodb.cc:928]
00000001401F9851    trx_print()[trx0trx.c:1750]
00000001402215F6    lock_print_info_all_transactions()[lock0lock.c:4374]
00000001401F6716    srv_printf_innodb_monitor()[srv0srv.c:1752]
00000001401F1B80    innodb_show_status()[ha_innodb.cc:8076]
0000000140091472    ha_show_status()[handler.cc:4418]
0000000140060F6E    mysql_execute_command()[sql_parse.cc:2456]
0000000140064FD5    mysql_parse()[sql_parse.cc:6007]
0000000140065B61    dispatch_command()[sql_parse.cc:1224]
0000000140066777    do_command()[sql_parse.cc:855]
000000014008BAA7    handle_one_connection()[sql_connect.cc:1131]
00000001402F93C5    pthread_start()[my_winthread.c:85]
00000001402D4697    _callthreadstart()[thread.c:295]
00000001402D4741    _threadstart()[thread.c:275]
0000000077D6B71A    BaseThreadStart()
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 00000000356E1C60=SHOW INNODB STATUS
thd->thread_id=1154304
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Event Type:	Error
Event Source:	Application Error
Event Category:	(100)
Event ID:	1000
Date:		10/8/2009
Time:		8:38:46 AM
User:		N/A
Computer:	X
Description:
Faulting application mysqld.exe, version 0.0.0.0, faulting module mysqld.exe, version 0.0.0.0, fault address 0x0000000000005755.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 46 61 69 6c   ion Fail
0010: 75 72 65 20 20 6d 79 73   ure  mys
0018: 71 6c 64 2e 65 78 65 20   qld.exe 
0020: 30 2e 30 2e 30 2e 30 20   0.0.0.0 
0028: 69 6e 20 6d 79 73 71 6c   in mysql
0030: 64 2e 65 78 65 20 30 2e   d.exe 0.
0038: 30 2e 30 2e 30 20 61 74   0.0.0 at
0040: 20 6f 66 66 73 65 74 20    offset 
0048: 30 30 30 30 30 30 30 30   00000000
0050: 30 30 30 30 35 37 35 35   00005755

So, you have a crash when you run SHOW INNODB STATUS under high load, on 64-bit Windows, on both 5.4.2 and 5.1.38 at least. 

How much RAM do you have on that box? Please, send your my.ini file content and SHOW GLOBAL STATUS results after running server for long enough time.

another my.ini with the same problems.

Attachment: my.ini (application/octet-stream, text), 9.05 KiB.

I left the MySQL Administrator open and that is probably why that shows up in the log. I've had this issue in the .39 as well.

6GB RAM (Upgraded from 2GB in hopes of fixing the crashing)
Quad Core E5405

Let me add some comments to your my.ini file content:

max_connections=2000

This is crazy by itself for Windows as we still have limit on open file handles, 2048. Set it to 500 at most. Check http://dev.mysql.com/doc/refman/5.4/en/limits-windows.html.

table_cache=2420

This is also crazy, for the same reason. Set it to 1000 at most.

Now, this:

tmp_table_size=35M

is also too big for your RAM if you really have a lot of concurrent connections. Send the results of:

show global status like 'max%';
show global status like '%tmp%';

For now I see wrong configuration as one of the most probable reasons for this crash.

See Bug #38883 also, but that one should be fixed since 5.1.31...

This was the my.ini from my server with 20GB of RAM which I copied over to the other server. The max connections used to be 800 but the crashing still occurred. You can see the previous my.ini settings at the top of this ticket.

Usage does not go over 4GB usage on the dedicated one with the recent configuration.

mysql> show global status like 'max%';
+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 189   |
+----------------------+-------+
1 row in set (0.00 sec)

mysql> show global status like '%tmp%';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 0     |
| Created_tmp_files       | 189   |
| Created_tmp_tables      | 22    |
+-------------------------+-------+
3 rows in set (0.00 sec)

looks like bug #38883 hits again!

Would this be causing intermittent stutters with MySQL just not accepting connections? Or is that issue unrelated?

another crash like this reported at bug #48115

See http://bugs.mysql.com/bug.php?id=48115.

I have disabled the query cache and the intermittent lockups appear to have mostly vanished. I guess the MySQL server takes a second to invalidate the cache and is unable to continue processing new connections.

As of the odd crash from SHOW INNODB STATUS from the administrator tool, I have not, since last reported, received this problem with the 5.4.2 beta.

The SHOW INNODB STATUS related crashes of this bug are most certainly a duplicate of Bug #38883 - fixed in 5.1.42, 5.0.89 -- (to be released soon)

There are a number of bugs reported against query cache already that address these intermittent lockups.

Closing this as duplicate.