Bug #36390 Memory overrun of buffer being freed on startup on x64 Windows, innodb
Submitted: 29 Apr 2008 3:40 Modified: 10 May 2008 21:33
Reporter: Gardner von Holt Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.1.24 OS:Windows (2003 R2 x64)
Assigned to: Heikki Tuuri CPU Architecture:Any

[29 Apr 2008 3:40] Gardner von Holt
Description:
Here is the log that repeates on startup.  The first crash happened when creating an index on a 2.5 million row table, which caused some sort of corruption.

I've tried setting innodb_force_recovery as high as 5 using either my.ini or the command line, and regardless of the setting I get this stack

Seems similar to http://bugs.mysql.com/bug.php?id=34297 , however that case indicates that a fix was checked in, and also this case has the issue of also not being able to restart even using innodb force recovery...

err file follows

080428  5:58:35  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 5 2132928512
InnoDB: Doing recovery: scanned up to log sequence number 5 2138171392
InnoDB: Doing recovery: scanned up to log sequence number 5 2142292413
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1324676 row operations to undo
InnoDB: Trx id counter is 0 525824
080428  5:58:51  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 InnoDB: Error: Memory area size 2048, next area size 0 not a power of 2!
InnoDB: Possibly a memory overrun of the buffer being freed here.
InnoDB: Apparent memory corruption: mem dump  len 500; hex 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000009268a5200000000074306d656d2e63002f0000000000000000000000000000000000000000000000000000000000000078001904000000000000000000000000b00400000000000000000000000000000000000000000000d80000000000000070000000000000000000000000000000e8e818040000000000000000000000000010100400000000f0e81804000000000000000000000000001090000000000010e91804000000000000000000000000001820000410004018e91804000000000000000000000000001c000000000000000000000000000000000000000000000000; asc                                                                                                                                                                                                                                                                                   ’h¥     t0mem.c /                               x               °                       Ø       p               èè                      ðè                       é                     @ é                                        ;
InnoDB: Scanning backward trying to find previous allocated mem blocks
Freed mem block at - 1000, file t0mem.c, line 47
Mem block at - 2024, file t0mem.c, line 47
Freed mem block at - 3048, file t0mem.c, line 47
Mem block at - 4072, file t0mem.c, line 246
Freed mem block at - 5096, file t0mem.c, line 246
Freed mem block at - 6120, file t0mem.c, line 246
Freed mem block at - 7144, file t0mem.c, line 246
Freed mem block at - 8168, file t0mem.c, line 47
Mem block at - 10216, file purge.c, line 177
Freed mem block at - 10472, file t0mem.c, line 246
InnoDB: Scanning forward trying to find next allocated mem blocks
Freed mem block at + 24, file t0mem.c, line 47
Freed mem block at + 1048, file t0mem.c, line 246
Freed mem block at + 2072, file t0mem.c, line 246
Freed mem block at + 3096, file t0mem.c, line 246
Freed mem block at + 4120, file t0mem.c, line 246
Freed mem block at + 6168, file t0mem.c, line 246
Freed mem block at + 7192, file t0mem.c, line 246
Freed mem block at + 8216, file t0mem.c, line 47
Freed mem block at + 14360, file t0mem.c, line 246
Freed mem block at + 16408, file t0mem.c, line 47
080428  5:58:54  InnoDB: Assertion failure in thread 3660 in file .\mem\mem0pool.c line 518
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Thread 744 stopped in file C:\cygwin\home\mysqldev\build\mysql-5.1.24-rc-winbuild\mysql-community-nt-5.1.24-rc-build\storage\innobase\include\sync0sync.ic line 115
080428  5:58:54 - mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=26214400
read_buffer_size=65536
max_used_connections=0
max_threads=60
threads_connected=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 45328 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
00000000007BE85F    mysqld.exe!???
0000000000778CEE    mysqld.exe!???
0000000000778FAF    mysqld.exe!???
00000000007F3958    mysqld.exe!???
00000000007F4151    mysqld.exe!???
00000000007BCA9A    mysqld.exe!???
00000000007B35CD    mysqld.exe!???
000000000077FB98    mysqld.exe!???
0000000077D6B6DA    kernel32.dll!BaseThreadStart()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
Repeats for me every time I try to start the service, but not sure how to repro the root cause.
[29 Apr 2008 14:18] Heikki Tuuri
I am not sure if the fix of:

http://bugs.mysql.com/bug.php?id=34297

made it to 5.1.24 on Windows.

This is probably the same bug if the fix is not yet there.
[29 Apr 2008 14:20] Heikki Tuuri
Gardner,

how did you build the mysqld binary or where did you get it?

--Heikki
[29 Apr 2008 14:29] Gardner von Holt
Heikki,

I did NOT build this, I downloaded the Windows 64 bit Installer (the full version with installer, not the essentials version) from some mirror of mysql.com reached from http://dev.mysql.com/downloads/mysql/5.1.html#winx64
[29 Apr 2008 14:33] Heikki Tuuri
Gardner,

ok. I guess the problem is that the bug fix of that other bug report is not yet in 5.1.24 :(. Since the fix requires changing the compilation flags, maybe getting it to the build is slower than for an ordinary bug fix.

--Heikki
[1 May 2008 2:09] MySQL Verification Team
Please see the attached file of bug: http://bugs.mysql.com/bug.php?id=36366.
After a crash InnoDB 5.0.25 was able to recovery in the 2nd server start and
with similar error messages. Thanks in advance.
[2 May 2008 2:20] Gardner von Holt
Thanks for the information that 5.1.25 is able to recover.

The status on this issue says "need feedback".  Not sure what I can tell you, since I don't have access to 5.1.25, do I?  (I assume you didn't really mean 5.0.25).
[10 May 2008 21:33] MySQL Verification Team
Thank you for the feedback. I am closing this bug as fixed in source tree,
please wait for the release or build it from the source according the
instructions in our Manual. Thanks in advance.