Bug #40796 Crash due to heap corruption in rpl.rpl_extraColmaster_myisam

• Submitted: 17 Nov 2008 19:02
• Modified: 27 Aug 18:48
• Reporter: Vladislav Vaintroub
• Status: Patch queued
• Category: Server: Replication
• Severity: S2 (Serious)
• Version: 5.1.30
• OS: Microsoft Windows
• Assigned to: Davi Arnaut
• Target Version: 5.1+
• Tags: disabled
• Triage: Triaged: D1 (Critical)

[17 Nov 2008 19:02] Vladislav Vaintroub

Description:
mysql crashed in  rpl.rpl_extraColmaster_myisam.

with this callstack 

81117 20:51:58 - mysqld got exception 0xc0000005 ;
thd: 0x9839a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
0000000077938B42    ntdll.dll!RtlAllocateHeap()
00000000779391F7    ntdll.dll!RtlAllocateHeap()
000000007771CEAA    kernel32.dll!HeapFree()
00000001403C5A1B    mysqld.exe!free()[free.c:110]
00000001402A450E    mysqld.exe!my_no_flags_free()[my_malloc.c:62]
00000001402AA0DC    mysqld.exe!end_io_cache()[mf_iocache.c:1828]
00000001400DA949    mysqld.exe!MYSQL_LOG::close()[log.cc:1977]
00000001400DC13E    mysqld.exe!MYSQL_BIN_LOG::close()[log.cc:4526]
00000001400DD7FE    mysqld.exe!MYSQL_BIN_LOG::reset_logs()[log.cc:2837]
00000001402411A4    mysqld.exe!purge_relay_logs()[rpl_rli.cc:923]
00000001401BC4A1    mysqld.exe!reset_slave()[sql_repl.cc:983]
0000000140199077    mysqld.exe!reload_acl_and_cache()[sql_parse.cc:6775]
00000001401A03A9    mysqld.exe!mysql_execute_command()[sql_parse.cc:3860]
00000001401A2F06    mysqld.exe!mysql_parse()[sql_parse.cc:5792]
00000001401A3C1A    mysqld.exe!dispatch_command()[sql_parse.cc:1203]
00000001401A4CD7    mysqld.exe!do_command()[sql_parse.cc:858]
0000000140246327    mysqld.exe!handle_one_connection()[sql_connect.cc:1116]
00000001402B82C5    mysqld.exe!pthread_start()[my_winthread.c:86]
00000001403CAC37    mysqld.exe!_callthreadstart()[thread.c:295]
00000001403CAD05    mysqld.exe!_threadstart()[thread.c:275]
000000007771495D    kernel32.dll!BaseThreadInitThunk()
0000000077918791    ntdll.dll!RtlUserThreadStart()
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 00000000028EF480=RESET SLAVE
thd->thread_id=1
thd->killed=NOT_KILLED

How to repeat:
perl mysql-test.run.pl --suite=rpl rpl_extraColmaster_myisam rpl_extraColmaster_myisam
rpl_extraColmaster_myisam rpl_extraColmaster_myisam

(insert many rpl_extraColmaster_myisams, the crash is not always reproducible, but likely
after running the test 20 times or so)

Suggested fix:
Don't know

[17 Nov 2008 19:08] Vladislav Vaintroub

Running test gives different crashes, but always something around malloc/free.
Here is an example of crash from the same test that prevents signal handler to work
properly.

 	mysqld.exe!__C_specific_handler(_EXCEPTION_RECORD * ExceptionRecord=0x000000000304ff00,
void * EstablisherFrame=0x000000000304ff00, _CONTEXT * ContextRecord=0x0000000003050000,
_DISPATCHER_CONTEXT * DispatcherContext=0x00000001403cac37)  + 0x96 bytes	C
 	ntdll.dll!RtlpExecuteHandlerForException()  + 0xd bytes	
 	ntdll.dll!RtlDispatchException()  + 0x1d7 bytes	
 	ntdll.dll!KiUserExceptionDispatcher()  + 0x2e bytes	
 	ntdll.dll!RtlpFreeHeap()  + 0x5e9 bytes	
 	ntdll.dll!RtlFreeHeap()  + 0x1967 bytes	
 	kernel32.dll!HeapFree()  + 0xa bytes	
 	mysqld.exe!free(void * pBlock=0x0000000002813230)  Line 110	C
 	mysqld.exe!_freefls(void * data=0x0000000000297cb0)  Line 754	C
 	mysqld.exe!_freeptd(_tiddata * ptd=0x0000000002068820)  Line 807	C
 	mysqld.exe!_endthread()  Line 364	C
 	mysqld.exe!handle_slave_io(void * arg=0x0000000002068820)  Line 2555	C++

[19 Nov 2008 0:14] Trudy Pelzer

Per Vlad, bug#35319 is probably related:
"After searching a bit in the bugdb, heap corruption with rpl involved
appears to be reported on MacOSX http://bugs.mysql.com/bug.php?id=35319 
Stack looks similar to what I've seen in http://bugs.mysql.com/bug.php?id=40796"

[25 Nov 2008 13:51] Giuseppe Maxia

Also verified on Mac OSX 10.5

[22 Apr 12:16] Alfranio Correia

Apparently the failures related to the following test cases are the same::
rpl_extraColmaster_myisam
rpl_extraColmaster_falcon
rpl_extraCol_myisam
rpl_extraCol_falcon
rpl_extraColmaster_innodb
rpl_extraCol_innodb
rpl_extraColmaster_myisam
rpl_extraCol_myisam
rpl_extraColmaster_innodb
rpl_extraCol_innodb

Look also at BUG#40930.

[7 Jul 16:42] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78150

3011 Georgi Kodinov	2009-07-07
      Bug #40796 : adding the tests to the experimental pb2 set.

[8 Jul 15:30] Bugs System

Pushed into 5.1.37 (revid:[e-mail]-20090708131116-kyz8iotbum8w9yic) (version source
revid:[e-mail]-20090707144134-58cwiaz4qzy6n35c) (merge vers: 5.1.37) (pib:11)

[9 Jul 9:36] Bugs System

Pushed into 5.1.37 (revid:[e-mail]-20090708131116-kyz8iotbum8w9yic) (version source
revid:[e-mail]-20090707144134-58cwiaz4qzy6n35c) (merge vers: 5.1.37) (pib:11)

[10 Jul 13:20] Bugs System

Pushed into 5.4.4-alpha
(revid:[e-mail]-20090710111017-bnh2cau84ug1hvei) (version source
revid:[e-mail]-20090708121727-rekm6n1iu4vmvcfa) (merge vers: 5.4.4-alpha) (pib:11)

[14 Aug 13:15] Davi Arnaut

Queued to 5.0-bugteam:

  http://lists.mysql.com/commits/80781

[26 Aug 15:45] Bugs System

Pushed into 5.1.37-ndb-7.0.8 (revid:[e-mail]-20090826132541-yablppc59e3yb54l)
(version source revid:[e-mail]-20090826132541-yablppc59e3yb54l) (merge vers:
5.1.37-ndb-7.0.8) (pib:11)

[26 Aug 15:46] Bugs System

Pushed into 5.1.37-ndb-6.3.27 (revid:[e-mail]-20090826105955-bkj027t47gfbamnc)
(version source revid:[e-mail]-20090826105955-bkj027t47gfbamnc) (merge vers:
5.1.37-ndb-6.3.27) (pib:11)

[26 Aug 15:48] Bugs System

Pushed into 5.1.37-ndb-6.2.19 (revid:[e-mail]-20090825194404-37rtosk049t9koc4)
(version source revid:[e-mail]-20090825194404-37rtosk049t9koc4) (merge vers:
5.1.37-ndb-6.2.19) (pib:11)

[27 Aug 18:07] Jon Stephens

Documented bugfix in the 5.1.37 changelog as follows:

        A memory allocation error in the internal vio_new() function
        could cause stack corruption leading to a crash of the slave.
        This issue was observed when replicating from tables having
        extra columns on the master as compared to the slave.
      
Set status to Patch Queued, waiting for push to 5.0 tree.

(Will this also be pushed to 5.4?)

[27 Aug 18:32] Bugs System

Pushed into 5.1.35-ndb-7.1.0 (revid:[e-mail]-20090827163030-6o3kk6r2oua159hr)
(version source revid:[e-mail]-20090826132541-yablppc59e3yb54l) (merge vers:
5.1.37-ndb-7.0.8) (pib:11)

[28 Aug 10:37] Jon Stephens

Removed this changelog entry, combined with Bug#45242 changelog entry (qv.) per Davi's
email.

Left status unchanged.