Bug #41868 crash or memory overrun with concat + upper, date_format functions
Submitted: 5 Jan 2009 14:09 Modified: 9 Mar 2009 14:49
Reporter: Germán Cáseres Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: DML Severity:S1 (Critical)
Version:5.1.30, 6.0.10, 6.0.8, 5.0.76, 5.0.74,, 5.0.40 OS:Any
Assigned to: Alexey Kopytov CPU Architecture:Any
Tags: concat, crash, date_format, upper, valgrind
Triage: Triaged: D1 (Critical)

[5 Jan 2009 14:09] Germán Cáseres
Description:
Hi,
   I have a problem with a database when executing some queries. Those queries seems to be random, or al least I was unable to reproduce the error.
   I receive exception 0xc0000005 (Access Violation) on mysqld.
   I tried Reinstalling the server, and also tried installing it on another computer.

   The error log shows two occurences of the same error with different queries. I tried executing the queries again, but no error appeared.

I use mysql with Apache 2.2 and PHP5. Last stable version on both.

How to repeat:
Can't find a way to repeat, the error appears almost randomnly
[5 Jan 2009 14:10] Germán Cáseres
MySQL Config file

Attachment: my.ini (application/octet-stream, text), 8.78 KiB.

[5 Jan 2009 14:10] Germán Cáseres
MySQL error log

Attachment: mysql.err (application/octet-stream, text), 8.51 KiB.

[5 Jan 2009 15:34] Miguel Solorzano
Thank you for the bug report. Are you able to provide the dump file of the database so we can try on our side?. The dump could be private if you wish.
Thanks in advance.
[5 Jan 2009 15:36] Valeriy Kravchuk
Thank you for hte problem report. Is it 32-bit or 64-bit Windows? How much RAM do you have? 

Please, try also to identify complete queries that lead to crash (I am not sure second one is complete) and send EXPLAIN results for them.
[5 Jan 2009 16:07] Germán Cáseres
Miguel,
       i will provide the dump as soon as i have access to the server.
[5 Jan 2009 16:08] Germán Cáseres
Valeriy,
        its 32-bits with 1Gb of RAM.

        I don't know why the second q
[5 Jan 2009 16:10] Germán Cáseres
Valeriy,
        its 32-bits with 1Gb of RAM.

        I don't know why the second query is incomplete. Maybe because it's too long. I'll try to identify it.

        Anyway if i execute the first query, it works well (from mysql console or from the web application)
[5 Jan 2009 18:12] Germán Cáseres
Miguel, i have uploaded the dump. Please take a look at it when you can.
[6 Jan 2009 7:29] Shane Bester
Hi,  Thanks for the dump.  I found a problem with first query:

Version: '5.1.30-enterprise-gpl-advanced-debug'  socket: ''  port: 3306  MySQL Enterprise Server - Advanced Edition Debug (GPL)
Error: Memory allocated at .\sql_string.cc:82 was overrun, discovered at '.\sql_string.cc:51'
Error: Memory allocated at .\sql_string.cc:82 was overrun, discovered at '.\sql_string.cc:74'
Error: Memory allocated at .\sql_string.cc:82 was overrun, discovered at '.\sql_string.cc:74'
Error: Memory allocated at .\sql_string.cc:82 was overrun, discovered at '.\sql_string.cc:74'

Will work to make a smaller testcase, and update the bug report when done.
[6 Jan 2009 8:49] Shane Bester
I updated the synopsis to be more specific.
Here's a minimal testcase to show the problem:

------
drop table if exists `t1`;
create table `t1` (`c` date,aa varchar(30)) engine=myisam;
insert into `t1` values ('2008-12-31','aaaaaa');
select  date_format(`c`,get_format(date,'eur')) `h`,
concat(upper(`aa`),', ', `aa`) `i` from `t1`;
------

valgrind errors and safe-malloc errors, or crashes can be observed on 5.1.30, 6.0.8, 6.0.10, 5.0.74, and 5.0.40. So this is an old bug.

The crash is not repeatable each time, so please use mysqld-debug or valgrind to
check:

1 errors in context 1 of 1:
Invalid write of size 1
 copy_if_not_alloced(String*, String*, unsigned) (sql_string.cc:86)
 Item_str_conv::val_str(String*) (item_strfunc.cc:1010)
 Item_func_concat::val_str(String*) (item_strfunc.cc:286)
 Item::send(Protocol*, String*) (item.cc:5093)
 select_send::send_data(List<Item>&) (sql_class.cc:1040)
 _ZL8end_sendP4JOINP13st_join_tableb (sql_select.cc:11527)
 _ZL9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure (sql_select.
 JOIN::exec() (sql_select.cc:2106)
 mysql_select(THD*, Item***, TABLE_LIST*, unsigned, List<Item>&, It
 handle_select(THD*, st_lex*, select_result*, unsigned long) (sql_s
[6 Jan 2009 17:05] Germán Cáseres
I'm glad that you've reached a testcase, you're really a genius. I'll try to work around that query.
Yesterday the error occured again, but with another query. This time a very simple query:

Insert into usuarios_actividad (idusuario, fecha, idactividad)
values('2795', '2009-01-06 08:22:11', 6)

I'm attaching the entire log for that error, please tell me if i should do this in a new bug notification.

Thank you, you all are really helpful
[6 Jan 2009 17:07] Germán Cáseres
MYSQL error log updated 06 Jan 2009

Attachment: mysql-06012009.err (application/octet-stream, text), 2.82 KiB.

[6 Jan 2009 17:25] Shane Bester
Germán, it's common in memory corruption/overrun bugs that internal data structures get trashed, and a crash only happens much later - seemingly with an innocent statement.  Especially when free() and malloc() crash, this is usually the case.  So, all these other crashes are probably related to the original one for which I made a testcase.
[7 Jan 2009 17:27] Shane Bester
bug #41927 might be related ..
[27 Jan 2009 14:05] Alexey Kopytov
Two problems here:

1. String::realloc() does not check whether the existing string data fits in the newly allocated buffer for cases when reallocating a String object with external buffer (i.e. alloced == FALSE).  As a result, when shrinking an existing string, the newly allocated buffer gets overrun. This is what was causing crashes/memory overruns in this bug.

2. Even when the above is fixed, we still have a problem with excessive malloc() calls in select_send::send_data(). Before iterating through Items and calling the send() method for each Item, we setup a 'buffer' object pointing to a preallocated memory chunk which is passed to Item::send() and apparently is supposed to be reused for each Item::send(). It turns out it's not always the case, since Item::send() (or rather, internal *::val_str() calls) may reset 'buffer' object to point to another, sometimes statically allocated, memory region. Therefore, next calls to Item::send() will likely not be using the preallocated memory, but instead do their own allocations. This is what was happening during execution of the query from this bug's test case.

#2 is easy to fix by resetting the 'buffer' object to its original state after each Item::send() call.
[27 Jan 2009 14:10] Alexey Kopytov
Bug #41927 is a duplicate of this one.
[6 Feb 2009 9:51] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65445

2741 Alexey Kopytov	2009-02-06
      Temporarily reverted patch for bug #41868 as it was causing problems in PB.
[6 Feb 2009 13:42] Bugs System
Pushed into 5.0.78 (revid:joro@sun.com-20090206133929-ybpbkkox9wkptopb) (version source revid:alexey.kopytov@sun.com-20090206095111-c7thecg2li0mgbcl) (merge vers: 5.0.78) (pib:6)
[9 Feb 2009 22:34] Bugs System
Pushed into 5.1.32 (revid:davi.arnaut@sun.com-20090209214102-gj3sb3ujpnvpiy4c) (version source revid:davi.arnaut@sun.com-20090209214102-gj3sb3ujpnvpiy4c) (merge vers: 5.1.32) (pib:6)
[10 Feb 2009 12:39] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65736

2735 Alexey Kopytov	2009-02-10
      Fix for bug #41868: crash or memory overrun with concat + upper,
                          date_format functions
      
      String::realloc() did not check whether the existing string data fits in
      the newly allocated buffer for cases when reallocating a String object
      with external buffer (i.e.alloced == FALSE).  This could lead to memory
      overruns in some cases.
[14 Feb 2009 13:01] Bugs System
Pushed into 6.0.10-alpha (revid:matthias.leich@sun.com-20090212211028-y72faag15q3z3szy) (version source revid:alexey.kopytov@sun.com-20090206095654-32mkrlbe7cuvw0v4) (merge vers: 6.0.10-alpha) (pib:6)
[17 Feb 2009 15:01] Bugs System
Pushed into 5.1.32-ndb-6.3.23 (revid:tomas.ulin@sun.com-20090217131017-6u8qz1edkjfiobef) (version source revid:tomas.ulin@sun.com-20090216083408-rmvyaxjt6mk8sg1y) (merge vers: 5.1.32-ndb-6.3.23) (pib:6)
[17 Feb 2009 16:48] Bugs System
Pushed into 5.1.32-ndb-6.4.3 (revid:tomas.ulin@sun.com-20090217134419-5ha6xg4dpedrbmau) (version source revid:tomas.ulin@sun.com-20090216083646-m8st11oj1hhfuuh5) (merge vers: 5.1.32-ndb-6.4.3) (pib:6)
[17 Feb 2009 18:24] Bugs System
Pushed into 5.1.32-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090217134216-5699eq74ws4oxa0j) (version source revid:tomas.ulin@sun.com-20090211111208-wf0acl7c1vl5653e) (merge vers: 5.1.32-ndb-6.2.17) (pib:6)
[19 Feb 2009 13:04] Bugs System
Pushed into 6.0.10-alpha (revid:sergey.glukhov@sun.com-20090218125737-5y5b2xo3duo1wlvo) (version source revid:azundris@mysql.com-20090213182753-tsqikd41gkwowrqq) (merge vers: 6.0.10-alpha) (pib:6)
[27 Feb 2009 20:51] Paul Dubois
Noted in 5.1.32, 6.0.10 changelogs.

String reallocation could cause memory overruns.

Setting report to NDI pending push into 5.0.x.
[9 Mar 2009 14:12] Bugs System
Pushed into 5.0.79 (revid:joro@sun.com-20090309135922-a0di9ebkxoj4d4wv) (version source revid:alexey.kopytov@sun.com-20090213161707-gax0uem47igu776s) (merge vers: 5.0.79) (pib:6)
[9 Mar 2009 14:49] Paul Dubois
Noted in 5.0.79 changelog.
[13 Mar 2009 19:08] Bugs System
Pushed into 5.1.33 (revid:joro@sun.com-20090313111355-7bsi1hgkvrg8pdds) (version source revid:vvaintroub@mysql.com-20090218093153-sjzxf01i4ezte0xp) (merge vers: 5.1.33) (pib:6)
[9 May 2009 16:47] Bugs System
Pushed into 5.1.34-ndb-6.2.18 (revid:jonas@mysql.com-20090508185236-p9b3as7qyauybefl) (version source revid:jonas@mysql.com-20090508100057-30ote4xggi4nq14v) (merge vers: 5.1.33-ndb-6.2.18) (pib:6)
[9 May 2009 17:43] Bugs System
Pushed into 5.1.34-ndb-6.3.25 (revid:jonas@mysql.com-20090509063138-1u3q3v09wnn2txyt) (version source revid:jonas@mysql.com-20090508175813-s6yele2z3oh6o99z) (merge vers: 5.1.33-ndb-6.3.25) (pib:6)
[9 May 2009 18:41] Bugs System
Pushed into 5.1.34-ndb-7.0.6 (revid:jonas@mysql.com-20090509154927-im9a7g846c6u1hzc) (version source revid:jonas@mysql.com-20090509073226-09bljakh9eppogec) (merge vers: 5.1.33-ndb-7.0.6) (pib:6)