MySQL Bugs: #16510: Repeated crash with 4.0.24 and 4.0.26

Bug #16510	Repeated crash with 4.0.24 and 4.0.26
Submitted:	15 Jan 2006 0:07	Modified:	24 Jan 2006 22:09
Reporter:	[ name withheld ]	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	4.0.26	OS:	Linux (Slackware Linux)
Assigned to:	Evgeny Potemkin	CPU Architecture:	Any

Description:
MySQL 4.0.24 and 4.0.26 crashes on a production (no symbol file, sorry) Slackware MP server (2 Xeons (i386) and HT). Here should be something interesting:

###

[...]
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=838860800
read_buffer_size=2093056
max_used_connections=56
max_connections=450
threads_connected=57
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2660596 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8ab11678
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb97fef48, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8104a71
0xb7f06c85
0x812b14d
0x8144949
0x81127c2
0x8115fb6
0x8110cc7
0x81107c2
0x81100db
0xb7f0154e
0xb7e1db8a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace.
 Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8af7338 = UPDATE `katalog` SET `#vyrobek`='W800i', `+Rozměry_y`='100', `+Rozměry_x`='46', `+Rozměry_z`='20', `#
Rozměry`='9108000', `Hmotnost`='99', `Konstrukce`='klasická', `Zvýšená odolnost`='ne', `Výměnné kryty`='ne', `Konektor externí 
antény`='ano', `GSM sítě`='900 / 1800 Mhz', `3G síť`='ano', `Udávané SAR`='5', `Integrovaná paměť`='', `Paměťové karty`='ne', `
Baterie`='Li-Ion', `**Baterie`='' WHERE `#vyrobek`='W800i'
thd->thread_id=121
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 3
[...]

###

The system has 2G RAM + 2G swap.

The 'UPDATE' query there is allways alike:
# grep -A 5 'query at' /var/log/mysql/mysqld.log
thd->query at 0x52311700  is invalid pointer
thd->thread_id=1374163
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
--
thd->query at 0x93d9658 = UPDATE `katalog` SET `#vyrobek`='W800i', `+Rozměry_y`='100', `+Rozměry_x`='46', `+Rozměry_z`='20', `#Rozměry`='9108000', `Hmotnost`='99', `Konstrukce`='klasická', `Zvýšená odolnost`='ne', `Výměnné kryty`='ne', `Konektor externí antény`='ano', `GSM sítě`='900 / 1800 Mhz', `3G síť`='ano', `Udávané SAR`='0', `Integrovaná paměť`='0', `Paměťové karty`='ne', `Baterie`='Li-Ion', `**Baterie'mAh`='' WHERE `#vyrobek`='W800i'
thd->thread_id=73470
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
--
thd->query at 0x9199c70 = UPDATE `katalog` SET `#vyrobek`='W800i', `+Rozměry_y`='100', `+Rozměry_x`='46', `+Rozměry_z`='20', `#Rozměry`='9108000', `Hmotnost`='99', `Konstrukce`='klasická', `Zvýšená odolnost`='ne', `Výměnné kryty`='ne', `Konektor externí antény`='ano', `GSM sítě`='900 / 1800 / 1900 Mhz', `3G síť`='ne', `Udávané SAR`='0', `Integrovaná paměť`='40', `Paměťové karty`='Memory Stick PRO Duo', `Baterie`='Li-Pol', `**Baterie`='900' WHERE `#vyrobek`='W800i'
thd->thread_id=4787
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 1
--
thd->query at 0x8de7bd0 = UPDATE `katalog` SET `#vyrobek`='W810i', `+Rozměry_y`='10', `+Rozměry_x`='', `+Rozměry_z`='', `#Rozměry`='0', `Hmotnost`='', `Konstrukce`='klasická', `Zvýšená odolnost`='ne', `Výměnné kryty`='ne', `Konektor externí antény`='ano', `GSM sítě`='900 / 1800 Mhz', `3G síť`='ano', `Udávané SAR`='', `Integrovaná paměť`='', `Paměťové karty`='ne', `Baterie`='Li-Ion', `**Baterie`='' WHERE `#vyrobek`='W810i'
thd->thread_id=187791
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
--
thd->query at 0x5746a218  is invalid pointer
thd->thread_id=5180
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
--
thd->query at 0x8af7338 = UPDATE `katalog` SET `#vyrobek`='W800i', `+Rozměry_y`='100', `+Rozměry_x`='46', `+Rozměry_z`='20', `#Rozměry`='9108000', `Hmotnost`='99', `Konstrukce`='klasická', `Zvýšená odolnost`='ne', `Výměnné kryty`='ne', `Konektor externí antény`='ano', `GSM sítě`='900 / 1800 Mhz', `3G síť`='ano', `Udávané SAR`='5', `Integrovaná paměť`='', `Paměťové karty`='ne', `Baterie`='Li-Ion', `**Baterie`='' WHERE `#vyrobek`='W800i'
thd->thread_id=121
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 3
#

I hope that's enough of information, feel free to ask for more otherwise: 0602 at eq.cz

How to repeat:
See 'Description'.

Suggested fix:
Sorry.

Could you please write output of resolve_stack_dump here?  (the description how to get it: http://dev.mysql.com/doc/refman/5.0/en/using-stack-trace.html )

Hi,
I was able to reproduce this on a different machine (MySQL 4.0.26, Slackware Linux, UP, i386); resolved stack trace:

# resolve_stack_dump -s /tmp/mysqld.sym -n /tmp/mysqld.stack
0x8111658 handle_segfault + 456
0xb7f11dfd _end + -1347440707
0x8139abb _Z11fill_recordR4ListI4ItemES2_b + 155
0x81576da _Z12mysql_updateP3THDP13st_table_listR4ListI4ItemES6_PS4_P8st_orderm15enum_duplicates + 1834
0x812249a _Z21mysql_execute_commandv + 4026
0x8125de6 _Z11mysql_parseP3THDPcj + 230
0x8120632 _Z16dispatch_command19enum_server_commandP3THDPcj + 1394
0x811ff94 _Z10do_commandP3THD + 228
0x811f6f8 handle_one_connection + 888
0xb7f0a463 _end + -1347471837
0xb7e3bd64 _end + -1348317404
#

mysqld.log:

mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=2093056
max_used_connections=25
max_connections=40
threads_connected=3
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 294750 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0xa5570fb8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbd7fefa8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8111658
0xb7f11dfd
0x8139abb
0x81576da
0x812249a
0x8125de6
0x8120632
0x811ff94
0x811f6f8
0xb7f0a463
0xb7e3bd64
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace.
 Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8471580 = UPDATE `katalog` SET `#vyrobek`='W800i', `+Rozm<EC>ry_y`='100', `+Rozm<EC>ry_x`='46', `+Rozm<EC>ry_z`
='20', `#Rozm<EC>ry`='9108000', `Hmotnost`='99', `Konstrukce`='klasick<E1>', `Zv<FD><B9>en<E1> odolnost`='ne', `V<FD>m<EC>nn
<E9> kryty`='ne', `Konektor extern<ED> ant<E9>ny`='ano', `GSM s<ED>t<EC>`='900 / 1800 Mhz', `3G s<ED><BB>`='ano', `Ud<E1>van
<E9> SAR`='5', `Integrovan<E1> pam<EC><BB>`='', `Pam<EC><BB>ov<E9> karty`='ne', `Baterie`='Li-Ion', `**Baterie`='' WHERE `#vyro
bek`='W800i'
thd->thread_id=3342
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0

I'm able to provide a test case which can be used to debug this crash. It's a table structure with one row and the 'UPDATE' query. If you are interested, email me privately and I'll answer with two compressed files (structure + one row of data and the query) as attachment.

Please, upload your test case (you may mark it as private) using the File tab. At least, the SHOW CREATE TABLE results for the table you are trying to UPDATE are absolutely needed.

Ok, I've uploaded the files.

verified against 4.0.26

got:

0x80f9b5b handle_segfault + 399
0x4005d715 _end + 936516237
0x811de08 _Z11fill_recordR4ListI4ItemES2_b + 80
0x81357ca _Z12mysql_updateP3THDP13st_table_listR4ListI4ItemES6_PS4_P8st_orderm15
enum_duplicates + 1390
0x8107ab6 _Z21mysql_execute_commandv + 3882
0x810b51d _Z11mysql_parseP3THDPcj + 329
0x8105f77 _Z16dispatch_command19enum_server_commandP3THDPcj + 1091
0x8105af8 _Z10do_commandP3THD + 100
0x810544d handle_one_connection + 841
0x400584eb _end + 936495203
0x401d3b0a _end + 938049154

The reported UPDATE statement tries to update field with name like '*name'. mysqld
sees '*' in first character of field name and wrongly expanded it to the list of all table fields.
Due to this, list of the fields becomes longer than list of the values, which leads to NULL dereferencing in fill_record() and crash.

Hi,

could you please point me to a patch as soon as it's cooked (cvsweb?)? (There is no need for a patch if 4.0.27 will be out in a week or so ...)

Thanks,

r.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/1520

Fixed in 4.0.27, cset 1.2172

Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Documented bugfix in 4.0.27 changelog. Closed.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/1628