Bug #22778 mysql server randomly crashed
Submitted: 28 Sep 2006 11:59 Modified: 27 Jan 2009 6:49
Reporter: Corin Langosch Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.0.26, 5.0.24a OS:Linux (linux debian amd64)
Assigned to: CPU Architecture:Any
Tags: crash, innodb_locks_unsafe_for_binlog, server

[28 Sep 2006 11:59] Corin Langosch
Description:
we ware running mysql 5.0.24a on a dual opteron unter debian amd64. the server is used by some big sites, mysql shows up to 3.000 - 4.000 queries/sec in peak times. unluckily the server randomly crahes, sometimes after some hours sometimes after 1-2 days. to find the reason we run the debug version (the others (x86 static and amd64) crash too). 

here is the log output:
--
060928  2:12:24 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.24a-debug'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Edition - Debug (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=2093056
max_used_connections=774
max_connections=800
threads_connected=764
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 62329 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

You seem to be running 32-bit Linux and have 764 concurrent connections.
If you have not changed STACK_SIZE in LinuxThreads and built the binary
yourself, LinuxThreads is quite likely to steal a part of the global heap for
the thread stack. Please read http://www.mysql.com/doc/en/Linux.html

thd=0x566c6b48
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xff01e938, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80aab62
0x8342708
0x8164461
0x80a60f0
0x80a53f8
0x80e40ff
0x811691b
0x80bfb57
0x80c56f2
0x80bc548
0x80bbe65
0x80bb144
0x833febc
0x8387cda
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x558bdd18  is invalid pointer
thd->thread_id=3632919
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
Fatal signal 6 while backtracing

Number of processes running now: 0
060928 13:47:27  mysqld restarted
--

How to repeat:
we cannot repeat it, it just happens randomly. 

Suggested fix:
if the output isn't enough, please tell me how to make a helpfull trace. the server is used in a production environment, so the stuff needed for a good helpfull trace should not affect the behaviour or degrade performance too much.
[28 Sep 2006 12:04] Corin Langosch
here is the resolved stack trace:
--
0x80aab62 handle_segfault + 426
0x8342708 pthread_sighandler + 184
0x8164461 store_lock__11ha_innobaseP3THDPP16st_thr_lock_data13thr_lock_type + 89
0x80a60f0 get_lock_data__FP3THDPP8st_tableUiUiT1 + 512
0x80a53f8 mysql_lock_tables__FP3THDPP8st_tableUiUiPb + 488
0x80e40ff lock_tables__FP3THDP13st_table_listUiPb + 407
0x811691b mysql_update__FP3THDP13st_table_listRt4List1Z4ItemT2P4ItemUiP8st_orderUx15enum_duplicatesb + 411
0x80bfb57 mysql_execute_command__FP3THD + 8315
0x80c56f2 mysql_parse__FP3THDPcUi + 350
0x80bc548 dispatch_command__F19enum_server_commandP3THDPcUi + 1752
0x80bbe65 do_command__FP3THD + 457
0x80bb144 handle_one_connection + 844
0x833febc pthread_start_thread + 220
0x8387cda thread_start + 4
[28 Sep 2006 12:50] Valeriy Kravchuk
Thank you for a problem report. Please, send your my.cnf used and the results of:

file mysqld
uname -a
free

commands. Looks like out of memory issue to me, not a bug.
[28 Sep 2006 13:19] Corin Langosch
mysqld: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.0.0, statically linked, for GNU/Linux 2.0.0, not stripped

Linux db1 2.6.18-1-amd64 #1 SMP Sun Sep 24 15:36:09 CEST 2006 x86_64 GNU/Linux

             total       used       free     shared    buffers     cached
Mem:       4062132    4047212      14920          0      31948    1626964
-/+ buffers/cache:    2388300    1673832
Swap:      3196892         52    3196840

NOTES:
we just installed the new kernel yesterday as we hoped the crash was system dependend and not a bug of mysql. before the hardware ran 2.6.15 and had an uptime of 101 days, but mysql crashed also. mysql sql started crashing when we switched from 4.1 to 5.0. if it really is a memory issue - shouldn't mysql report this instead of just crashing? (especially the debug build)
[28 Sep 2006 13:20] Corin Langosch
my.cnf:
--
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]

# general stuff
user                    = mysql
pid-file                = /var/run/mysqld/mysqld.pid
socket                  = /var/run/mysqld/mysqld.sock
port                    = 3306
basedir                 = /usr/local/mysql
tmpdir                  = /tmp
language                = /usr/local/mysql/share/mysql/english
datadir                 = /data-drive/mysql
server-id               = 1
character-set-server    = latin1
collation-server        = latin1_german1_ci
transaction_isolation   = READ-COMMITTED
low-priority-updates
skip-external-locking
skip-name-resolve
skip-locking
skip-grant-tables
#memlock

# performance options
key_buffer                      =       256M
max_allowed_packet              =       16M
table_cache                     =       2048
sort_buffer                     =       8M
read_buffer                     =       2M
read_rnd_buffer_size            =       16M
bulk_insert_buffer_size         =       64M
myisam_sort_buffer_size         =       128M       # used by alter table etc..
myisam_max_sort_file_size       =       10G
myisam_max_extra_sort_file_size =       10G
thread_cache                    =       64
wait_timeout                    =       28800
connect_timeout                 =       5
thread_concurrency              =       8
max_connections                 =       800
query_cache_size                =       0M
query_cache_type                =       0
delay-key-write                 =       OFF
tmp_table_size                  =       64M

# *** INNODB Specific options ***
innodb_additional_mem_pool_size =       16M
innodb_buffer_pool_size         =       2000M
innodb_data_file_path           =       ibdata1:10M:autoextend
innodb_thread_concurrency       =       16
innodb_flush_log_at_trx_commit  =       0
innodb_log_buffer_size          =       16M
innodb_log_file_size            =       256M
innodb_log_files_in_group       =       3
innodb_log_group_home_dir       =       /mysql-tmp
innodb_max_dirty_pages_pct      =       90
innodb_lock_wait_timeout        =       10
innodb_locks_unsafe_for_binlog

# some misc stuff
skip-bdb

# the following options will be passed to all MySQL clients
[client]
port            = 3306
socket          = /tmp/mysql.sock

[mysqldump]
quick
max_allowed_packet      = 16M

[myisamchk]
key_buffer      =       384M
sort_buffer     =       384M
read_buffer     =       8M
write_buffer    =       8M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
open-files-limit = 8192
[28 Sep 2006 18:02] Valeriy Kravchuk
How many concurrent connections do you really have? Up to:

max_connections                 =       800

really? Please, check with SHOW GLOBAL STATUS. But even with 100 you have obvious out of memory problem here.
[29 Sep 2006 6:49] Corin Langosch
"SHOW GLOBAL STATUS" reports "Max_used_connections 464". Again i wonder why the system does not swap nor print a warning / fatal error, if it really is a out of memory issue.
[29 Sep 2006 6:54] Corin Langosch
by the way, just noticed:
--
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 62329
K bytes of memory
--
shouldn't this be ... M (not K) bytes of memory?
[30 Sep 2006 10:12] Corin Langosch
today mysql crashed again two times in the night - once at 00:53 and once 02:48. both stacktraces are exactly the same as the one i already sent. afer the second crash it's now running 10 hours again without any crash (nothing changed at the system, mysql always recovers by itself).
[10 Oct 2006 8:32] Corin Langosch
i just installed the new 5.0.26 binary, but it crashes too :-(

061009 20:56:46 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.26-standard'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Edition - Standard (GPL)

0x80a0a12 handle_segfault + 430
0x82fb4e8 pthread_sighandler + 184
0x81450df store_lock__11ha_innobaseP3THDPP16st_thr_lock_data13thr_lock_type + 95
0x809d224 get_lock_data__FP3THDPP8st_tableUiUiT1 + 444
0x809c904 mysql_lock_tables__FP3THDPP8st_tableUiUiPb + 452
0x80d6560 lock_tables__FP3THDP13st_table_listUiPb + 196
0x80d6392 open_and_lock_tables__FP3THDP13st_table_list + 78
0x80fa54d mysql_insert__FP3THDP13st_table_listRt4List1Z4ItemRt4List1Zt4List1Z4ItemT2T215enum_duplicatesb + 441
0x80b4332 mysql_execute_command__FP3THD + 7966
0x80b944a mysql_parse__FP3THDPcUi + 286
0x80b0e54 dispatch_command__F19enum_server_commandP3THDPcUi + 1876
0x80b06f3 do_command__FP3THD + 195
0x80afc84 handle_one_connection + 764
0x82f8c9c pthread_start_thread + 220
0x83412ea thread_start + 4
[18 Oct 2006 12:33] Corin Langosch
have you already come any further with this bug? our mysql keeps crashing every 1-3 days which is really bad and annoying. if i can provide you with any help so this bug gets fixed asap, please let me know! :)
[18 Oct 2006 14:35] Valeriy Kravchuk
As I already explained, it is out of memory problem, almost surely. You have to decrease several values in my.cnf. But I can not give you more details in frames of bug report. I am checking your stack traces, although.
[18 Oct 2006 14:46] Corin Langosch
thank's for your reply- but i do not believe it is a out-of-memroy issue.

1. the debug build crashes also without any warning nor error message. doesnt the doc say it has a special memory allocator for safety checks?

2. it crashes always at EXACTLY the same position, but i think memory is allocated at a lot of different places...

3. the system does not swap, but swapping is enabled and memlock is disabled in mysql. so shouldn't the system swap before mysql crashes?

look at the latest error message, mysql only used 116 this time:
--
key_buffer_size=268435456
read_buffer_size=2093056
max_used_connections=116
max_connections=800
threads_connected=15
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 62329 K
bytes of memory

the error with the K bytes of memory (should be M bytes of memory) is still in there too. otherwise my mysql woudl only use up to 62329 K bytes = ~62 M bytes of memory.
[18 Oct 2006 15:00] Chad MILLER
I know little of this, but I'm just curious:  

corinl: As the user running the server, run 'ulimit -aS' and 'ulimit -aH' and paste the results, please.
[18 Oct 2006 15:03] Corin Langosch
in my start script i have:

# set some new limits
ulimit -n 8192
ulimit -a

so the resulsts are:

db1:/usr/local/bin# ulimit -aS
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) unlimited
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

db1:/usr/local/bin# ulimit -aH
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
max nice                        (-e) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) unlimited
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[26 Oct 2006 10:43] Corin Langosch
hello!
have you come any further with the stacktraces yet? :)

i decreased a lot of variables, but mysql keeps crashing:

key_buffer_size=134217728
read_buffer_size=2093056
max_used_connections=197
max_connections=200
threads_connected=37
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1359070 K
bytes of memory

i now further decreased my variables, but i think it won't help anyway, because i have 4 GB ram and now it alerady uses only 1,4 GB...and still no swapping.
[26 Oct 2006 12:23] Valeriy Kravchuk
What is your current value of innodb_buffer_pool_size? 2000M was too high for a 32-bit Linux. Use 1000M instead.
[26 Oct 2006 12:47] Corin Langosch
the system is a dual opteron amd64 system with 64bit kernel and 32-bit emulation. i already ran your precompiled amd64 binaries, but these crashed even more often so i switched to the 686 binaries. i'll now install the amd64 binaries again and look how they run...
[26 Oct 2006 13:00] Corin Langosch
oh, the amd64 binary crashed already just after about 5 minutes uptime :-(

--
061026 14:52:23  InnoDB: Started; log sequence number 115 1374154164
061026 14:52:23 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.26-standard'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Edition - Standard (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=1044480
max_used_connections=146
max_connections=200
threads_connected=35
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1154270 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2aaab2fb7970
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x4655a128, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2d2da00 = INSERT  INTO hits_week (domain,year,week,hits) VALUES ('www.gimy.de',YEAR(CURDATE()),WEEK(CURDATE()),1) ON DUPLICATE KEY UPDATE hits=hits+1
thd->thread_id=45373
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
061026 14:58:21  mysqld restarted
061026 14:58:21 [Warning] options --log-slow-admin-statements and --log-queries-not-using-indexes have no effect if --log-slow-queries is not set
061026 14:58:22  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
061026 14:58:22  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 115 1374320439.
InnoDB: Doing recovery: scanned up to log sequence number 115 1379563008
InnoDB: Doing recovery: scanned up to log sequence number 115 1384805888
InnoDB: Doing recovery: scanned up to log sequence number 115 1385844971
061026 14:58:23  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
061026 14:58:44  InnoDB: Started; log sequence number 115 1385844971
061026 14:58:44 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.26-standard'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Edition - Standard (GPL)
[26 Oct 2006 13:02] Corin Langosch
basedir  	/usr/local/mysql-standard-5.0.26-linux-x86_64-glibc23/
version  	5.0.26-standard
version comment 	MySQL Community Edition - Standard (GPL)
version compile machine 	x86_64
version compile os 	unknown-linux-gnu
[26 Oct 2006 13:04] Corin Langosch
and mysql crashed and again... :-(
i switch back to the 686 binary....

--
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=1044480
max_used_connections=46
max_connections=200
threads_connected=24
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1154270 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2aaaaad7f6d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x444da128, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2aaaaaf7f230  is invalid pointer
thd->thread_id=13288
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
061026 15:02:56  mysqld restarted
061026 15:02:56 [Warning] options --log-slow-admin-statements and --log-queries-not-using-indexes have no effect if --log-slow-queries is not set
061026 15:02:56  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
061026 15:02:56  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 115 1374323831.
InnoDB: Doing recovery: scanned up to log sequence number 115 1379566592
InnoDB: Doing recovery: scanned up to log sequence number 115 1384809472
InnoDB: Doing recovery: scanned up to log sequence number 115 1390052352
InnoDB: Doing recovery: scanned up to log sequence number 115 1392557710
061026 15:02:58  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
061026 15:03:25  InnoDB: Started; log sequence number 115 1392557710
061026 15:03:25 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.26-standard'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Edition - Standard (GPL)
[30 Oct 2006 18:13] Corin Langosch
have you come any further with the stack traces, better said, did you already had a look at them? mysql keeps crashing, although i decreased the server vars a lot and there is plenty of free memory as you can see from my supplied data.

i kindly ask you, to take this problem a bit more serious than you did the last weeks. thank you in advance.

system:
--
top - 19:12:53 up 32 days, 18:01,  1 user,  load average: 0.97, 1.14, 1.23
Tasks: 136 total,   4 running, 132 sleeping,   0 stopped,   0 zombie
Cpu0  : 25.3%us,  5.2%sy,  0.0%ni, 64.4%id,  2.3%wa,  0.2%hi,  2.5%si,  0.0%st
Cpu1  : 11.0%us,  2.3%sy,  0.0%ni, 79.9%id,  6.4%wa,  0.1%hi,  0.3%si,  0.0%st
Mem:   4062132k total,  4048804k used,    13328k free,    20568k buffers
Swap:  3196892k total,       52k used,  3196840k free,  1816132k cached
--

crash:
---- 
061030 12:58:32  mysqld started
061030 12:58:32 [Warning] options --log-slow-admin-statements and --log-queries-                                                                             not-using-indexes have no effect if --log-slow-queries is not set
061030 12:58:33  InnoDB: Started; log sequence number 120 2727470453
061030 12:58:33 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.26-standard'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  M                                                                             ySQL Community Edition - Standard (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=67108864
read_buffer_size=1044480
max_used_connections=179
max_connections=250
threads_connected=25
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 108853                                                                             4 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x7322d260
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xff49e958, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80a0a12
0x82fb4e8
0x81450df
0x809d224
0x809c904
0x80d6560
0x80d6392
0x80fa54d
0x80b4332
0x80b944a
0x80b0e54
0x80b06f3
0x80afc84
0x82f8c9c
0x83412ea
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow                                                                              instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9b0a8f0 = INSERT  INTO hits_week (domain,year,week,hits) VALUES                                                                              ('www.yakbeni.com',YEAR(CURDATE()),WEEK(CURDATE()),1) ON DUPLICATE KEY UPDATE hi                                                                             ts=hits+1
thd->thread_id=3429134
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
061030 18:42:03  mysqld restarted
061030 18:42:03 [Warning] options --log-slow-admin-statements and --log-queries-                                                                             not-using-indexes have no effect if --log-slow-queries is not set
061030 18:42:04  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
061030 18:42:04  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 120 3308845099.
InnoDB: Doing recovery: scanned up to log sequence number 120 3314087936
InnoDB: Doing recovery: scanned up to log sequence number 120 3319330816
InnoDB: Doing recovery: scanned up to log sequence number 120 3324573696
InnoDB: Doing recovery: scanned up to log sequence number 120 3329816576
InnoDB: Doing recovery: scanned up to log sequence number 120 3335059456
InnoDB: Doing recovery: scanned up to log sequence number 120 3340302336
InnoDB: Doing recovery: scanned up to log sequence number 120 3345545216
InnoDB: Doing recovery: scanned up to log sequence number 120 3350788096
InnoDB: Doing recovery: scanned up to log sequence number 120 3356030976
InnoDB: Doing recovery: scanned up to log sequence number 120 3361273856
InnoDB: Doing recovery: scanned up to log sequence number 120 3366516736
InnoDB: Doing recovery: scanned up to log sequence number 120 3371759616
InnoDB: Doing recovery: scanned up to log sequence number 120 3377002496
InnoDB: Doing recovery: scanned up to log sequence number 120 3382245376
InnoDB: Doing recovery: scanned up to log sequence number 120 3387488256
InnoDB: Doing recovery: scanned up to log sequence number 120 3392731136
InnoDB: Doing recovery: scanned up to log sequence number 120 3397974016
InnoDB: Doing recovery: scanned up to log sequence number 120 3403216896
InnoDB: Doing recovery: scanned up to log sequence number 120 3408459776
InnoDB: Doing recovery: scanned up to log sequence number 120 3413702656
InnoDB: Doing recovery: scanned up to log sequence number 120 3418945536
InnoDB: Doing recovery: scanned up to log sequence number 120 3424188416
InnoDB: Doing recovery: scanned up to log sequence number 120 3429431296
InnoDB: Doing recovery: scanned up to log sequence number 120 3434674176
InnoDB: Doing recovery: scanned up to log sequence number 120 3439917056
InnoDB: Doing recovery: scanned up to log sequence number 120 3445159936
InnoDB: Doing recovery: scanned up to log sequence number 120 3450402816
InnoDB: Doing recovery: scanned up to log sequence number 120 3455645696
InnoDB: Doing recovery: scanned up to log sequence number 120 3460888576
InnoDB: Doing recovery: scanned up to log sequence number 120 3466131456
InnoDB: Doing recovery: scanned up to log sequence number 120 3471374336
InnoDB: Doing recovery: scanned up to log sequence number 120 3476617216
InnoDB: Doing recovery: scanned up to log sequence number 120 3480772046
061030 18:42:24  InnoDB: Starting an apply batch of log records to the database.                                                                             ..
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19                                                                              20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46                                                                              47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 7                                                                             3 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
061030 18:44:43  InnoDB: Started; log sequence number 120 3480772046
061030 18:44:43 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.26-standard'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  M                                                                             ySQL Community Edition - Standard (GPL)
----
[1 Nov 2006 7:37] MySQL Verification Team
Crashes seem isolated to the `hits_week` table.  What is output of:
SHOW CREATE TABLE `hits_week`\G
CHECK TABLE `hits_week`;
SHOW TABLE STATUS LIKE 'hits_week'\G

Are there triggers on this table?  If yes, pleased upload them (in private section) if needed.
[1 Nov 2006 10:02] Corin Langosch
Hi!

Here are the requested outputs:

SHOW CREATE TABLE `hits_week`;
--
CREATE TABLE `hits_week` (
  `domain` varchar(100) collate latin1_german1_ci NOT NULL default '',
  `year` mediumint(8) unsigned NOT NULL default '0',
  `week` tinyint(3) unsigned NOT NULL default '0',
  `hits` int(10) unsigned NOT NULL default '0',
  PRIMARY KEY  (`domain`,`year`,`week`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci 
--

CHECK TABLE `hits_week`;
--
+----------------------+-------+----------+----------+
| Table                | Op    | Msg_type | Msg_text |
+----------------------+-------+----------+----------+
| my_ranking.hits_week | check | status   | OK       |
+----------------------+-------+----------+----------+
1 row in set (12.81 sec)
--

SHOW TABLE STATUS LIKE 'hits_week';
--
Name      | Engine | Version | Row_format | Rows   | Avg_row_length
hits_week | InnoDB | 10      | Compact    | 213852 | 86            

Data_length | Max_data_length | Index_length | 
18415616    | 0               |	0            | 	

Data_free | Auto_increment | Create_time | Update_time 
0         |	NULL           | 2006-07-25  | 09:04:32    
	
	
Check_time | Collation | Checksum          |	Create_options | Comment
NULL       | NULL      | latin1_german1_ci |	NULL           | InnoDB free: 4268032 kB
--

I nowhere use triggers. I do use a lot of "INSERT ... ON DUPLICATE KEY ...", may be the problem is related to this.
[1 Nov 2006 10:10] Corin Langosch
sorry, i malformated the outputa bit. it should be:

--
Data_free | Auto_increment | Create_time         | Update_time 
0         |	NULL           | 2006-07-25 09:04:32 | NULL
	
	
Check_time | Collation         | Checksum |	Create_options | Comment
NULL       | latin1_german1_ci |	NULL    |                | InnoDB free: 4268032 kB
--
[1 Nov 2006 19:23] Corin Langosch
hm, may be it's not related to the query above...mysql just crashed again, this time a different sql is given. but the backtrace (crash point) is again exactly the same as all the times before.

--
Hope that's ok; if not, decrease some variables in the equation.

thd=0x6f761960
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xff27eb18, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80a0a12
0x82fb4e8
0x81450df
0x809d224
0x809c904
0x80d6560
0x8105df9
0x80b415a
0x80b944a
0x80b0e54
0x80b06f3
0x80afc84
0x82f8c9c
0x83412ea
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow                                                                              instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x978e348 = UPDATE mail_l AS ml,mail_folders AS s_f,mail_folders A                                                                             S d_f,mail_mails AS mm SET ml.folder_id=93807,ml.expire_stamp=IF(d_f.keep=0,0,mm                                                                             .create_stamp+d_f.keep*3600*24),s_f.num_new=s_f.num_new-IF(FIND_IN_SET('red',ml.                                                                             flags)>0,0,1),s_f.num_total=s_f.num_total-1,d_f.num_total=d_f.num_total+1,d_f.nu                                                                             m_new=d_f.num_new+IF(FIND_IN_SET('red',ml.flags)>0,0,1) WHERE ml.mail_id=2134432                                                                              AND ml.folder_id=32262 AND s_f.owner_id=20714 AND d_f.owner_id=20714 AND s_f.id                                                                             =ml.folder_id AND d_f.id=93807 AND mm.id=ml.mail_id
thd->thread_id=15086016
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
Fatal signal 6 while backtracing

Number of processes running now: 0
061101 19:57:54  mysqld restarted
[3 Nov 2006 9:47] Corin Langosch
to track this bug down finally i ran mysql '5.0.27-debug' in gdb. i really hope this helps in fixing this bug asap. 

here is the output:
----
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1851390896 (LWP 20351)]
0x08241395 in ha_innobase::store_lock (this=0xaa6a068, thd=0x71478cc8,
    to=0xa44ba00, lock_type=TL_WRITE) at ha_innodb.cc:6595
6595    ha_innodb.cc: Datei oder Verzeichnis nicht gefunden.
        in ha_innodb.cc
#0  0x08241395 in ha_innobase::store_lock (this=0xaa6a068, thd=0x71478cc8,
    to=0xa44ba00, lock_type=TL_WRITE) at ha_innodb.cc:6595
        prebuilt = (row_prebuilt_t *) 0xf71f6a68
#1  0x0817a25f in get_lock_data (thd=0x71478cc8, table_ptr=0xae0b688, count=1,
    flags=2, write_lock_used=0x6e59e5b8) at lock.cc:719
        table = (TABLE *) 0xaa69820
        lock_type = -148936088
        org_locks = (THR_LOCK_DATA **) 0xa44ba00
        i = 0
        tables = 172276224
---Type <return> to continue, or q <return> to quit---
        lock_count = 178690080
        sql_lock = (MYSQL_LOCK *) 0xa44b9f0
        locks = (THR_LOCK_DATA **) 0xa44ba00
        locks_buf = (THR_LOCK_DATA **) 0xa44ba00
        to = (TABLE **) 0xa44ba08
        table_buf = (TABLE **) 0xa44ba08
        _db_func_ = 0x3d <Address 0x3d out of bounds>
        _db_file_ = 0x6e59e574 "ðåYn¸åYn"
        _db_level_ = 1851385208
        _db_framep_ = (char **) 0x6e59e57c
#2  0x081793ae in mysql_lock_tables (thd=0x71478cc8, tables=0xae0b688,
    count=1, flags=4, need_reopen=0x6e59e64b) at lock.cc:126
        sql_lock = (MYSQL_LOCK *) 0x0
        write_lock_used = (TABLE *) 0xaa69820
        rc = 1832908472
        _db_func_ = 0x6e59e5d4 "È\214Gq(æYn8\230\033\bÈ\214Gq\210¶à\n\001"
        _db_file_ = 0x8e9 <Address 0x8e9 out of bounds>
        _db_level_ = 1851385312
        _db_framep_ = (char **) 0x71478ce8
#3  0x081b9838 in lock_tables (thd=0x71478cc8, tables=0xae0ab60,
    count=1832908472, need_reopen=0x6e59e64b) at sql_base.cc:2597
        start = (TABLE **) 0xae0b688
        ptr = (TABLE **) 0xf71f6a68
---Type <return> to continue, or q <return> to quit---
        table = (TABLE_LIST *) 0x0
        _db_func_ = 0x0
        _db_file_ = 0x71478cc8 "è\200O\bÈi`\bPâ\201nü\200O\bеà\nè\214Gq"
        _db_level_ = 182496096
        _db_framep_ = (char **) 0x6e59e668
#4  0x081b9479 in open_and_lock_tables (thd=0x71478cc8, tables=0xae0ab60)
    at sql_base.cc:2452
        counter = 1
        need_reopen = false
        _db_func_ = 0x0
        _db_file_ = 0xae0ab60 ""
        _db_level_ = 1851385976
        _db_framep_ = (char **) 0x81e265b
#5  0x081e27b5 in mysql_insert (thd=0x71478cc8, table_list=0xae0ab60,
    fields=@0x714791f8, values_list=@0x7147921c, update_fields=@0x71479210,
    update_values=@0x71479204, duplic=DUP_UPDATE, ignore=false)
    at sql_insert.cc:397
        error = 182495832
        res = 0
        log_on = true
        transactional_table = 96
        joins_freed = false
        changed = 223
---Type <return> to continue, or q <return> to quit---
        value_count = 1
        counter = 1
        id = 7951640743573697624
        info = {records = 5398914429184, deleted = 593300900015630408,
  updated = 7951663350572675328, copied = 593299704345832288,
  error_count = 8162647640253661185, handle_duplicates = 1851385992,
  escape_char = 1, last_errno = 1900514504, ignore = 200,
  update_fields = 0x6e59e888, update_values = 0x817b02b, view = 0x4ea}
        table = (TABLE *) 0x0
        its = {<base_list_iterator> = {list = 0x7147921c, el = 0x7147921c,
    prev = 0x0, current = 0x0}, <No data fields>}
        values = (List_item *) 0x0
        context = (Name_resolution_context *) 0xae0ab60
        ctx_state = {save_table_list = 0x71478cc8,
  save_first_name_resolution_table = 0x6,
  save_next_name_resolution_table = 0xae0ab60,
  save_resolve_in_select_list = false, save_next_local = 0x1}
        query = 0xae0aa50 "INSERT  INTO hits_week (domain,year,week,hits) VALUES ('www.yakbeni.com',YEAR(CURDATE()),WEEK(CURDATE()),1) ON DUPLICATE KEY UPDATE hits=hits+1"
        lock_type = TL_WRITE
        unused_conds = (class Item *) 0x0
        _db_func_ = 0xae0ab38 ""
---Type <return> to continue, or q <return> to quit---
        _db_file_ = 0x9 <Address 0x9 out of bounds>
        _db_level_ = 4
        _db_framep_ = (char **) 0x71478cc8
#6  0x08196bf3 in mysql_execute_command (thd=0x71478cc8) at sql_parse.cc:3369
        res = false
        need_start_waiting = true
        result = 0
        lex = (LEX *) 0x71478d08
        select_lex = (SELECT_LEX *) 0xae0ab60
        first_table = (TABLE_LIST *) 0xae0ab60
        all_tables = (TABLE_LIST *) 0xae0ab60
        unit = (SELECT_LEX_UNIT *) 0x71478d6c
        _db_func_ = 0x0
        _db_file_ = 0x0
        _db_level_ = 140531568
        _db_framep_ = (char **) 0x6e59e948
#7  0x0819bd5b in mysql_parse (thd=0x71478cc8,
    inBuf=0xae0aa50 "INSERT  INTO hits_week (domain,year,week,hits) VALUES ('www.yakbeni.com',YEAR(CURDATE()),WEEK(CURDATE()),1) ON DUPLICATE KEY UPDATE hits=hits+1", length=1900514568) at sql_parse.cc:5870
        lex = (LEX *) 0x71478d08
        _db_func_ = 0x83bd716 "ÉÃU\211å\203ì\f\215EüP\215EøPÿu\bèÂbÔÿ\203Ä\020ºÿÿÿÿ\205Àu1\203=دP\b"
---Type <return> to continue, or q <return> to quit---
        _db_file_ = 0x6e59fbb0 "°ûYn49Ïm°ûYn\001"
        _db_level_ = 0
        _db_framep_ = (char **) 0x6e59edc4
#8  0x08193796 in dispatch_command (command=1900514504, thd=0x71478cc8,
    packet=0x7140ac49 "INSERT  INTO hits_week (domain,year,week,hits) VALUES ('www.yakbeni.com',YEAR(CURDATE()),WEEK(CURDATE()),1) ON DUPLICATE KEY UPDATE hits=hits+1", packet_length=144) at sql_parse.cc:1766
        packet_end = 0xae0aadf ""
        net = (NET *) 0x714794ec
        error = false
        _db_func_ = 0x0
        _db_file_ = 0x0
        _db_level_ = 0
        _db_framep_ = (char **) 0x0
#9  0x08193137 in do_command (thd=0x71478cc8) at sql_parse.cc:1550
        packet = 0x7140ac48 "\003INSERT  INTO hits_week (domain,year,week,hits) VALUES ('www.yakbeni.com',YEAR(CURDATE()),WEEK(CURDATE()),1) ON DUPLICATE KEY UPDATE hits=hits+1"
        old_timeout = 30
        packet_length = 144
        net = (NET *) 0x714794ec
        command = COM_QUERY
        _db_func_ = 0x816f4ca "\203Ä\020Ç\203\224\021"
---Type <return> to continue, or q <return> to quit---
        _db_file_ = 0x71479ef4 "\230ÎÙ\n"
        _db_level_ = 8192
        _db_framep_ = (char **) 0x2000
#10 0x08192496 in handle_one_connection (arg=0x6d3ff6b8) at sql_parse.cc:1181
        error = 1832908472
        net = (NET *) 0x714794ec
        sctx = (Security_context *) 0x71479cbc
        thd = (class THD *) 0x71478cc8
        launch_time = 1832908472
        set = {__val = {0 <repeats 32 times>}}
#11 0xf7f85fbc in start_thread () from /lib32/libpthread.so.0
No symbol table info available.
#12 0xf7eb270e in clone () from /lib32/libc.so.6
No symbol table info available.
(gdb)
[14 Nov 2006 10:37] Corin Langosch
tonight i installed our server completely from scratch. this means: i made a database dump (mysqldump), formated all discs, installed new os (debian stable using 2.6.8 kernel) from scratch, imported the whole database using the dump (not copying datafiles).

i then started the official amd64 binany and...mysql crashed just about after 5-10 minutes. :-(

so far so bad, but here comes the good news:

i then played again a lot with the config. after i outcommented "innodb_locks_unsafe_for_binlog" (set it to off), the server keeps running for about 5 hours without crashing so far now!!

so my conclusion is: there is a grave bug with "innodb_locks_unsafe_for_binlog" in mysql. i'll keep you informed, if mysql keeps running stable with this setting disabled for the next few days, which means there is a bug with this setting for sure. if you need my help in fixing this bug, please feel free to contact me.
[17 Nov 2006 9:34] Corin Langosch
with innodb_locks_unsafe_for_binlog set to off, mysql is now running for 3 days without any problems. so it was never an out of memory problem, but a bug (which still exists) in mysql when this setting is turned on. may be this setting should be marked experimental ;-)
[21 Nov 2006 12:33] Valeriy Kravchuk
Yes, it really looks like a bug now. I'll try to create small and repeatable test case for it.
[1 Dec 2006 22:27] Bruno Kilian
Hello Corin/Valeriy

i got the same problem, i`m working on that for a week.

We are migrating your database SQL Server to MySQL, but MySQL keep crashing after a while working.

------------------------------

mysql error:

061201 15:37:31 [Note] mysqld: ready for connections.
Version: '5.0.27-standard'  socket: '/tmp/mysql.sock'  port: 53306  MySQL Community Edition - Standard (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=536870912
read_buffer_size=2093056
max_used_connections=101
max_connections=200
threads_connected=64
It is possible that mysqld could use up to 
536870912 + (2093056 + 2093056)*200 = 1342686 Kbytes of memory
                                      1374093 k
Hope that's ok; if not, decrease some variables in the equation.

thd=0x948aef28
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x92a98e4c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x816e95c
0x823fd6f
0x80f3ad0
0x827851b
0x8280256
0x827ba3e
0x827ae7b
0x827b9ff
0x82791c9
0x827d5d7
0x81143df
0x8114452
0x81160b8
0x80f3c81
0x81adb51
0x81d86d3
0x81d8591
0x81d022a
0x81d1c64
0x81d22fc
0x818b6ce
0x827b0e1
0x827ae7b
0x827e58c
0x82791c9
0x827cc96
0x8287fc8
0x81d4b3d
0x81d85ef
0x81d022a
0x81d1c64
0x81d22fc
0x818b6ce
0x827b0e1
0x827ae7b
0x827e58c
0x82791c9
0x827d1eb
0x818b3b8
0x818cb10
0x818d8f1
0x818e238
0x818e95c
0x870371
0x7b8ffe
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa4acbf0 = INSERT INTO COOL.USUARIOS_HOST
	(
		`ID_USUARIO`,
		`STR_LOGIN`,
		`INT_CELULAR`,
		`STR_HOST`,
		`INT_STATUS`
	)
	SELECT
		NEW.ID,
		NEW.STR_LOGIN,
		NEW.INT_CELULAR,
		COOL.FN_GET_PARAMETER_VALUE( 'LOCALHOST' ),
		0
thd->thread_id=29033
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

------------------------------

mysql error 2: 

061201 18:47:10 [Note] mysqld: ready for connections.
Version: '5.0.27-standard'  socket: '/tmp/mysql.sock'  port: 53306  MySQL Community Edition - Standard (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=536870912
read_buffer_size=2093056
max_used_connections=101
max_connections=100
threads_connected=63
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 933487 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x92f5c458
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x8e536e4c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x816e95c
0x823fd6f
0x80f3ad0
0x827851b
0x8280256
0x827ba3e
0x827ae7b
0x827b9ff
0x82791c9
0x827d5d7
0x81143df
0x8114452
0x81160b8
0x80f3c81
0x81adb51
0x81d86d3
0x81d8591
0x81d022a
0x81d1c64
0x81d22fc
0x818b6ce
0x827b0e1
0x827ae7b
0x827e58c
0x82791c9
0x827cc96
0x8287fc8
0x81d4b3d
0x81d85ef
0x81d022a
0x81d1c64
0x81d22fc
0x818b6ce
0x827b0e1
0x827ae7b
0x827e58c
0x82791c9
0x827d1eb
0x818b3b8
0x818cb10
0x818d8f1
0x818e238
0x818e95c
0x870371
0x7b8ffe
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8dfbdf78  is invalid pointer
thd->thread_id=1015
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
061201 18:51:34 [Note] mysqld: ready for connections.
Version: '5.0.27-standard'  socket: '/tmp/mysql.sock'  port: 53306  MySQL Community Edition - Standard (GPL)
[1 Dec 2006 22:28] Bruno Kilian
#file /usr/sbin/mysqld:

/usr/sbin/mysqld: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped

#free
             total       used       free     shared    buffers     cached
Mem:       1213408    1093644     119764          0      14540     923796
-/+ buffers/cache:     155308    1058100
Swap:       524280          4     524276

# uname -a

Linux yourdomain.com.br 2.6.9-42.ELsmp #1 SMP Sat Aug 12 09:39:11 CDT 2006 i686 athlon i386 GNU/Linux

------------------------------------------------

/etc/my.cnf

# Example MySQL config file for very large systems.
#
# This is for a large system with memory of 1G-2G where the system runs mainly
# MySQL.
#
# You can copy this file to
# /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options (in this
# installation this directory is /usr/local/var) or
# ~/.my.cnf to set user-specific options.
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.

# The following options will be passed to all MySQL clients
[client]
#password	= your_password
	port		= 53306
socket		= /tmp/mysql.sock
	default_character_set ="latin1"

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
	max_heap_table_size = 50331648
	basedir		= "/usr/sbin"
	datadir		= "/var/lib/mysql"
	port		= 53306
	max-connections		= 100
	default_character_set   ="latin1"
	collation               ="latin1_general_ci"
	wait_timeout            = 40
	max_connect_error	= 10000
	log_error		= "/var/lib/mysql/mysql_error." 
	log_warnings		= 1
	innodb_locks_unsafe_for_binlog = 0
	skip_ndbcluster
	skip-innodb
	init-file		= "/etc/executeonstart.sql"
socket		= /tmp/mysql.sock
skip-locking
	key_buffer = 256M
max_allowed_packet = 1M
	table_cache = 2048 #antes 512

sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
	myisam_sort_buffer_size = 2M

	thread_cache_size = 100 #antes era 8
	query_cache_size = 8M #antes era 32

	# Try number of CPU's*2 for thread_concurrency
	thread_concurrency = 4

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
# 
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
	#log-bin=mysql-bin

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id	= 1
	auto-increment-increment	= 10
	auto-increment-offset		= 1
	replicate-same-server-id        = 0
	report-host			= 72.232.2.250

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    change in this file to the variables' values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id       = 2
#
# The replication master for this slave - required
#master-host     =   <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user     =   <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =   <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port     =  <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Point the following paths to different dedicated disks
#tmpdir		= /tmp/		
#log-update 	= /path-to-dedicated-directory/hostname

# Uncomment the following if you are using BDB tables
#bdb_cache_size = 384M
#bdb_max_lock = 100000

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /usr/local/var/
#innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
#innodb_log_group_home_dir = /usr/local/var/
#innodb_log_arch_dir = /usr/local/var/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 384M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 100M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

------------------------------------------------

What else can i say for you guys?!

Oh yes, first, we was trying MySQL + Apache on Windows 2003 Server (Web Edition), but Mysqld.exe process was always closing with out ANY log details, even in "Windows Event Viewer". So we decided to install a virtual machine (VMWare) with CentOS just because we knew in Linux we was able to see log, and there is the log above.

Out serves crash a lot, like 15 or 20 times per day.

Can i help in anything else? 

Tks!

Bruno Kilian

PS: Sorry about my english.
[22 Dec 2006 13:00] Valeriy Kravchuk
Bruno,

I am not sure you have the same bug here, as there is 

innodb_locks_unsafe_for_binlog = 0

in your my.cnf. Anyway, please, try to resolve stack trace you sent on 1 Dec 23:27, and send the results.
[23 Jan 2007 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[23 Jan 2007 7:56] Corin Langosch
just to reopen, as the bug described by me still exists...
[26 Jan 2007 12:42] Valeriy Kravchuk
Corin,

Can you identify the exact statement that leads to this crash? Can you also try to use 64-bit version of MySQL 5.0.27, MySQL binaries.

Bruno,

Please, send the resolved stack trace.
[26 Jan 2007 17:43] Corin Langosch
sorry, i cannot give you a sql query that procudes this error. the system simply randomly crashes. for the stack traces, please have a look at my previous posts..i already provided a lot.
[28 Jan 2007 8:05] Valeriy Kravchuk
Corin,

I asked Bruno to resolve his stack traces, not you. I want to check if his case is related.

Still, can you, please, try to use 64-bit binaries of 5.0.27 created by MySQL, not by Debian, and check if this will make any difference.
[28 Jan 2007 11:12] Corin Langosch
hi.
sorry for the misunderstanding, my mistake. as for the binaries, i always use the original precompiled binaries from mysql and not the ones provided by debian.
corin
[29 Jan 2007 11:18] Valeriy Kravchuk
Corin,

Do you use 64-bit binaries or 32-bit (as I think)? Please, send the results of:

file mysqld

command (from the directory where your mysqld is located), just to be sure.
[1 Mar 2007 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[26 Jul 2007 17:31] MySQL Verification Team
yes, innodb_locks_unsafe_for_binlog has a serious bug.
see bug #27294 for which a fix is not yet in any released version.
[26 Dec 2008 15:19] Valeriy Kravchuk
All reporters:

Please, try to repeat with a newer version, 5.0.67 at least, and inform about the results.
[27 Jan 2009 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[27 Jan 2009 6:49] MySQL Verification Team
duplicate of bug #27294