MySQL Bugs: #14044: mysqld crashes with an out-of-bounds exception and a SIGSEGV segment violation.

Bug #14044	mysqld crashes with an out-of-bounds exception and a SIGSEGV segment violation.
Submitted:	15 Oct 2005 1:38	Modified:	27 Oct 2008 15:50
Reporter:	Joseph McHenry	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	4.1.14-standard-log	OS:	Solaris (Solaris 10 FCS x86-64)
Assigned to:	Sveta Smirnova	CPU Architecture:	Any

Description:
mysqld is crashes intermittantly.  A trace indicagtes it crashes with an out-of-bounds exception and a SIGSEGV segment violation signal after a call to time() as part of an alarm routine.  The Server is a Sun V40z, 4x AMD Opteron Dual Core CPU with 16GB Memory.  

The OS is using the default configuration of O/S: Solaris 10 FCS x86-64 with the following patches. 
    Patch: 116299-08 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWxsrt, SUNWjaxp, SUNWxrgrt, SUNWxrpcrt
    Patch: 116303-02 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWxrpcrt

The MySQL was installed from the mysql-standard-4.1.14-pc-solaris2.10-x86_64 binary.  The configuration is basically the settings from that install's my-huge.cnf options with table_cache = 5120, max-connections=1024, skip_innodb with replication enabled.  The MySQL table engine is used exclusively.

The crash was at unix time 1129319661.  The following are status variables about 21 seconds before the crash: 
___________________________________________________________________
Unixtime:       1129319640      
Threads: 150  
Questions: 30175744  
Slow queries: 112  
Opens: 178  
Flush tables: 1  
Open tables: 153  
Queries per second avg: 235.101 
Current database:       da      
Aborted_clients 0       
Aborted_connects        0    
Binlog_cache_disk_use   0       
Binlog_cache_use        0       
Bytes_received  4347196663      
Bytes_sent      12023108247  
Com_admin_commands      619982  
Com_alter_db    0       
Com_alter_table 1       
Com_analyze     0       
Com_backup_table     0       
Com_begin       0       
Com_change_db   648616  
Com_change_master       0       
Com_check       0       
Com_checksum 0       
Com_commit      0       
Com_create_db   0       
Com_create_function     0       
Com_create_index        0    
Com_create_table        0       
Com_dealloc_sql 0       
Com_delete      107192  
Com_delete_multi        0       
Com_do       0       
Com_drop_db     0       
Com_drop_function       0       
Com_drop_index  0       
Com_drop_table  0       
Com_drop_user        0       
Com_execute_sql 0       
Com_flush       0       
Com_grant       0       
Com_ha_close    0       
Com_ha_open  0       
Com_ha_read     0       
Com_help        0       
Com_insert      12708   
Com_insert_select       0       
Com_kill     0       
Com_load        0       
Com_load_master_data    0       
Com_load_master_table   0       
Com_lock_tables 240  
Com_optimize    1       
Com_preload_keys        0       
Com_prepare_sql 0       
Com_purge       0       
Com_purge_before_date        0       
Com_rename_table        0       
Com_repair      0       
Com_replace     83656   
Com_replace_select          0       
Com_reset       0       
Com_restore_table       0       
Com_revoke      0       
Com_revoke_all  0       
Com_rollback 0       
Com_savepoint   0       
Com_select      1198312 
Com_set_option  586     
Com_show_binlog_events  0       
Com_show_binlogs     6       
Com_show_charsets       128     
Com_show_collations     128     
Com_show_column_types   0       
Com_show_create_db   6       
Com_show_create_table   20      
Com_show_databases      53      
Com_show_errors 0       
Com_show_fields      58      
Com_show_grants 6       
Com_show_innodb_status  0       
Com_show_keys   52      
Com_show_logs   0       
Com_show_master_status       0       
Com_show_new_master     0       
Com_show_open_tables    0       
Com_show_privileges     0    
Com_show_processlist    3       
Com_show_slave_hosts    1       
Com_show_slave_status   1       
Com_show_status 604  
Com_show_storage_engines        0       
Com_show_tables 388     
Com_show_variables      259     
Com_show_warnings            0       
Com_slave_start 0       
Com_slave_stop  0       
Com_stmt_prepare        0       
Com_stmt_execute        0    
Com_stmt_send_long_data 0       
Com_stmt_reset  0       
Com_stmt_close  0       
Com_truncate    0       
Com_unlock_tables    240     
Com_update      584747  
Com_update_multi        0       
Connections     29282   
Created_tmp_disk_tables 38719
Created_tmp_files       0       
Created_tmp_tables      115416  
Delayed_errors  0       
Delayed_insert_threads  0    
Delayed_writes  0       
Flush_commands  1       
Handler_commit  0       
Handler_delete  83080   
Handler_discover             0       
Handler_read_first      107085  
Handler_read_key        17107990        
Handler_read_next       52277415             
Handler_read_prev       2805289 
Handler_read_rnd        1084636 
Handler_read_rnd_next   274070951       
Handler_rollback     0       
Handler_update  1359840 
Handler_write   1372242 
Key_blocks_not_flushed  0       
Key_blocks_unused       306459       
Key_blocks_used 22822   
Key_read_requests       63275124        
Key_reads       28465   
Key_write_requests      1102662      
Key_writes      664071  
Max_used_connections    192     
Not_flushed_delayed_rows        0       
Open_files      238        
Open_streams    0       
Open_tables     153     
Opened_tables   178     
Qcache_free_blocks      1870    
Qcache_free_memory   8040672 
Qcache_hits     27509003        
Qcache_inserts  1192955 
Qcache_lowmem_prunes    190878  
Qcache_not_cached            5115    
Qcache_queries_in_cache 20437   
Qcache_total_blocks     42805   
Questions       30175748        
Rpl_status           NULL    
Select_full_join        0       
Select_full_range_join  0       
Select_range    552     
Select_range_check           0       
Select_scan     4737    
Slave_open_temp_tables  0       
Slave_running   ON      
Slave_retried_transactions           0       
Slow_launch_threads     0       
Slow_queries    112     
Sort_merge_passes       0       
Sort_range      10730        
Sort_rows       2051531 
Sort_scan       115556  
Table_locks_immediate   2443150 
Table_locks_waited      2696    
Threads_cached       0       
Threads_connected       150     
Threads_created 8992    
Threads_running 2       
Uptime  128352  ___________________________________________________________________

How to repeat:
The problem is intermittant with outages of about 1 second based on the expanded binary logs.  No error occurs in the error log except "mysqld restarted".

So far we cannot isolate the problem to any specific SQL.  The binary logs show the kinds of queries before the restart that have run ok before and since the restarts.  A new database was recently added and was processing low volumes at the time of later restarts.  

The general log was enabled for analysis of the problem, and it showed the following just prior to the crash: 

30218163:                 20731 Query       UPDATE sessions (... a normal sessions table update) 
30218164-                 29236 Quit       
30218165-                 29222 Quit       
30218166-/usr/local/mysql-standard-4.1.14-pc-solaris2.10-x86_64/bin/mysqld, Version: 4.1.14-standard-log. started with:
30218167-Tcp port: 3306  Unix socket: /tmp/mysql.sock

Thank you for a problem report.

Please, send the results of SHOW CREATE TABLE sessions statement. The results of df -k command and content of my.cnf (I saw your note about how it was created from the standard, but just to be sure) will be useful.

How many rows are there in the sessions table? Do you have any unusual messages in the error log before the crash? Have you performed any maintenance actions (CHECK, REPAIR etc.) before the crash?

Please, do not forget to change the status when you send additional information.

Thank you for the additional information. I have 2 more questions to you:

1. Please, send that "UPDATE sessions (... a normal sessions table update)" statement that was executed during the crash. Does it include a value for a TEXT column? How large is this value, if any?

2. If I understood your information right, this is a slave server, replicating from the db2 master. Can you, please, send the content of the error log of the master server for the same period. For all that 3 crashes you encountered.

You may send the answeres as a private comment or as a comment to the issue you created, if you want.

Bug is not repeatable for me with test data. So I close the report as "Can't repeat"