Bug #20364 MySQL-Max Kernel Panic
Submitted: 9 Jun 2006 16:04 Modified: 27 Jun 2006 11:15
Reporter: Luiz Boaretto Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:4.1.19 OS:Linux (RHEL 4/kernel 2.6.9-22.EL)
Assigned to: CPU Architecture:Any

[9 Jun 2006 16:04] Luiz Boaretto 
Description:
My server goes offline two times.

I see in my log file the problem!

Jun  9 04:50:39 servidor01 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000001
Jun  9 04:50:39 servidor01 kernel:  printing eip:
Jun  9 04:50:39 servidor01 kernel: c001c540
Jun  9 04:50:39 servidor01 kernel: *pde = 00000000
Jun  9 04:50:39 servidor01 kernel: Oops: 0002 [#1]
Jun  9 04:50:39 servidor01 kernel: Modules linked in: ipt_state ip_conntrack ipt_REJECT ipt_LOG ipt_recent iptable_filter ip_tables md5 ipv6 autofs4 dm_mirror dm_mod button battery ac snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore 8139too mii floppy ext3 jbd
Jun  9 04:50:39 servidor01 kernel: CPU:    0
Jun  9 04:50:39 servidor01 kernel: EIP:    0060:[<c001c540>]    Not tainted VLI
Jun  9 04:50:39 servidor01 kernel: EFLAGS: 00010002   (2.6.9-22.EL)
Jun  9 04:50:39 servidor01 kernel: EIP is at 0xc001c540
Jun  9 04:50:39 servidor01 kernel: eax: c19e2130   ebx: c19e2130   ecx: 00000000   edx: 00000001
Jun  9 04:50:39 servidor01 kernel: esi: d235ff34   edi: 0000000e   ebp: 00000000   esp: d235ff18
Jun  9 04:50:39 servidor01 kernel: ds: 007b   es: 007b   ss: 0068
Jun  9 04:50:39 servidor01 kernel: Process mysqld-max (pid: 16654, threadinfo=d235f000 task=f1323970)
Jun  9 04:50:39 servidor01 kernel: Stack: c012c41b c012ca7d c19e2130 0000000e 0000087f d235f000 c012fddb 0000000e
Jun  9 04:50:39 servidor01 kernel:        00000000 fffffffa 0000087f 00000064 c0368480 f5d84300 f1323970 c011e59b
Jun  9 04:50:39 servidor01 kernel:        d235ff58 d235ff58 c02ac441 00000000 0000000d d235fe8c 0430e220 d235ffa8
Jun  9 04:50:39 servidor01 kernel: Call Trace:
Jun  9 04:50:39 servidor01 kernel:  [<c012c41b>] signal_wake_up+0x18/0x19
Jun  9 04:50:39 servidor01 kernel:  [<c012ca7d>] specific_send_sig_info+0x86/0x8d
Jun  9 04:50:39 servidor01 kernel:  [<c012fddb>] sys_tgkill+0x17f/0x211
Jun  9 04:50:39 servidor01 kernel:  [<c011e59b>] autoremove_wake_function+0x0/0x2d
Jun  9 04:50:39 servidor01 kernel:  [<c02ac441>] sys_setsockopt+0x80/0x89
Jun  9 04:50:39 servidor01 kernel:  [<c012f381>] sys_rt_sigprocmask+0x260/0x270
Jun  9 04:50:39 servidor01 kernel:  [<c030f91f>] syscall_call+0x7/0xb
Jun  9 04:50:39 servidor01 kernel:  [<c030007b>] __xfrm_policy_destroy+0x26/0x41
Jun  9 04:50:39 servidor01 kernel: Code: 00 00 00 00 00 00 00 1c c5 01 c0 1c c5 01 c0 00 d0 91 f8 80 cf 91 f8 00 24 6f f7 00 00 00 00 38 c5 01 c0 58 c4 01 c0 00 00 00 00 <20> 02 00 00 00 00 00 00 3c 4b 24 1d 00 00 00 00 05 00 00 00 5e
Jun  9 04:50:39 servidor01 kernel:  <0>Fatal exception: panic in 5 seconds

BUIILD INFO:
rpm src build MySQL-4.1.19-0.src.rpm

rpm -qa|grep MySQL
MySQL-debuginfo-4.1.19-0.glibc23
MySQL-server-4.1.19-0.glibc23
MySQL-client-4.1.19-0.glibc23
MySQL-embedded-4.1.19-0.glibc23
MySQL-shared-4.1.19-0.glibc23
MySQL-devel-4.1.19-0.glibc23
MySQL-Max-4.1.19-0.glibc23

How to repeat:
I don't know
[11 Jun 2006 7:48] Valeriy Kravchuk 
Thank you for a problem report. Please, send your my.cnf file content. How much RAM do you have? Is there anything unusual in MySQL server's error log?
[11 Jun 2006 15:08] Luiz Boaretto 
my my.cnf

Attachment: my.cnf (application/octet-stream, text), 4.87 KiB.
[11 Jun 2006 15:09] Luiz Boaretto 
I have 1GB of memory.

The my.cnf is attached;

Thank you
[16 Jun 2006 22:25] Luiz Boaretto 
My server stopped again. Of this time they had not been recorded information in logfile. Exists some way to twirl in way debug, or some option when to give kernel panic to be recorded as showed in the first time?
[18 Jun 2006 23:31] Luiz Boaretto 
Kernel panic had been constant. Exists some thing that I can make to improve the information and thus to verify better this problem to arrange a solution?
[18 Jun 2006 23:32] Luiz Boaretto 
Kernel panic had been constant. Exists some thing that I can make to improve the information and thus to verify better this problem to arrange a solution?
[20 Jun 2006 2:20] Luiz Boaretto 
I read the documentation and I'm rum mysql with debug mode "--debug=d,info,error,query,general,where:O,/tmp/mysqld.trace"

When the problem come back, what procedures I must take for better informs them on the situation?

Thanks.
[20 Jun 2006 9:49] Valeriy Kravchuk 
Do you have anything unusual in MySQL server's error log for that kernel panic periods?

According to your my.cnf:

key_buffer = 384M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
query_cache_size = 32M

You may use between:

384 + (2+2)*100 + 32 = 816M

and

384 + (2+2+8)*100 + 32 + M*64 = 1680+ M

of memory for mysqld only, in case of default maximum 100 concurrent connections allowed. How many concurrent connections do you usually have?

Please, send "free" Linux command results.

Please, send that /tmp/mysqld.trace file in case of next kernel panic.

I am almost sure that this is not a MySQL bug, but just improper configuration for your usual load.
[20 Jun 2006 13:11] Luiz Boaretto 
> Do you have anything unusual in MySQL server's error log for that kernel
> panic periods?

I don't see any errors in mysqld-err.log... But I will be placing the log here.

> of memory for mysqld only, in case of default maximum 100 concurrent
> connections allowed. How many concurrent connections do you usually
> have?

Uptime: 104328  Threads: 12  Questions: 1354170  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.980  Memory in use: 428420K  Max memory used: 431456K
Uptime: 104333  Threads: 12  Questions: 1354329  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.981  Memory in use: 428420K  Max memory used: 431456K
Uptime: 104338  Threads: 12  Questions: 1354424  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.981  Memory in use: 428420K  Max memory used: 431456K
Uptime: 104343  Threads: 11  Questions: 1354590  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.982  Memory in use: 428338K  Max memory used: 431456K
Uptime: 104348  Threads: 12  Questions: 1354688  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.982  Memory in use: 428421K  Max memory used: 431456K
Uptime: 104353  Threads: 14  Questions: 1354900  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.984  Memory in use: 428584K  Max memory used: 431456K
Uptime: 104358  Threads: 12  Questions: 1354993  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.984  Memory in use: 428421K  Max memory used: 431456K
Uptime: 104363  Threads: 12  Questions: 1355122  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.985  Memory in use: 428421K  Max memory used: 431456K
Uptime: 104368  Threads: 12  Questions: 1355201  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.985  Memory in use: 428421K  Max memory used: 431456K
Uptime: 104373  Threads: 13  Questions: 1355351  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.986  Memory in use: 428503K  Max memory used: 431456K
Uptime: 104378  Threads: 12  Questions: 1355429  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.986  Memory in use: 428421K  Max memory used: 431456K
Uptime: 104383  Threads: 13  Questions: 1355548  Slow queries: 0  Opens: 132  Flush tables: 1  Open tables: 126  Queries per second avg: 12.986  Memory in use: 428502K  Max memory used: 431456K

> Please, send "free" Linux command results.
[root@servidor01 mysql]# free
             total       used       free     shared    buffers     cached
Mem:       1027668     655148     372520          0       3092      23548
-/+ buffers/cache:     628508     399160
Swap:      2048276     372008    1676268

MySQL is using 40%/average of memory

You recommend another command for more detailed statistics?

Really, my server uses the MySQL due to the email server uses it constantly...

You recommend some changes in my "my.cnf" for a better performance or to fix the problem?

Thank you.
[20 Jun 2006 13:12] Luiz Boaretto 
my mysqld-err log file

Attachment: mysqld-err.log (application/octet-stream, text), 8.28 KiB.
[20 Jun 2006 15:49] Luiz Boaretto 
mysql> SHOW STATUS;
+----------------------------+-----------+
| Variable_name              | Value     |
+----------------------------+-----------+
| Aborted_clients            | 16        |
| Aborted_connects           | 195       |
| Binlog_cache_disk_use      | 0         |
| Binlog_cache_use           | 0         |
| Bytes_received             | 124422810 |
| Bytes_sent                 | 621774573 |
| Com_admin_commands         | 1845      |
| Com_alter_db               | 0         |
| Com_alter_table            | 0         |
| Com_analyze                | 0         |
| Com_backup_table           | 0         |
| Com_begin                  | 510       |
| Com_change_db              | 256527    |
| Com_change_master          | 0         |
| Com_check                  | 0         |
| Com_checksum               | 0         |
| Com_commit                 | 510       |
| Com_create_db              | 0         |
| Com_create_function        | 0         |
| Com_create_index           | 0         |
| Com_create_table           | 0         |
| Com_dealloc_sql            | 0         |
| Com_delete                 | 106402    |
| Com_delete_multi           | 0         |
| Com_do                     | 0         |
| Com_drop_db                | 0         |
| Com_drop_function          | 0         |
| Com_drop_index             | 0         |
| Com_drop_table             | 0         |
| Com_drop_user              | 0         |
| Com_execute_sql            | 0         |
| Com_flush                  | 0         |
| Com_grant                  | 0         |
| Com_ha_close               | 0         |
| Com_ha_open                | 0         |
| Com_ha_read                | 0         |
| Com_help                   | 0         |
| Com_insert                 | 12242     |
| Com_insert_select          | 0         |
| Com_kill                   | 0         |
| Com_load                   | 0         |
| Com_load_master_data       | 0         |
| Com_load_master_table      | 0         |
| Com_lock_tables            | 0         |
| Com_optimize               | 0         |
| Com_preload_keys           | 0         |
| Com_prepare_sql            | 0         |
| Com_purge                  | 0         |
| Com_purge_before_date      | 0         |
| Com_rename_table           | 0         |
| Com_repair                 | 0         |
| Com_replace                | 65950     |
| Com_replace_select         | 0         |
| Com_reset                  | 0         |
| Com_restore_table          | 0         |
| Com_revoke                 | 0         |
| Com_revoke_all             | 0         |
| Com_rollback               | 0         |
| Com_savepoint              | 0         |
| Com_select                 | 342313    |
| Com_set_option             | 169       |
| Com_show_binlog_events     | 0         |
| Com_show_binlogs           | 5         |
| Com_show_charsets          | 19        |
| Com_show_collations        | 19        |
| Com_show_column_types      | 0         |
| Com_show_create_db         | 0         |
| Com_show_create_table      | 5         |
| Com_show_databases         | 3         |
| Com_show_errors            | 0         |
| Com_show_fields            | 10        |
| Com_show_grants            | 7         |
| Com_show_innodb_status     | 0         |
| Com_show_keys              | 2         |
| Com_show_logs              | 0         |
| Com_show_master_status     | 0         |
| Com_show_ndb_status        | 0         |
| Com_show_new_master        | 0         |
| Com_show_open_tables       | 0         |
| Com_show_privileges        | 0         |
| Com_show_processlist       | 23        |
| Com_show_slave_hosts       | 0         |
| Com_show_slave_status      | 0         |
| Com_show_status            | 34        |
| Com_show_storage_engines   | 0         |
| Com_show_tables            | 26651     |
| Com_show_variables         | 47        |
| Com_show_warnings          | 0         |
| Com_slave_start            | 0         |
| Com_slave_stop             | 0         |
| Com_stmt_close             | 0         |
| Com_stmt_execute           | 0         |
| Com_stmt_prepare           | 0         |
| Com_stmt_reset             | 0         |
| Com_stmt_send_long_data    | 0         |
| Com_truncate               | 0         |
| Com_unlock_tables          | 0         |
| Com_update                 | 103330    |
| Com_update_multi           | 0         |
| Connections                | 199635    |
| Created_tmp_disk_tables    | 365       |
| Created_tmp_files          | 4         |
| Created_tmp_tables         | 932       |
| Delayed_errors             | 0         |
| Delayed_insert_threads     | 0         |
| Delayed_writes             | 0         |
| Flush_commands             | 1         |
| Handler_commit             | 0         |
| Handler_delete             | 7652      |
| Handler_discover           | 0         |
| Handler_read_first         | 131622    |
| Handler_read_key           | 292418    |
| Handler_read_next          | 2000089   |
| Handler_read_prev          | 84133     |
| Handler_read_rnd           | 79934     |
| Handler_read_rnd_next      | 3055135   |
| Handler_rollback           | 261       |
| Handler_update             | 165839    |
| Handler_write              | 190113    |
| Key_blocks_not_flushed     | 0         |
| Key_blocks_unused          | 348100    |
| Key_blocks_used            | 800       |
| Key_read_requests          | 1125104   |
| Key_reads                  | 856       |
| Key_write_requests         | 62031     |
| Key_writes                 | 41142     |
| Max_used_connections       | 50        |
| Not_flushed_delayed_rows   | 0         |
| Open_files                 | 220       |
| Open_streams               | 0         |
| Open_tables                | 132       |
| Opened_tables              | 138       |
| Qcache_free_blocks         | 5061      |
| Qcache_free_memory         | 16057608  |
| Qcache_hits                | 523252    |
| Qcache_inserts             | 340490    |
| Qcache_lowmem_prunes       | 66687     |
| Qcache_not_cached          | 1758      |
| Qcache_queries_in_cache    | 16127     |
| Qcache_total_blocks        | 37490     |
| Questions                  | 1637441   |
| Rpl_status                 | NULL      |
| Select_full_join           | 1         |
| Select_full_range_join     | 0         |
| Select_range               | 1657      |
| Select_range_check         | 0         |
| Select_scan                | 47957     |
| Slave_open_temp_tables     | 0         |
| Slave_retried_transactions | 0         |
| Slave_running              | OFF       |
| Slow_launch_threads        | 3         |
| Slow_queries               | 0         |
| Sort_merge_passes          | 0         |
| Sort_range                 | 2157      |
| Sort_rows                  | 55531     |
| Sort_scan                  | 1580      |
| Table_locks_immediate      | 666742    |
| Table_locks_waited         | 876       |
| Threads_cached             | 5         |
| Threads_connected          | 12        |
| Threads_created            | 638       |
| Threads_running            | 1         |
| Uptime                     | 114290    |
+----------------------------+-----------+
163 rows in set (0.05 sec)
[23 Jun 2006 10:11] Valeriy Kravchuk 
I just pointed out possible misconfigurations. If you are sure that MySQL server never used more that 400+ M of RAM, then I see no need to change anything. 

I also do not see any bug/crash at the MySQL side. I would recommend you to uprgade kernel/glibc, check RAM, and try to repeat kernel panic situation again.

In case of easily repeatable kernel panics, please, enable general query log and upload it. I need to know what exact actions lead to the situation you described.
[26 Jun 2006 23:47] Luiz Boaretto 
Running 'memtester'. I have a hardware problem.

Sorry!

Apologies for any inconvenience this may cause.

Thanks.
[27 Jun 2006 11:15] Valeriy Kravchuk 
Closed as not a bug in MySQL code.