Bug #19849 Segfault of mysql cli
Submitted: 16 May 2006 12:12 Modified: 27 Nov 2006 8:23
Reporter: Sergey Kostyliov Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: Command-line Clients Severity:S3 (Non-critical)
Version:4.1.19 (5.0.21, 5.1.11) OS:Linux (Linux x86-64)
Assigned to: CPU Architecture:Any

[16 May 2006 12:12] Sergey Kostyliov
Description:
Segfault as the result of  ~7GB sql dump restore attempt

How to repeat:
zcat /home/rathamahata/snapshot.sql.gz | mysql

Always lead to this segfault (this is for 5.0.21 mysql official binary).

#0  0x00002aaaab240efc in strcpy () from /lib/tls/libc.so.6
(gdb) bt
#0  0x00002aaaab240efc in strcpy () from /lib/tls/libc.so.6
#1  0x0000000000439763 in com_connect (buffer=0x6fa620,
    line=0x2aaaab8c13f1 "\\r\\n\\r\\nìîæåòå ïîêà çàéòè ïî÷èòàòü íàïèñàíûå\\r\\nhttp://www.stihi.ru/author.html?polegstive',1,1,0,0,0,5,3),(5339960,100124,1034203826,'tes..., õîòü îïðîñ óñòðàâèâàé,íî ëþáîïûòíî ìíå ñòàëî...\\r\\n Ê"...)
    at mysql.cc:2852
#2  0x322c272e2e2e2eea in ?? ()
#3  0x302c302c302c312c in ?? ()

This also happens for 4.1.19 and 5.1.11

Unfortunately, I could not reproduce this with a smaller dump. If you wish - I'll send you a link to ~2Gb gziped file.
[16 May 2006 12:16] Sergey Kostyliov
Category updated
[16 May 2006 16:43] Valeriy Kravchuk
Thank you for a problem report. Please, send your my.cnf content. How large is the table that is loading when this crash happens?
[17 May 2006 8:49] Sergey Kostyliov
mysql> SHOW TABLE STATUS LIKE "dnevnik_reply" \G
*************************** 1. row ***************************
           Name: dnevnik_reply
         Engine: MyISAM
        Version: 9
     Row_format: Dynamic
           Rows: 9425434
 Avg_row_length: 26
    Data_length: 252521144
Max_data_length: 4294967295
   Index_length: 352422912
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2005-01-12 03:49:44
    Update_time: 2006-05-17 12:48:46
     Check_time: 2006-05-07 04:48:52
      Collation: cp1251_general_ci
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (0.01 sec)

mysql> 

<my.cnf>
# /etc/mysql/my.cnf: The global mysql configuration file.
# $Header: /var/cvsroot/gentoo-x86/dev-db/mysql/files/my.cnf-4.1-r1,v 1.3 2006/02/01 20:25:56 vivo Exp $

# The following options will be passed to all MySQL clients
[client]
#password                                       = your_password
port                                            = 3306
socket                                          = /var/run/mysqld/mysqld.sock

[mysql]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=cp1251

[mysqladmin]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=cp1251

[mysqlcheck]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=cp1251

[mysqldump]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=cp1251

[mysqlimport]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=cp1251

[mysqlshow]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=cp1251

[myisamchk]
character-sets-dir=/usr/share/mysql/charsets

[myisampack]
character-sets-dir=/usr/share/mysql/charsets

# use [safe_mysqld] with mysql-3
[mysqld_safe]
err-log                                         = /var/log/mysql/mysql.err

# add a section [mysqld-4.1] or [mysqld-5.0] for specific configurations.
[mysqld]
character-set-server            = cp1251
default-character-set           = cp1251
user                                            = mysql
port                                            = 3306
socket                                          = /var/run/mysqld/mysqld.sock
pid-file                                        = /var/run/mysqld/mysqld.pid
log-error                                       = /var/log/mysql/mysqld.err
basedir                                         = /usr
datadir                                         = /var/lib/mysql
skip-locking
#skip-innodb
key_buffer                                      = 16M
max_allowed_packet                      = 1M
table_cache                             = 64
sort_buffer_size                        = 512K
net_buffer_length                       = 8K
read_buffer_size                        = 256K
read_rnd_buffer_size            = 512K
myisam_sort_buffer_size         = 8M
language                                        = /usr/share/mysql/english

#security:
#using "localhost" in connects use sockets by default
#skip-networking
bind-address                            = 127.0.0.1

log-bin

max-relay-log-size = 1000M
relay-log-space-limit = 2100M

server-id                       = 19801
master-host                     = 10.0.0.1
master-user                     = replication
master-password                 = prqwet1006as

replicate-do-db                 = damochka
replicate-do-db                 = smsgate

skip-slave-start
slave_compressed_protocol       = 1
report-host                     = sqlmaster.damochka.ru

# Point the following paths to different dedicated disks
tmpdir                                          = /tmp/
#log-update                             = /path-to-dedicated-directory/hostname

# you need debug use flag enabled to use this ones.
# if needed uncomment them, start the server and issue
# #tail -f /tmp/mysqld.sql /tmp/mysqld.trace
# this will show you *exactly* what's appening in your server ;)

#log                                            = /tmp/mysqld.sql
#gdb
#debug                                          = d:t:i:o,/tmp/mysqld.trace
#one-thread

# Uncomment the following if you are using BDB tables
#bdb_cache_size                         = 4M
#bdb_max_lock                           = 10000

# The following is the InnoDB configuration
# if you wish to disable innodb instead
# uncomment just the next line
#skip-innodb
#
# the rest of the innodb config follows:
# don't eat too much memory, we're trying to be safe on 64Mb boxes.
# you might want to bump this up a bit on boxes with more RAM
innodb_buffer_pool_size         = 16M
# this is the default, increase if you have lots of tables
innodb_additional_mem_pool_size = 2M
#
# i'd like to use /var/lib/mysql/innodb, but that is seen as a database :-(
# and upstream wants things to be under /var/lib/mysql/, so that's the route
# we have to take for the moment
innodb_data_home_dir            = /var/lib/innodb
innodb_log_arch_dir             = /var/lib/innodb
innodb_log_group_home_dir       = /var/lib/innodb
# you may wish to change this size to be more suitable for your system
# the max is there to avoid run-away growth on your machine
innodb_data_file_path           = ibdata1:10M:autoextend
# we keep this at around 25% of of innodb_buffer_pool_size
# sensible values range from 1MB to (1/innodb_log_files_in_group*innodb_buffer_pool_size)
innodb_log_file_size            = 64M
# this is the default, increase if you have very large transactions.
innodb_log_buffer_size          = 64M
# this is the default, and won't hurt you.
# you shouldn't need to tweak it.
set-variable                            = innodb_log_files_in_group=3
# see the innodb config docs, the other options are not always safe
innodb_flush_log_at_trx_commit  = 1
innodb_lock_wait_timeout        = 500

[mysqldump]
quick
max_allowed_packet                      = 16M

[mysql]
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer                                      = 20M
sort_buffer_size                        = 20M
read_buffer                             = 2M
write_buffer                            = 2M

[myisamchk]
key_buffer                                      = 20M
sort_buffer_size                        = 20M
read_buffer                             = 2M
write_buffer                            = 2M

[mysqlhotcopy]
interactive-timeout

#vim: set tabstop=4 :
#vim: set shiftwidth=4 :
</my.cnf>
[17 May 2006 17:42] Valeriy Kravchuk
Please, try to increase max_allowed_packet for both mysql client and server:

max_allowed_packet=128M

and check again. 

Please, send the ulimit -a results from the user who runs that mysql command.
[18 May 2006 9:27] Sergey Kostyliov
This (max_allowed_packet=128M) hasn't fixed segfault.

localhost damochka # ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 36864
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 36864
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
localhost damochka #
[8 Jun 2006 17:29] Valeriy Kravchuk
Please, describe your hardware. How much RAM do you have? Send df -k results also. Do you use 64-bit binaries of MySQL server (just to be sure)? What glibc version do you use?
[13 Jun 2006 8:00] Sergey Kostyliov
Please, describe your hardware. How much RAM do you have? Send df -k results
also. Do you use 64-bit binaries of MySQL server (just to be sure)? What glibc
version do you use?

Yes, in all cases it was 64-bit binaries.

Glibc is current gentoo glibc:
*  sys-libs/glibc
      Latest version available: 2.3.6-r3
      Latest version installed: 2.3.6-r3
      Size of downloaded files: 48,559 kB
      Homepage:    http://www.gnu.org/software/libc/libc.html
      Description: GNU libc6 (also called glibc2) C library
      License:     LGPL-2

It is dual emt64 server box:
rathamahata@localhost ~ $ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 4
cpu MHz         : 2793.099
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl cid xtpr
bogomips        : 5594.32
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 4
cpu MHz         : 2793.099
cache size      : 1024 KB
physical id     : 3
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl cid xtpr
bogomips        : 5586.51
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 4
cpu MHz         : 2793.099
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl cid xtpr
bogomips        : 5586.49
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 4
cpu MHz         : 2793.099
cache size      : 1024 KB
physical id     : 3
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl cid xtpr
bogomips        : 5586.42
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

rathamahata@localhost ~ $

localhost ~ # lspci
00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 0a)
00:00.1 Class ff00: Intel Corporation E7525/E7520 Error Reporting Registers (rev 0a)
00:01.0 System peripheral: Intel Corporation E7520 DMA Controller (rev 0a)
00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 0a)
00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 0a)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
01:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)
01:00.1 PIC: Intel Corporation 6700/6702PXH I/OxAPIC Interrupt Controller A (rev 09)
01:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)
01:00.3 PIC: Intel Corporation 6700PXH I/OxAPIC Interrupt Controller B (rev 09)
03:04.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
03:04.1 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
04:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)
04:00.1 PIC: Intel Corporation 6700/6702PXH I/OxAPIC Interrupt Controller A (rev 09)
04:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)
04:00.3 PIC: Intel Corporation 6700PXH I/OxAPIC Interrupt Controller B (rev 09)
06:01.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 03)
07:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID (rev 02)
08:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
localhost ~ #

It has about 4Gb RAM
rathamahata@localhost ~ $ free -m
             total       used       free     shared    buffers     cached
Mem:          3956       3787        169          0         65       1628
-/+ buffers/cache:       2093       1862
Swap:         1906          0       1906
rathamahata@localhost ~ $

rathamahata@localhost ~ $ df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2             33202180  13743672  19458508  42% /
udev                   2025568       196   2025372   1% /dev
/dev/sdc1             71253576  19861880  51391696  28% /var/lib
shm                    2025568         0   2025568   0% /dev/shm
rathamahata@localhost ~ $
[4 Jul 2006 15:17] Valeriy Kravchuk
Please, send the result of SHOW CREATE TABLE for that dnevnik_reply problematic table. Do you have any error messages in server's error log, or it is just mysql command line client's crash?
[4 Jul 2006 15:30] Sergey Kostyliov
It is just mysql client crash (it doesn't affect mysqld - i.e. no any log messages in mysqld error log).

mysql> SHOW CREATE TABLE dnevnik_reply;
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table         | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| dnevnik_reply | CREATE TABLE `dnevnik_reply` (
  `mid` int(11) NOT NULL default '0',
  `pmid` int(11) NOT NULL default '0',
  `uid` int(11) NOT NULL default '0',
  `time` int(11) NOT NULL default '0',
  `flags` int(11) default NULL,
  `img` varchar(20) default NULL,
  `imgTitle` varchar(80) default NULL,
  `changed` datetime default NULL,
  PRIMARY KEY  (`mid`),
  KEY `pmid` (`pmid`,`time`),
  KEY `time` (`time`),
  KEY `uid` (`uid`)
) ENGINE=MyISAM DEFAULT CHARSET=cp1251 |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

mysql>
[23 Jul 2006 10:39] Valeriy Kravchuk
What is in that /home/rathamahata/snapshot.sql.gz file? Single INSERT with a lot of (values), (values), ... listed or several INSERTs? Can you repeat similar segfault with much smaller file? < 4G in size? < 128M in size?
[23 Aug 2006 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[26 Nov 2006 19:07] Sergey Kostyliov
I believe this is a duplicate of <a href="http://bugs.mysql.com/bug.php?id=21042">#21042</a>.

All symptomps are the same. Please close it accordingly. Tnank you!
[27 Nov 2006 8:23] Valeriy Kravchuk
Duplicate of bug #21042.