MySQL Bugs: #25777: mysqld segfaults when used in NDB cluster

Bug #25777	mysqld segfaults when used in NDB cluster
Submitted:	23 Jan 2007 11:44	Modified:	14 Jun 2007 18:50
Reporter:	Anders Nordby	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	5.0.37	OS:	FreeBSD (FreeBSD 6.2)
Assigned to:		CPU Architecture:	Any
Tags:	cluster, freebsd, ndb

Description:
I have an NDB cluster, with two mysqld processes, two ndbd processes and two ndb_mgmd processes set up:

ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=3    @XX.XX.40.27  (Version: 5.0.33, Nodegroup: 0)
id=4    @XX.XX.40.28  (Version: 5.0.33, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 2 node(s)
id=1    @XX.XX.40.22  (Version: 5.0.33)
id=2    @XX.XX.40.23  (Version: 5.0.33)

[mysqld(API)]   2 node(s)
id=5    @XX.XX.40.27  (Version: 5.0.33)
id=6    @XX.XX.40.28  (Version: 5.0.33)

Every now and then, one of the mysqld processes gets a segmentation fault. In /var/log/messages:

Jan 23 10:38:51 videovm1 kernel: pid 44812 (mysqld), uid 88: exited on signal 11
 (core dumped)

In mysql.err, I see:

Version: '5.0.33-log'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysq
l-server-5.0.33
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong   
and this may fail.

key_buffer_size=16777216
read_buffer_size=258048
max_used_connections=8
max_connections=2048
threads_connected=7
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 158104
0 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

But I have plenty of free/inactive memory, and the server hasn't even started using swap yet.

So, checking out the core dump, I see:

# gdb -s /usr/local/libexec/mysqld -c mysqld.core 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Core was generated by `mysqld'.
Program terminated with signal 11, Segmentation fault.
#0  0x08321002 in Vector<NdbColumnImpl*>::size (this=0x149) at Vector.hpp:31

warning: Source file is more recent than executable.

31        unsigned size() const { return m_size; };
(gdb) bt
#0  0x08321002 in Vector<NdbColumnImpl*>::size (this=0x149) at Vector.hpp:31
#1  0x0832c2f6 in NdbDictionary::Table::getColumn (this=0x149, 
    name=0x83f8748 <Address 0x83f8748 out of bounds>)
    at NdbDictionaryImpl.hpp:481
#2  0x0833c4fa in NdbBlob::atPrepare (this=0xfbe2c00, aCon=0x149, 
    anOp=0xfb49500, aColumn=0x84d6e00) at NdbBlob.cpp:1119
#3  0x08323953 in NdbOperation::getBlobHandle (this=0xfbe6000, aCon=0xacc8d00, 
    tAttrInfo=0x84d6e00) at NdbOperationDefine.cpp:591
#4  0x08320931 in NdbOperation::getBlobHandle (this=0xfbe6000, anAttrId=3)
    at NdbOperation.cpp:345
#5  0x0816b365 in ha_ndbcluster::get_ndb_value (this=0xc8b5010, 
    ndb_op=0xfbe6000, field=0xc89d358, fieldnr=3, buf=0xc8b6500 "")
    at ha_ndbcluster.cc:891
#6  0x0816c08d in ha_ndbcluster::pk_read (this=0xc8b5010, 
    key=0xc73cd90 "l\002", key_len=4, buf=0xc8b6500 "")
    at ha_ndbcluster.cc:1411
#7  0x0816e72a in ha_ndbcluster::index_read (this=0xc8b5010, buf=0xc8b6500 "", 
    key=0xc73cd90 "l\002", key_len=4, find_flag=HA_READ_KEY_EXACT)
    at ha_ndbcluster.cc:2815
#8  0x0810d753 in join_read_key (tab=0xc6fd174) at sql_select.cc:10740
#9  0x0810cb99 in sub_select (join=0xc73b010, join_tab=0xc6fd174, 
    end_of_records=false) at sql_select.cc:10246
#10 0x0810cd61 in evaluate_join_record (join=0xc73b010, join_tab=0xc6fd010, 
    error=0, report_error=0x149 <Address 0x149 out of bounds>)
    at sql_select.cc:10364
#11 0x0810cbbe in sub_select (join=0xc73b010, join_tab=0xc6fd010, 
    end_of_records=false) at sql_select.cc:10253
#12 0x0810c8a2 in do_select (join=0xc73b010, fields=0x0, table=0xc6ff010, 
    procedure=0x0) at sql_select.cc:10010
#13 0x080fe010 in JOIN::exec (this=0xc73b010) at sql_select.cc:1498
#14 0x080ff2ba in mysql_select (thd=0xacc6000, rref_pointer_array=0xacc63f4, 
    tables=0xacd3420, wild_num=1, fields=@0xacc6358, conds=0xacd4070, 
    og_num=2, order=0xacd42f8, group=0xacd41f8, having=0x0, proc_param=0x0, 
    select_options=2156153344, result=0xacd43e0, unit=0xacc60a0, 
    select_lex=0xacc62cc) at sql_select.cc:2046
#15 0x080fb6c8 in handle_select (thd=0xacc6000, lex=0xacc603c, 
    result=0xacd43e0, setup_tables_done_option=0) at sql_select.cc:245
#16 0x080cfe21 in mysql_execute_command (thd=0xacc6000) at sql_parse.cc:2615
#17 0x080d61f3 in mysql_parse (thd=0xacc6000, 
    inBuf=0xacd3010 "select sum(vv.counter) as videoviews, videos.* from videoviews vv left join videos on vv.video_id=videos.id where vv.created >= '2007-01-22 10:38:46' and vv.created <= now() group by video_id order by"..., 
    length=181166140) at sql_parse.cc:5833
#18 0x080ce603 in dispatch_command (command=COM_QUERY, thd=0xacc6000, 
    packet=0xaccb001 "select sum(vv.counter) as videoviews, videos.* from videoviews vv left join videos on vv.video_id=videos.id where vv.created >= '2007-01-22 10:38:46' and vv.created <= now() group by video_id order by"..., 
    packet_length=228) at sql_parse.cc:1775
#19 0x080cdfb1 in do_command (thd=0xacc6000) at sql_parse.cc:1557
(gdb) q

How to repeat:
Apply http://anders.fupp.net/test/mysql50-server.diff to the ports tree to upgrade the MySQL in FreeBSDs ports system from 5.0.27 to 5.0.33.

Build ports/databases/mysql50-server from the ports system, with WITH_NDB=yes and WITH_ARCHIVE=yes set. The make toggles adds --with-ndbcluster and --with-archive-storage-engine to configure.

Set up a NDB cluster with multiple mysqld, ndbd and ndb_mgmd processes.

PS: I have the same problems with MySQL 5.0.27, but 5.0.33 obviously did not solve them. It's possible I will get the FreeBSD port updated to 5.0.33, but currently it is at 5.0.27.

Suggested fix:
N/A

Thank you for a problem report. Please, send the results of SHOW CREATE TABLE for those videoviews and videos tables mentined in crashing query:

"select sum(vv.counter) as videoviews, videos.* from
videoviews vv left join videos on vv.video_id=videos.id where vv.created >=
'2007-01-22 10:38:46' and vv.created <= now() group by video_id ..."

The videos table:

| videos | CREATE TABLE `videos` (
  `id` int(11) NOT NULL auto_increment,
  `title` varchar(100) default '(no title)',
  `filename` varchar(50) default '',
  `ingress` text,
  `url` varchar(255) default NULL,
  `show_ads` tinyint(4) default '1',
  `publish_mobile_version` tinyint(4) default '0',
  `user_name` varchar(20) default NULL,
  `videotype_id` int(11) default NULL,
  `branch_id` int(11) default NULL,
  `category_id` int(11) default NULL,
  `article_url` varchar(255) default NULL,
  `source_id` int(11) default NULL,
  `externalcategory_id` int(11) default NULL,
  `barrel_id` int(11) default NULL,
  `status_id` int(11) default NULL,
  `status` varchar(200) default '',
  `date_published` datetime default NULL,
  `date_unpublished` datetime default NULL,
  `created` datetime default NULL,
  `modified` datetime default NULL,
  `mobile_status_id` int(11) default NULL,
  `duration` float default NULL,
  `live_url` varchar(255) default NULL,
  `keywords` text,
  `show_on_front` tinyint(4) NOT NULL default '0',
  `show_on_front_category` int(11) default NULL,
  `descriptiveurl` text,
  `date_live` datetime default NULL,
  PRIMARY KEY  (`id`),
  KEY `show_on_front` (`show_on_front`),
  KEY `show_on_front_category` (`show_on_front_category`),
  KEY `date_published` (`date_published`),
  KEY `status_id` (`status_id`),
  KEY `barrel_id` (`barrel_id`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1 | 

The videoviews table:

| videoviews | CREATE TABLE `videoviews` (
  `id` int(11) NOT NULL auto_increment,
  `video_id` int(11) default NULL,
  `player` char(5) default NULL,
  `bandwidth` int(11) default NULL,
  `index_id` int(11) default NULL,
  `created` datetime default NULL,
  `format_id` int(11) default NULL,
  `counter` int(11) default '1',
  PRIMARY KEY  (`id`),
  KEY `created` (`created`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1 | 

PS: Every now and then, I get this cluster related error in mysql.err:

070123 11:18:38 [ERROR] Got error 4263 when reading table './tvs/videos'

According to http://dev.mysql.com/doc/ndbapi/en/ndb-error-codes-other.html, that  means "Invalid blob attributes or invalid blob parts table". But there's no blob in my tvs/videos table. How come I get that error? Should I try to re-create the table?

I don't know if this error is related?

Got another core dump today:

mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=258048
max_used_connections=43
max_connections=2048
threads_connected=43
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections =
1581040 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Writing a core file
070124 10:18:06 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.0.33-log'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port:
mysql-server-5.0.33

GDB backtrace:

# gdb -s /usr/local/libexec/mysqld -c mysqld.core 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Core was generated by `mysqld'.
Program terminated with signal 11, Segmentation fault.
#0  0x2860968b in ?? ()
(gdb) bt
#0  0x2860968b in ?? ()
#1  0x285c419d in ?? ()
#2  0x00018868 in ?? ()
#3  0x0000000b in ?? ()
#4  0x285bd397 in ?? ()
#5  0x0ae53000 in ?? ()
#6  0x0000000b in ?? ()
#7  0x081a930e in write_core (sig=677147292) at stacktrace.c:245
#8  0x080bcaf6 in handle_segfault (sig=11) at mysqld.cc:2132
#9  0xbfbfff94 in ?? ()
#10 0x0000000b in ?? ()
#11 0x0832c2f6 in NdbDictionary::Table::getColumn (this=0x0, 
    name=0x83f8748 <Address 0x83f8748 out of bounds>)
    at NdbDictionaryImpl.hpp:481
#12 0x0833c4fa in NdbBlob::atPrepare (this=0xb098500, aCon=0x0, anOp=0x0, 
    aColumn=0x0) at NdbBlob.cpp:1119
#13 0x08323953 in NdbOperation::getBlobHandle (this=0xb096400, aCon=0xae10e00, 
    tAttrInfo=0x84d6e00) at NdbOperationDefine.cpp:591
#14 0x08320931 in NdbOperation::getBlobHandle (this=0xb096400, anAttrId=3)
    at NdbOperation.cpp:345
#15 0x0816b365 in ha_ndbcluster::get_ndb_value (this=0xa5d4010, 
    ndb_op=0xb096400, field=0xa5fbb58, fieldnr=3, buf=0xa601500 "")
    at ha_ndbcluster.cc:891
#16 0x0816c08d in ha_ndbcluster::pk_read (this=0xa5d4010, 
    key=0xaeef154 "+\002", key_len=4, buf=0xa601500 "")
    at ha_ndbcluster.cc:1411
#17 0x0816f6b0 in ha_ndbcluster::rnd_pos (this=0xa5d4010, buf=0xa601500 "", 
    pos=0xaeef154 "+\002") at ha_ndbcluster.cc:3101
#18 0x08152eda in rr_from_pointers (info=0xaef188c) at records.cc:377
#19 0x0810cbb3 in sub_select (join=0xaefa448, join_tab=0xaef1850, 
    end_of_records=false) at sql_select.cc:10252
#20 0x0810c8a2 in do_select (join=0xaefa448, fields=0xaedf358, table=0x0, 
    procedure=0x0) at sql_select.cc:10010
#21 0x080feeac in JOIN::exec (this=0xaefa448) at sql_select.cc:1879
#22 0x080ff2ba in mysql_select (thd=0xaedf000, rref_pointer_array=0xaedf3f4, 
    tables=0xaefa138, wild_num=1, fields=@0xaedf358, conds=0x0, og_num=1, 
    order=0xaefa3b0, group=0x0, having=0x0, proc_param=0x0, 
    select_options=2156153344, result=0xaefa438, unit=0xaedf0a0, 
    select_lex=0xaedf2cc) at sql_select.cc:2046
#23 0x080fb6c8 in handle_select (thd=0xaedf000, lex=0xaedf03c, 
    result=0xaefa438, setup_tables_done_option=0) at sql_select.cc:245
#24 0x080cfe21 in mysql_execute_command (thd=0xaedf000) at sql_parse.cc:2615
#25 0x080d61f3 in mysql_parse (thd=0xaedf000, 
    inBuf=0xaefa010 "select * from videos order by created desc", 
    length=183365692) at sql_parse.cc:5833
#26 0x080ce603 in dispatch_command (command=COM_QUERY, thd=0xaedf000, 
    packet=0xaeeb001 "-19 14:12:30\0232007-01-19 14:23:09û\00242", 
    packet_length=43) at sql_parse.cc:1775
#27 0x080cdfb1 in do_command (thd=0xaedf000) at sql_parse.cc:1557
#28 0x080cd5fb in handle_one_connection (arg=0x0) at sql_parse.cc:1188
#29 0x285be5c7 in ?? ()
#30 0x0aedf000 in ?? ()
#31 0x00000000 in ?? ()
#32 0x00000000 in ?? ()
#33 0x285be565 in ?? ()
#34 0x00000000 in ?? ()
#35 0x00000000 in ?? ()
#36 0x00000000 in ?? ()
#37 0x0ae53008 in ?? ()
#38 0x00000000 in ?? ()
#39 0x00000000 in ?? ()
#40 0x00000000 in ?? ()
#41 0x00000000 in ?? ()
#42 0x00000000 in ?? ()
#43 0x00000000 in ?? ()
#44 0x00000000 in ?? ()
#45 0x285c729c in ?? ()
#46 0x0ae53000 in ?? ()
#47 0x00000000 in ?? ()
#48 0x00000000 in ?? ()
#49 0xbf1c7fec in ?? ()
#50 0x0ae53000 in ?? ()
#51 0x00030000 in ?? ()
#52 0x00001000 in ?? ()
#53 0xbf167000 in ?? ()
Cannot access memory at address 0xbf197000
(gdb) 

Maybe I should build the OS with debug symbols as well. Hm.

Please, try to repeat with a newer version, 5.0.37, and inform about the results.

small note: datatype "text" is implemented as a blob

I have upgraded OS to FreeBSD 6.2, and MySQL to 5.0.37 today. The database is the same. But it still crashes:

Apr  3 12:47:01 videovm2 kernel: pid 95650 (mysqld), uid 88: exited on signal 11 (core dumped)

root@nettvdb2:/var/db/mysql# gdb -s /usr/local/libexec/mysqld -c mysqld.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Core was generated by `mysqld'.
Program terminated with signal 11, Segmentation fault.
#0  0x286285e3 in ?? ()
(gdb) bt
#0  0x286285e3 in ?? ()
#1  0x285e31e9 in ?? ()
#2  0x000187fe in ?? ()
#3  0x0000000b in ?? ()
#4  0x285dc4d1 in ?? ()
#5  0x09d07100 in ?? ()
#6  0x0000000b in ?? ()
#7  0x081b016a in write_core (sig=677274300) at stacktrace.c:244
#8  0x080bd44a in handle_segfault (sig=11) at mysqld.cc:2134
#9  0xbfbfff94 in ?? ()
#10 0x0000000b in ?? ()
#11 0x08333796 in NdbDictionary::Table::getColumn (this=0x0, 
    name=0x84005b4 <Address 0x84005b4 out of bounds>)
    at NdbDictionaryImpl.hpp:480
#12 0x083439f6 in NdbBlob::atPrepare (this=0xaae8b00, aCon=0x0, anOp=0x187fe, 
    aColumn=0x0) at NdbBlob.cpp:1118
#13 0x0832adc3 in NdbOperation::getBlobHandle (this=0xaae3800, aCon=0x9a4dc00, 
    tAttrInfo=0x9f90080) at NdbOperationDefine.cpp:590
#14 0x08327da1 in NdbOperation::getBlobHandle (this=0xaae3800, anAttrId=27)
    at NdbOperation.cpp:344
#15 0x08171371 in ha_ndbcluster::get_ndb_value (this=0xcd30010, 
    ndb_op=0xaae3800, field=0xcd71ef0, fieldnr=27, buf=0xcd70500 "")
    at ha_ndbcluster.cc:890
#16 0x081720e5 in ha_ndbcluster::pk_read (this=0xcd30010, 
    key=0x9988140 "\234", key_len=4, buf=0xcd70500 "") at ha_ndbcluster.cc:1420
#17 0x081756f0 in ha_ndbcluster::rnd_pos (this=0xcd30010, buf=0xcd70500 "", 
    pos=0x9988140 "\234") at ha_ndbcluster.cc:3105
#18 0x08158b0e in rr_from_pointers (info=0x9972898) at records.cc:376
#19 0x0810ee8b in sub_select (join=0x99f3448, join_tab=0x9972858, 
    end_of_records=254) at sql_select.cc:10360
#20 0x0810eb72 in do_select (join=0x99f3448, fields=0x999a35c, table=0x0, 
    procedure=0x0) at sql_select.cc:10118
#21 0x08100d44 in JOIN::exec (this=0x99f3448) at sql_select.cc:1930
#22 0x0810119a in mysql_select (thd=0x999a000, rref_pointer_array=0x999a3ec, 
    tables=0x99f3138, wild_num=1, fields=@0x999a35c, conds=0x0, og_num=1, 
    order=0x99f33b0, group=0x0, having=0x0, proc_param=0x0, 
    select_options=2156153344, result=0x99f3438, unit=0x999a0a0, 
    select_lex=0x999a2d0) at sql_select.cc:2097
#23 0x080fd038 in handle_select (thd=0x999a000, lex=0x999a03c, 
    result=0x99f3438, setup_tables_done_option=0) at sql_select.cc:244
#24 0x080d0e41 in mysql_execute_command (thd=0x999a000) at sql_parse.cc:2628
#25 0x080d73ef in mysql_parse (thd=0x999a000, 
    inBuf=0x99f3010 "select * from videos order by created desc", 
    length=161062972) at sql_parse.cc:5948
#26 0x080cf4d3 in dispatch_command (command=COM_QUERY, thd=0x999a000, 
    packet=0x99ee001 "Per Tomas Govertsen for julestemning på Montmartre.", 
    packet_length=43) at sql_parse.cc:1786
#27 0x080cee81 in do_command (thd=0x999a000) at sql_parse.cc:1568
#28 0x080ce477 in handle_one_connection (arg=0x0) at sql_parse.cc:1194
#29 0x285dd645 in ?? ()
#30 0x0999a000 in ?? ()
#31 0x00000000 in ?? ()
#32 0x00000000 in ?? ()
#33 0x285dd582 in ?? ()
#34 0x00000000 in ?? ()
#35 0x00000000 in ?? ()
#36 0x00000000 in ?? ()
#37 0x00000000 in ?? ()
#38 0x00000000 in ?? ()
#39 0x00000000 in ?? ()
#40 0x00000000 in ?? ()
#41 0x285e62bc in ?? ()
#42 0x084da000 in ?? ()
#43 0x00000000 in ?? ()
#44 0x00000000 in ?? ()
#45 0xbe92afec in ?? ()
#46 0x09d07100 in ?? ()
#47 0x00030000 in ?? ()
#48 0x00001000 in ?? ()
#49 0xbe7a4000 in ?? ()
Cannot access memory at address 0xbe7d4000
(gdb) 

Could someone please look at this?

I'll send updates if I see more crashes.

Anders,

can you provide us with config files, schema, data, and the query that causes the segv, and we will try to reproduce.

BR,

Tomas

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".