| Bug #11926 | New Simple Stress test scripts cause MySQLD to crash | ||
|---|---|---|---|
| Submitted: | 13 Jul 2005 23:22 | Modified: | 22 Sep 2005 16:57 |
| Reporter: | Jonathan Miller | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Cluster: Cluster (NDB) storage engine | Severity: | S1 (Critical) |
| Version: | 5.1-wl2325 | OS: | Linux (Linux) |
| Assigned to: | Tomas Ulin | CPU Architecture: | Any |
[10 Aug 2005 18:38]
Jonathan Miller
This also happens when there is no cluster activity, just using the two scripts with about 5 of each scripts running.
[18 Aug 2005 12:56]
Tomas Ulin
needed to do a create database BANK to get inout.pl to work. Can't repeat with the following: - running all localhost - running 10 of the inout.pl Is there info missing? Should there not be some tables in BANK? What is the significance of the cd_db.pl... it creates a database unrelated to the inout.pl... should be reproducable without this extra process? How long typically before mysql cores? Have done close to 300000 connections without the issue showing up (see below). master> show processlist; +--------+-------------+-----------+---------------------+---------+------+-----------------------------------+----------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-------------+-----------+---------------------+---------+------+-----------------------------------+----------------------+ | 1 | system user | | | Daemon | 0 | Waiting for event from ndbcluster | NULL | | 107302 | root | localhost | test | Query | 0 | NULL | show processlist | | 296192 | root | localhost | cluster_replication | Query | 0 | cleaning up | DROP DATABASE TESTER | | 296194 | root | localhost | BANK | Quit | 0 | cleaning up | NULL | | 296196 | root | localhost | BANK | Quit | 0 | cleaning up | NULL | | 296199 | root | localhost | BANK | Sleep | 0 | NULL | NULL | | 296202 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296204 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296205 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296206 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296207 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296208 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296209 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296210 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296211 | root | localhost | BANK | Quit | 0 | NULL | NULL | | 296212 | root | localhost | NULL | Connect | NULL | login | NULL | | 296213 | root | localhost | NULL | Connect | NULL | login | NULL | | 296214 | root | localhost | NULL | Connect | NULL | login | NULL | | 296215 | root | localhost | NULL | Connect | NULL | login | NULL | | 296216 | root | localhost | NULL | Connect | NULL | login | NULL | | 296217 | root | localhost | NULL | Connect | NULL | login | NULL | | 296218 | root | localhost | NULL | Connect | NULL | login | NULL | | 296219 | root | localhost | NULL | Connect | NULL | login | NULL | | 296220 | root | localhost | NULL | Connect | NULL | login | NULL | | 296221 | root | localhost | NULL | Connect | NULL | login | NULL | | 296222 | root | localhost | NULL | Connect | NULL | login | NULL | +--------+-------------+-----------+---------------------+---------+------+-----------------------------------+----------------------+ 26 rows in set (0.01 sec)
[18 Aug 2005 13:19]
Jonathan Miller
needed to do a create database BANK to get inout.pl to work. or *you can just change the script to use test. Should there not be some tables in BANK? *Not needed What is the significance of the cd_db.pl... it creates a database unrelated to the inout.pl... should be reproducable without this extra process? *It adds stress, I.E the purpose of stress testing it to add stress to the system. Note: 2 sessions of BANK, 2 sessions of Sabre were also being done. Also, stack trace was provided: 0x816e55c 0x4005e5cd 0x835c2f7 0x835b6fe 0x834ad6a 0x8349bad 0x8349bad 0x838fd4e 0x8398930 0x83972a2 0x8390203 0x839017f 0x83805f5 0x400586de 0x401d86c7
[22 Sep 2005 16:57]
Jonathan Miller
We are no longer able to reproduce the core associated with this bug report with the latest version of the 5.1-wl2325.
Description: Was running several stress test against cluster replication. 2 sessions of BANK, 2 sessions of Sabre and introduced to simple perl scripts. One that did nothing more then login to a cluster database (BANK) and logout, and another one that connected, created a database (TESTER) , drop the database and then disconnected. Both of these ran in continous loops. Stack Trace is as follow: 0x816e55c handle_segfault + 392 0x4005e5cd _end + 934750233 0x835c2f7 _Z19find_bucket_chainedP6VectorI13Gci_containerEy + 243 0x835b6fe _ZN14NdbEventBuffer24execSUB_GCP_COMPLETE_REPEPK17SubGcpCompleteRep + 1114 0x834ad6a _ZN3Ndb20handleReceivedSignalEP12NdbApiSignalP16LinearSectionPtr + 3606 0x8349bad _ZN3Ndb14executeMessageEPvP12NdbApiSignalP16LinearSectionPtr + 33 0x838fd4e _Z7executePvP12SignalHeaderhPjP16LinearSectionPtr + 1022 0x8398930 _ZN19TransporterRegistry6unpackEPjjt7IOState + 948 0x83972a2 _ZN19TransporterRegistry14performReceiveEv + 390 0x8390203 _ZN17TransporterFacade17threadMainReceiveEv + 123 0x839017f runReceiveResponse_C + 27 0x83805f5 ndb_thread_wrapper + 53 0x400586de _end + 934725930 0x401d86c7 _end + 936298771 ----------------------------------------- key_buffer_size=8388600 read_buffer_size=131072 max_used_connections=9 max_connections=100 threads_connected=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=(nil) Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbe9ff2d8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x816e55c 0x4005e5cd 0x835c2f7 0x835b6fe 0x834ad6a 0x8349bad 0x8349bad 0x838fd4e 0x8398930 0x83972a2 0x8390203 0x839017f 0x83805f5 0x400586de 0x401d86c7 How to repeat: Not sure how reatable it is, scripts as follows: create drop: #!/user/bin/perl -w #/* Copyright (C) 2000-2005 MySQL AB # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # Version 1.1 ####################### Includes ############################ use DBI; ####################### Globals ############################ my $m_host=''; my $m_port=''; my $m_user=''; my $m_pass=''; my $dbhM=''; ####################### Sub Pototypes ############################ sub CollectCommandPromptInfo; sub ConnectToDatabases; sub DisconnectFromDatabases; sub CreateDB; sub DropDB; ######################## Program Main ########################### while(1) { CollectCommandPromptInfo; ConnectToDatabases; CreateDB; DropDB; DisconnectFromDatabases; } ###################### Collect Command Prompt Info ############# sub CollectCommandPromptInfo { ### Check that user has supplied correct number of command line args die "Usage:\n cd_db.pl <master MySQL host> <master MySQL port> <master user> < master pass>\n All 4 arguments must be passed. Use BLANK for NULL passwords\n" unless @ARGV == 4; $m_host =$ARGV[0]; $m_port = $ARGV[1]; $m_user = $ARGV[2]; $m_pass = $ARGV[3]; if ($m_pass eq "BLANK") { $m_pass = '';} } ###################### Collect Command Prompt Info ############# sub ConnectToDatabases { ### Connect to both master and slave clusters databases ### Connect to master $dbhM = DBI->connect("dbi:mysql:database=cluster_replication;host=$m_host;port=$m_port", "$m_user", "$m_pass") or die "Can't connect to Master Cluster MySQL process! Error: $DBI::errstr\n"; } sub DisconnectFromDatabases { ### Disconnect from master $dbhM->disconnect or warn " Disconnection failed: $DBI::errstr\n"; } sub CreateDB { $sth = $dbhM->prepare("CREATE DATABASE TESTER;") or die "Prepare error: ", $dbhM->errstr; $sth->execute or die "Create Error: ", $sth->errstr; $sth->finish; } sub DropDB { $sth = $dbhM->prepare("DROP DATABASE TESTER;") or die "Prepare drop error: ", $dbhM->errstr; $sth->execute or die "drop error: ", $sth->errstr; $sth->finish; } ----------------------------- Login and out: ndb08:~/jmiller> cat inout.pl #!/user/bin/perl -w #/* Copyright (C) 2000-2005 MySQL AB # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # Version 1.1 ####################### Includes ############################ use DBI; ####################### Globals ############################ my $m_host=''; my $m_port=''; my $m_user=''; my $m_pass=''; my $s_host=''; my $s_port=''; my $s_user=''; my $s_pass=''; my $dbhM=''; my $dbhS=''; ####################### Sub Pototypes ############################ sub CollectCommandPromptInfo; sub ConnectToDatabases; sub DisconnectFromDatabases; ######################## Program Main ########################### while(1) { CollectCommandPromptInfo; ConnectToDatabases; DisconnectFromDatabases; } ###################### Collect Command Prompt Info ############# sub CollectCommandPromptInfo { ### Check that user has supplied correct number of command line args die "Usage:\n inout.pl <master MySQL host> <master MySQL port> <master user> < master pass>\n All 4 arguments must be passed. Use BLANK for NULL passwords\n" unless @ARGV == 4; $m_host =$ARGV[0]; $m_port = $ARGV[1]; $m_user = $ARGV[2]; $m_pass = $ARGV[3]; if ($m_pass eq "BLANK") { $m_pass = '';} } ###################### Collect Command Prompt Info ############# sub ConnectToDatabases { ### Connect to both master and slave clusters databases ### Connect to master $dbhM = DBI->connect("dbi:mysql:database=BANK;host=$m_host;port=$m_port", "$m_user", "$m_pass") or die "Can't connect to Master Cluster MySQL process! Error: $DBI::errstr\n"; } sub DisconnectFromDatabases { ### Disconnect from master $dbhM->disconnect or warn " Disconnection failed: $DBI::errstr\n"; }