Bug #46985 Got error 4350 'Transaction already aborted' from NDBCLUSTER
Submitted: 28 Aug 2009 21:41 Modified: 23 Jan 2012 9:58
Reporter: Matthew Bilek Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.1-telco-7.0 OS:Linux
Assigned to: Martin Skold CPU Architecture:Any
Tags: 4350, cluster, MySQL Cluster 7.0.6, ndb

[28 Aug 2009 21:41] Matthew Bilek 
Description:
Receive random "Got error 4350 'Transaction already aborted' from NDBCLUSTER" messages when inserting rows with some duplicates into a simple table.  This will occur only after inserting a 10 rows, 10000 rows or anywhere between.  This condition occurs when a primary key with auto increment is defined and another column has an unique key constraint.  If you remove the primary key field, remove the unique index or just index the other column, the error does not occur.

How to repeat:
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=21   @192.168.0.84  (mysql-5.1.34 ndb-7.0.6, Nodegroup: 0)
id=22   @192.168.0.85  (mysql-5.1.34 ndb-7.0.6, Nodegroup: 0)
id=23   @192.168.0.86  (mysql-5.1.34 ndb-7.0.6, Nodegroup: 1)
id=24   @192.168.0.87  (mysql-5.1.34 ndb-7.0.6, Nodegroup: 1, Master)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.0.80  (mysql-5.1.34 ndb-7.0.6)

[mysqld(API)]   9 node(s)
id=11   @192.168.0.81  (mysql-5.1.34 ndb-7.0.6)
id=12   @192.168.0.81  (mysql-5.1.34 ndb-7.0.6)
id=13   @192.168.0.81  (mysql-5.1.34 ndb-7.0.6)
id=14   @192.168.0.82  (mysql-5.1.34 ndb-7.0.6)
id=15   @192.168.0.82  (mysql-5.1.34 ndb-7.0.6)
id=16   @192.168.0.82  (mysql-5.1.34 ndb-7.0.6)
id=17   @192.168.0.83  (mysql-5.1.34 ndb-7.0.6)
id=18   @192.168.0.83  (mysql-5.1.34 ndb-7.0.6)
id=19   @192.168.0.83  (mysql-5.1.34 ndb-7.0.6)

ndb_mgm>

/etc/config.ini
[TCP DEFAULT]
SendBufferMemory=16M
ReceiveBufferMemory=16M

[NDB_MGMD DEFAULT]
DataDir=/usr/local/mysql/data
PortNumber=1186

[MYSQLD DEFAULT]
BatchSize=512

[NDBD DEFAULT]
NoOfReplicas=2
Datadir=/usr/local/mysql/data
DataMemory=10G
IndexMemory=2G
LockPagesInMainMemory=1
MaxNoOfConcurrentTransactions=8192
MaxNoOfConcurrentOperations=1048576
MaxNoOfTables=4096
MaxNoOfOrderedIndexes=2048
MaxNoOfUniqueHashIndexes=512
MaxNoOfAttributes=24576
RedoBuffer=256M

[NDB_MGMD]
Id=1
Hostname=192.168.0.80

[MYSQLD]
Id=11
Hostname=192.168.0.81

[MYSQLD]
Id=12
Hostname=192.168.0.81

[MYSQLD]
Id=13
Hostname=192.168.0.81

[MYSQLD]
Id=14
Hostname=192.168.0.82

[MYSQLD]
Id=15
Hostname=192.168.0.82

[MYSQLD]
Id=16
Hostname=192.168.0.82

[MYSQLD]
Id=17
Hostname=192.168.0.83

[MYSQLD]
Id=18
Hostname=192.168.0.83

[MYSQLD]
Id=19
Hostname=192.168.0.83

[NDBD]
Id=21
Hostname=192.168.0.84

[NDBD]
Id=22
Hostname=192.168.0.85

[NDBD]
Id=23
Hostname=192.168.0.86

[NDBD]
Id=24
Hostname=192.168.0.87

Management Node
/usr/local/mysql/libexec/ndb_mgmd -f /etc/config.ini
/usr/local/mysql/bin/ndb_mgm

API Nodes
/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3307 --user=mysql --ndbcluster --ndb-nodeid=11 --ndb-connectstring=192.168.0.80:1186 --port=3307 --log-error=/usr/local/mysql/var/3307/3307.err --pid-file=/usr/local/mysql/var/3307/3307.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3308 --user=mysql --ndbcluster --ndb-nodeid=12 --ndb-connectstring=192.168.0.80:1186 --port=3308 --log-error=/usr/local/mysql/var/3308/3308.err --pid-file=/usr/local/mysql/var/3308/3308.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3309 --user=mysql --ndbcluster --ndb-nodeid=13 --ndb-connectstring=192.168.0.80:1186 --port=3309 --log-error=/usr/local/mysql/var/3309/3309.err --pid-file=/usr/local/mysql/var/3309/3309.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3307 --user=mysql --ndbcluster --ndb-nodeid=14 --ndb-connectstring=192.168.0.80:1186 --port=3307 --log-error=/usr/local/mysql/var/3307/3307.err --pid-file=/usr/local/mysql/var/3307/3307.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3308 --user=mysql --ndbcluster --ndb-nodeid=15 --ndb-connectstring=192.168.0.80:1186 --port=3308 --log-error=/usr/local/mysql/var/3308/3308.err --pid-file=/usr/local/mysql/var/3308/3308.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3309 --user=mysql --ndbcluster --ndb-nodeid=16 --ndb-connectstring=192.168.0.80:1186 --port=3309 --log-error=/usr/local/mysql/var/3309/3309.err --pid-file=/usr/local/mysql/var/3309/3309.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3307 --user=mysql --ndbcluster --ndb-nodeid=17 --ndb-connectstring=192.168.0.80:1186 --port=3307 --log-error=/usr/local/mysql/var/3307/3307.err --pid-file=/usr/local/mysql/var/3307/3307.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3308 --user=mysql --ndbcluster --ndb-nodeid=18 --ndb-connectstring=192.168.0.80:1186 --port=3308 --log-error=/usr/local/mysql/var/3308/3308.err --pid-file=/usr/local/mysql/var/3308/3308.pid --memlock &

/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var/3309 --user=mysql --ndbcluster --ndb-nodeid=19 --ndb-connectstring=192.168.0.80:1186 --port=3309 --log-error=/usr/local/mysql/var/3309/3309.err --pid-file=/usr/local/mysql/var/3309/3309.pid --memlock &

Data Nodes
/usr/local/mysql/libexec/ndbmtd -c 192.168.0.80 --ndb-nodeid=21
/usr/local/mysql/libexec/ndbmtd -c 192.168.0.80 --ndb-nodeid=22
/usr/local/mysql/libexec/ndbmtd -c 192.168.0.80 --ndb-nodeid=23
/usr/local/mysql/libexec/ndbmtd -c 192.168.0.80 --ndb-nodeid=24

Table Definition
CREATE TABLE `table1` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `data` binary(16) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_table1_data` (`data`)
) TABLESPACE ts_test STORAGE DISK ENGINE=ndbcluster DEFAULT CHARSET=ascii;

Then perform inserts, including some duplicates, using a c++ or C# client.
[7 Sep 2009 3:40] Susanne Ebrecht 
This looks like a cluster specific problem to me.

Set category to cluster.
[7 Sep 2009 13:23] Jørgen Austvik 
Thanks for the bug report.
Can you possibly add the code that shows what you are doing to trigger this error?
[8 Sep 2009 16:29] Matthew Bilek 
Using C# with .NET Connector version 6.0.4.  Also have the same problem with other versions of .NET Connectors.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace MySQLCluster4350
{
    class Program
    {
        static void Main(string[] args)
        {
            MySQLCluster4350 MySQLCluster4350 = new MySQLCluster4350();
            MySQLCluster4350.Run();
        }
    }

    class MySQLCluster4350
    {
        public void Run()
        {
            System.Threading.Thread[] SystemThreadingThreads;

            SystemThreadingThreads = new System.Threading.Thread[16];

            for (int x = 0; x < SystemThreadingThreads.Length; x++)
            {
                SystemThreadingThreads[x] = new System.Threading.Thread(new System.Threading.ThreadStart((new MySQLCluster4350Thread(x)).TestThread));
                SystemThreadingThreads[x].Start();
            }

            while (true)
            {
                System.Threading.Thread.Sleep(5000);
            }
        }
    }

    class MySQLCluster4350Thread
    {
        private MySql.Data.MySqlClient.MySqlConnection MySqlDataMySqlClientMySqlConnection = null;
        private System.Int32 SystemInt32Number;

        public MySQLCluster4350Thread(System.Int32 SystemInt32Number)
        {
            this.SystemInt32Number = SystemInt32Number;
        }

        public void TestThread()
        {
            System.String SystemStringQuery = System.String.Empty;
            System.Int32 SystemInt32Temp;
            System.Int32 SystemInt32ReturnValue;

            try
            {
                System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] begin.");
                MySqlDataMySqlClientMySqlConnection = new MySql.Data.MySqlClient.MySqlConnection();
                MySqlDataMySqlClientMySqlConnection.ConnectionString = "Data Source=192.168.0.xxx;Port=3307;Database=test;User Id=xxx;Password=xxx;";
                MySqlDataMySqlClientMySqlConnection.Open();
                System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] database connection openned.");

                for (System.Int16 SystemInt16IterationCount = 0; SystemInt16IterationCount < 1000; SystemInt16IterationCount++)
                {
                    SystemStringQuery = @"INSERT IGNORE INTO table1 (data) VALUES ";

                    for (System.Int16 SystemInt16RowCount = 0; SystemInt16RowCount < 5; SystemInt16RowCount++)
                    {
                        SystemInt32Temp = (this.SystemInt32Number * 65636) + (SystemInt16IterationCount * 256) + SystemInt16RowCount;
                        SystemStringQuery += "(" + ((SystemInt16IterationCount * 256) + SystemInt16RowCount).ToString("D16") + "),";
                    }

                    SystemStringQuery = SystemStringQuery.Remove(SystemStringQuery.Length - 1);

                    MySql.Data.MySqlClient.MySqlCommand MySqlDataMySqlClientMySqlCommand = new MySql.Data.MySqlClient.MySqlCommand(SystemStringQuery, this.MySqlDataMySqlClientMySqlConnection);
                    SystemInt32ReturnValue = MySqlDataMySqlClientMySqlCommand.ExecuteNonQuery();
                    MySqlDataMySqlClientMySqlCommand = null;

                    System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] insert.");
                }

                MySqlDataMySqlClientMySqlConnection.Close();
                MySqlDataMySqlClientMySqlConnection = null;
                System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] database connection closed.");

                System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] end.");
            }

            catch (MySql.Data.MySqlClient.MySqlException MySqlDataMySqlClientMySqlException)
            {
                System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] " + MySqlDataMySqlClientMySqlException.Message);
                System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] " + MySqlDataMySqlClientMySqlException.StackTrace);
                return;
            }

            catch (System.Exception SystemException)
            {
                System.Console.WriteLine("[" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString() + "] " + SystemException.Message);
                return;
            }
        }
    }
}
[8 Sep 2009 16:31] Matthew Bilek 
Error message and stack trace:

Got error 4350 'Transaction already aborted' from NDBCLUSTER

   at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
   at MySql.Data.MySqlClient.NativeDriver.ReadResult(UInt64& affectedRows, Int64& lastInsertId)
   at MySql.Data.MySqlClient.MySqlDataReader.GetResultSet()
   at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader()
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
   at MySQLCluster4350.MySQLCluster4350Thread.TestThread() in Z:\Documents and Settings\mbilek\My Documents\Visual Studio 2005\Projects\MySQLCluster4350\MySQLCluster4350\Program.cs:line 75
[8 Sep 2009 22:02] Justin Bertman 
Another test case that causes the same issue:

using System;
using System.Collections.Generic;
using System.Text;
using MySql.Data.MySqlClient;
using System.Threading;

namespace TestDBQueries
{
    class Program
    {
        static void Main(string[] args)
        {
            List<Thread> threadlist = new List<Thread>();

            for(int i = 0; i < 100; ++i)
            {
                Thread thread = new Thread(Test4350);
                threadlist.Add(thread);
            }

            foreach (Thread thread in threadlist)
                thread.Start();

            foreach (Thread thread in threadlist)
                thread.Join();

            Console.ReadLine();

        }

        static public void Test4350()
        {
            MySqlConnection m_oDBConnection = new MySqlConnection(Properties.Settings.Default.ConnectionString);
            m_oDBConnection.Open();

            MySqlCommand oCommand = new MySqlCommand();

            oCommand.Connection = m_oDBConnection;
            oCommand.CommandText = "INSERT IGNORE INTO table1 (data) VALUES (6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6),(6)";

            List<string> urls = new List<string>();

            try
            {
                oCommand.ExecuteReader();
            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString() + "\r\n\r\n");
            }
            finally
            {
                m_oDBConnection.Close();
            }
        }
    }
}

If you change the values in the insert statement to only about 10 instances of "6", the program works fine.  I'm only seeing this behavior with large transactions.
[10 Sep 2009 8:21] Sveta Smirnova 
Thank you for the report.

Verified as described.
[5 Oct 2009 15:03] Matthew Bilek 
Any time frame when this will be looked at and when a solution provided?
[22 Feb 2010 15:51] Matthew Bilek 
Any time frame when this will be looked at and when a solution provided?
[1 Apr 2011 18:23] Kristian Srensen 
Any plans to fix this?
[24 May 2011 22:26] Justin Bertman 
Frustrating that no movement has ever been made on this bug.  This caused us to find another distributed database solution, as this bug made mysql cluster unusable for us.
[18 Aug 2011 9:16] Martin Skold 
I managed to reproduce the problem, but it seems to be a configuration issue.
Transactions are aborted due to overload, transactions takes too long
to complete and are aborted. By increasing TransactionDeadlockDetectionTimeout
the problem goes away. What is your setting for this parameter?
[18 Aug 2011 16:19] Jean-Joseph YVON 
HI,

I have the same error with this query 
INSERT INTO ges_process.sys_process_var_prices (`VPR_SYS_FROM_ID`, `VPR_NEW_PRICE`) SELECT CEN_SYS_FROM_ID, IF(CEN_MDT_TYPE_TRANSACTION_INT=1,CEN_VTE_PRIX_VENTE,IF(CEN_MDT_TYPE_TRANSACTION_INT=2, CEN_LOC_LOYER_MENSUEL_CC,-1)) as prix FROM t_centralisation 
  ON DUPLICATE KEY UPDATE VPR_NEW_PRICE=IF(CEN_MDT_TYPE_TRANSACTION_INT=1,CEN_VTE_PRIX_VENTE,IF(CEN_MDT_TYPE_TRANSACTION_INT=2, CEN_LOC_LOYER_MENSUEL_CC,-1));

I have increased TransactionDeadlockDetectionTimeout to 60000

the query take 2s with ges_process table under MyISAM and it crash with this table under ndbcluster and it take 60 sec before it crash.

theris my configuration :
3 ubuntu server 11.04 mysql 5.1.41 and NDB 7.0.13 one manger and 2 data node with mysql.
Each seerver have 16GB of ram and 8 core.
[18 Aug 2011 17:09] Sveta Smirnova 
Martin,

in my case it was default value.
[23 Jan 2012 9:58] Jon Stephens 
Thank you for your bug report. This issue has already been fixed in the latest released version of that product, which you can download at

  http://www.mysql.com/downloads/
[23 Jan 2012 10:21] Jon Stephens 
Documented the fix for this bug in the NDB 7.0.27, 7.1.17, and 7.2.1 changelogs, as follows:

      AUTO_INCREMENT values were not set correctly for INSERT IGNORE 
      statements affecting NDB tables. This could lead such statements to 
      fail with -Got error 4350 'Transaction already aborted' from 
      NDBCLUSTER- when inserting multiple rows containing duplicate values.

Closed.
[23 Jan 2012 10:33] Jon Stephens 
7.1 version fixed/documented in was 7.1.16, not 7.1.17.