Bug #54152 ArgumentOutOfRangeException in QueryNormalizer.CollapseValueList
Submitted: 1 Jun 2010 16:00 Modified: 28 Jun 2010 12:58
Reporter: Yvan Rodrigues Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / NET Severity:S1 (Critical)
Version:6.2.3 OS:Windows
Assigned to: Vladislav Vaintroub CPU Architecture:Any
Tags: ArgumentOutOfRangeException, CollapseValueList, INDEX, QueryNormalizer, tok

[1 Jun 2010 16:00] Yvan Rodrigues 
Description:
In QueryNormalizer line 182:

            // now mark all the other tokens as no output
            while (index <= parenIndices[parenIndices.Count - 1])
                tok[index++].Output = false;

The length of tok and the length of parenIndices[parenIndices.Count - 1] can be equal. Since tok is zero-indexed like all arrays in C#, the last line will be executed when index == tok.Length, throwing an unhandled ArgumentOutOfRangeException.

How to repeat:
The query that triggered this exception is:
INSERT INTO `Website_Language` (`Key`, `Type`, `en`, `en_US`, `fr`, `es`, `MimeType`, `Phase2`, `LastRequested`) VALUES (@Key, @Type, @en, @en_US, @fr, @es, @MimeType, @Phase2, @LastRequested)

The stack trace at time of exception is:

>	MySql.Data.dll!MySql.Data.Common.QueryNormalizer.CollapseValueList(System.Collections.Generic.List<MySql.Data.Common.Token> tok = Count = 77, ref int pos = 77) Line 184	C#
 	MySql.Data.dll!MySql.Data.Common.QueryNormalizer.CollapseValueLists(System.Collections.Generic.List<MySql.Data.Common.Token> tok = Count = 77) Line 144 + 0xf bytes	C#
 	MySql.Data.dll!MySql.Data.Common.QueryNormalizer.Normalize(string sql = "INSERT INTO `Website_Language` (`Key`, `Type`, `en`, `en_US`, `fr`, `es`, `MimeType`, `Phase2`, `LastRequested`) VALUES ('TITLE_CLASSIC_UBTSC', 'SIMPLE', NULL, NULL, NULL, NULL, 'text/plain', NULL, NULL), ('PRODUCT_DESCRIPTION_CLASSIC_ULTIMATE_BACK-TO-SCHOOL_COMBO_2010','SUBSTITUTE','',NULL,NULL,NUL") Line 64 + 0xe bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.TracingDriver.SendQuery(MySql.Data.MySqlClient.MySqlPacket p = {MySql.Data.MySqlClient.MySqlPacket}) Line 69 + 0xd bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.Statement.ExecuteNext() Line 95 + 0x16 bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.PreparableStatement.ExecuteNext() Line 167 + 0x8 bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.Statement.Execute() Line 85 + 0x8 bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.PreparableStatement.Execute() Line 122 + 0x8 bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(System.Data.CommandBehavior behavior = Default) Line 432 + 0xb bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.MySqlCommand.ExecuteReader() Line 346 + 0xa bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery() Line 305 + 0x8 bytes	C#
 	MySql.Data.dll!MySql.Data.MySqlClient.MySqlDataAdapter.ExecuteBatch() Line 170 + 0x11 bytes	C#
 	System.Data.dll!System.Data.Common.DbDataAdapter.UpdateBatchExecute(System.Data.Common.DbDataAdapter.BatchCommandInfo[] batchCommands = {System.Data.Common.DbDataAdapter.BatchCommandInfo[3]}, int commandCount = 3, System.Data.Common.RowUpdatedEventArgs rowUpdatedEvent = {MySql.Data.MySqlClient.MySqlRowUpdatedEventArgs}) + 0x2f bytes	
 	System.Data.dll!System.Data.Common.DbDataAdapter.Update(System.Data.DataRow[] dataRows, System.Data.Common.DataTableMapping tableMapping = {Table}) + 0x74a bytes	
 	System.Data.dll!System.Data.Common.DbDataAdapter.UpdateFromDataTable(System.Data.DataTable dataTable, System.Data.Common.DataTableMapping tableMapping) + 0x2b bytes	
 	System.Data.dll!System.Data.Common.DbDataAdapter.Update(System.Data.DataTable dataTable) + 0x98 bytes	
 	MisMabel.Core.dll!MisMabel.Core.DataAccess.MySqlDataAccess.OnUpdate(System.Data.DataTable DataTable = {Website_Language}) Line 221 + 0xd bytes	C#
 	MisMabel.Core.dll!MisMabel.Core.DataAccess.MySqlDataAccess.UpdateBatch(System.Data.DataTable DataTable = {Website_Language}) Line 271 + 0xb bytes	C#
 	MisMabel.Marketing.dll!MisMabel.Marketing.WebsitePageEditor.Save() Line 117 + 0x32 bytes	C#

The segment of the query post-tokenization is:
INSERT INTO `Website_Language` (`Key`, `Type`, `en`, `en_US`, `fr`, `es`, `MimeType`, `Phase2`, `LastRequested`) VALUES ('TITLE_CLASSIC_UBTSC', 'SIMPLE', NULL, NULL, NULL, NULL, 'text/plain', NULL, NULL), ('PRODUCT_DESCRIPTION_CLASSIC_ULTIMATE_BACK-TO-SCHOOL_COMBO_2010','SUBSTITUTE','',NULL,NULL,NUL

Suggested fix:
--- C:/Users/YVAN~1.HAM/AppData/Local/Temp/QueryNormalizer.cs-revBASE.svn000.tmp.cs	Tue Jun  1 11:50:55 2010
+++ C:/Users/yvan.HAM1/Documents/Visual Studio 2008/Projects/MisMabel Trunk/MySql/Source/common/QueryNormalizer.cs	Tue Jun  1 11:50:55 2010
@@ -179,7 +179,7 @@
             index++;
 
             // now mark all the other tokens as no output
-            while (index <= parenIndices[parenIndices.Count - 1])
+            while (index < parenIndices[parenIndices.Count - 1])
                 tok[index++].Output = false;
         }
[2 Jun 2010 6:03] Tonci Grgin 
Yvan, same as for Bug#53865... I do need a complete test case for this before checking it out regardless of how clear this all looks to you.
I can reproduce your problem by, let's say, improperly forming a long string value, i.e. instead of writing
 cmd.CommandText = "SELECT CONCAT('2010-05-27 16:42:57 Good writeup, but I dont think it has to be that complicated - at first. Theres still an issue that theres an implicit hierarchy, even in the interfaces."+

I can do
 cmd.CommandText = "SELECT CONCAT('2010-05-27 16:42:57 Good writeup, but I don't think it has to be that complicated - at first. There's still an issue that there's an implicit hierarchy, even in the interfaces."+

(check the "don't" and 2x "There's" which actually break my long string!) but that is not a bug in parser...
[2 Jun 2010 6:04] Tonci Grgin 
And yes, could you test against latest release please? I am obliged to do so.
[16 Jun 2010 12:41] Tonci Grgin 
-		this	{MySql.Data.Common.QueryNormalizer}	MySql.Data.Common.QueryNormalizer
		fullSql	"INSERT INTO bug54152 VALUES (1, 1, 'name 1', 1, 1, 1, 1, 1, 1, 1), (2,2,'name 2',2,2,2,2,2,2,2), (3,3,'name 3',3,3,3,3,3,3,3), (4,4,'name 4',4,4,4,4,4,4,4), (5,5,'name 5',5,5,5,5,5,5,5), (6,6,'name 6',6,6,6,6,6,6,6), (7,7,'name 7',7,7,7,7,7,7,7), (8,8,'name 8',8,8,8,8,8,8,8), (9,9,'name 9',9,9,9,9,9"	string
		pos	300	int

Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
    at QueryNormalizer.CollapseValueList(List`1 tok, Int32& pos)  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\common\QueryNormalizer.cs(158)
    at QueryNormalizer.CollapseValueLists(List`1 tok)  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\common\QueryNormalizer.cs(144)
    at QueryNormalizer.Normalize(String sql)  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\common\QueryNormalizer.cs(64)
    at TracingDriver.SendQuery(MySqlPacket p)  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\TracingDriver.cs(69)
    at Statement.ExecuteNext()  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\Statement.cs(95)
    at PreparableStatement.ExecuteNext()  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\PreparableStatement.cs(167)
    at Statement.Execute()  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\Statement.cs(85)
    at PreparableStatement.Execute()  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\PreparableStatement.cs(122)
    at MySqlCommand.ExecuteReader(CommandBehavior behavior)  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\command.cs(421)
    at MySqlCommand.ExecuteReader()  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\command.cs(344)
    at MySqlCommand.ExecuteNonQuery()  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\command.cs(302)
    at MySqlDataAdapter.ExecuteBatch()  E:\bzr-repos\connectors\cNET\clones\trunk\MySql.Data\Provider\Source\dataadapter.cs(170)
    at DbDataAdapter.UpdateBatchExecute(BatchCommandInfo[] batchCommands, Int32 commandCount, RowUpdatedEventArgs rowUpdatedEvent)  
    at DbDataAdapter.Update(DataRow[] dataRows, DataTableMapping tableMapping)  
    at DbDataAdapter.UpdateFromDataTable(DataTable dataTable, DataTableMapping tableMapping)  
    at DbDataAdapter.Update(DataTable dataTable)
[16 Jun 2010 12:42] Tonci Grgin 
Test case:
            MySqlConnection conn = new MySqlConnection("Server=**;Database=test;Uid=**;Pwd=**;allow batch=true;logging=true");
            conn.Open();
            MySqlCommand cmd = new MySqlCommand("DROP TABLE IF EXISTS bug54152", conn);
            cmd.ExecuteNonQuery();
            cmd.CommandText = "CREATE TABLE bug54152 (id INT, expr INT,name VARCHAR(20), fld4 VARCHAR(10), fld5 VARCHAR(10), fld6 VARCHAR(10), fld7 VARCHAR(10), fld8 VARCHAR(10), fld9 VARCHAR(10), fld10 VARCHAR(10), PRIMARY KEY(id))";
            cmd.ExecuteNonQuery();

            int changed = 0;

            MySqlDataAdapter da = new MySqlDataAdapter("SELECT * FROM bug54152", conn);
            MySqlCommand ins = new MySqlCommand("INSERT INTO bug54152 VALUES (?p1, ?p2, ?p3, ?p4, ?p5, ?p6, ?p7, ?p8, ?p9, ?p10)", conn);
            ins.UpdatedRowSource = UpdateRowSource.None;
            ins.Parameters.Add("?p1", MySqlDbType.Int32).SourceColumn = "id";
            ins.Parameters.Add("?p2", MySqlDbType.Int32).SourceColumn = "expr";
            ins.Parameters.Add("?p3", MySqlDbType.VarChar, 20).SourceColumn = "name";
            ins.Parameters.Add("?p4", MySqlDbType.Int32).SourceColumn = "fld4";
            ins.Parameters.Add("?p5", MySqlDbType.Int32).SourceColumn = "fld5";
            ins.Parameters.Add("?p6", MySqlDbType.Int32).SourceColumn = "fld6";
            ins.Parameters.Add("?p7", MySqlDbType.Int32).SourceColumn = "fld7";
            ins.Parameters.Add("?p8", MySqlDbType.Int32).SourceColumn = "fld8";
            ins.Parameters.Add("?p9", MySqlDbType.Int32).SourceColumn = "fld9";
            ins.Parameters.Add("?p10", MySqlDbType.Int32).SourceColumn = "fld10";
            DataTable dt = new DataTable();
            da.Fill(dt);
            da.InsertCommand = ins;
            da.UpdateBatchSize = 10;
            for (int i = 1; i <= 10; i++)
            {
                DataRow row = dt.NewRow();
                row["id"] = i;
                row["expr"] = i;
                row["name"] = "name " + i;
                dt.Rows.Add(row);
                row["fld4"] = i;
                row["fld5"] = i;
                row["fld6"] = i;
                row["fld7"] = i;
                row["fld8"] = i;
                row["fld9"] = i;
                row["fld10"] = i;
            }

            changed = da.Update(dt);
            Console.Out.WriteLine("INSERT INTO bug54152: {0} rows", changed);
[16 Jun 2010 12:43] Tonci Grgin 
This should be a duplicate of another Yvan's bug (Bug#53865) as the code works as expected with logging=false.

Yvan, I think you can see now how important it is to attach *test case* showing me what you actually did...
[16 Jun 2010 18:03] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/111312

859 Vladislav Vaintroub	2010-06-16
      Bug #53865 , Bug #54152 -fixed crashes in long queries when logging is turned on.
      Crashes mostly result from the assumption that query is welll-formed, the assumption will be wrong in most cases, since query is  truncated to 300 chars,
[16 Jun 2010 18:07] Vladislav Vaintroub 
queued to 6.2+
[24 Jun 2010 10:16] Vladislav Vaintroub 
fixed in 6.0.7, 6.1.5, 6.2.4, and 6.3.3+
[28 Jun 2010 12:58] Tony Bedford 
An entry has been added to the 6.0.7, 6.1.5, 6.2.4, and 6.3.3 changelogs:

The method MySql.Data.Common.QueryNormalizer.CollapseValueList generated an ArgumentOutOfRangeException.