MySQL Bugs: #68983: Fatal error encountered during data read

Bug #68983	Fatal error encountered during data read
Submitted:	17 Apr 2013 10:05	Modified:	10 May 2016 11:42
Reporter:	Louis at oracle	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	Connector / NET	Severity:	S2 (Serious)
Version:	e.g. 6.7.2	OS:	Windows (Windwos7 64 bit)
Assigned to:		CPU Architecture:	Any
Tags:	FATAL ERROR, read

Description:
Hello,

When processing data form innodb-table-1 storing results into innodb-table-2

I encouter the "Fatal error encountered during data read" bug.

Both tables contain Multi million records especially the second one

Louis

How to repeat:
- windows7 64 bit
- MySQL 5.6.10 64 bit server
- 6.7.x .net adapter probably also other versions
- visual studio 2012 (Visual Basic) 

- I added the stopwatch to the loop to monitor the readtimes as specially at the moment of the problem
- In fact I did expect to see timeouts, which turns out to be a wrong assumption
(the problem message takes a few ms to arrise)
- I did raise some time out counters before, assuming that whas causing the problem (not true), that dit help a bit !??, but did not solve the problem
- there seams to be a relation between the "high number" of inserts on table-2 and the read problem related to table-1 directly after that
 

code structure

rdrSourceTab = mysqlreader on innodb-table-1

try

MyStopWatch.Start()
MyStopWatch.Restart()
While rdrSourceTab.Read
MyStopWatch.Stop()

       Do Some processing 
       Insert 0 .... 1000 rows into innodb-table-2
       while
          DBcmd.CommandText = sqlInsertRow
          NoInserted = DBcmd.ExecuteNonQuery()
       Wend (insert_loop)

MyStopWatch.Restart()
Wend (read_loop)

catch

@At the moment of the error
@StopWatch:0000.003.4 (3.4 ms) Err:5 Fatal error encountered during data read.

end try

Suggested fix:
- in the past using ODBC I did not encouter this problem

- this is of course a critical bug, but since the .net adapter is not mature I used severity "serieus". The work arround is using ODBC

Hello,

Some extra info
- I added another timer to the loop to measure the time of the inserts just before the read error occurs. That time is extremely low. I asume there is already a problem before the read.
- I use multiple connections. E.g. one for the read and one for the inserts
- The insert connection uses shared memory the other TCP. Reason for that is
* shared memory is a bit faster, so I used that for the most important connection
* there is a bug in the shared memory implementation. two connections via shared memory is a NO GO    
- I did update to 5.6.11. The problem is unchanged
- reads take about 1/10 of the time of inserts
- the program is waiting for the DB (with the read + insert statements) about 30% of the execution time
(where the program is spending some extra time due to the debug code added, trying to locate the read problem and to measure performance)

Louis

Hello,

I have been doing more testing. 
- I verified if the inset/update (in this case update), just before the readproblem was executed correctly. ==> I is. 
- I switch back to 6.6.5.net adapter to check if that together with 5.6.11 solves the problem. ==> no 
(note that i switched to 6.7.1 before, because 6.6.5. was not working, it still is not)
- the 1000 updates + inserts (mostly updates) just before the read problem did take 391 ms. So quite normal
- the faulty reed took 2.3 ms so normal / a bit long as wel

So, do not kwow what more to test. I do not think I can test deeper than this.
It just looks like the connection has some error / is gone or the task listening to the connection is gone or hanging. SOmething in that direction

Louis

I think we can call this a duplicate of bug #68827

Hello,

For info.

Today I tested the fatal read problem with:
- mysql-connector-net-6.7.2 in conjunction with
- mysql-5.6.11-winx64

Regrettable, the problem is still there.

No Rows processed just before the problem:1000
ReadStopWatch:0000.013.4  (error message returns 13.4 mseconds after entering the read statement
PrevInsertStopWatch:0000.375.9 last processed update or insert took 375.9 mseconds (376 ms is not fast but far away from timeouts)
ErrorMessage: Err:5 Fatal error encountered during data read.

Louis

For Info

I also tested with
- mysql-5.7.1-m11-winx64 and
- mysql-connector-net-6.7.2 

Same problem

Louis

Hello,

Since I raised this error a couple of weeks ago, I did a lot of testing, with latest server versions and latest connector versions, adding extra debug code etc.

In all cases I get the same (for me) reproducable bug. 

I assume the bug is related to the heavy inserts and updates on the server via other connections, just before the fatal read error. 

For info I am using one thread per connection and I am using only a small number of local connections.

Since the bug is still not assigned, I reased the severity, since it is a fatal error.

Louis
Perhaps a good idea to define extra error conditions in the source in order to identify the problem location more exactly.

Hello,

I might have found a clue. I raised the time out levels to an extreme level:
net_write_timeout = 60 => 99999 (seconds)
net_read_timeout = 30 => 99999 (seconds)
And surprisently that seems to help.

Where the original timeouts where allready way beyond the read interval times I measured (0.5 seconds).

How is that possible??, i have been thinking. 
- The reader is reading next item every 0.5 seconds
- the timeouts 100 times bigger

I do not know for sure but the answer may be catching. Suppose the following
There are three processes. 
- (1) the reading client proces ^.net adapter^
- (2) the server who guards the connection
- (3) the application reading from the ^.net adapter^
- suppose the mysql reader is ^smart^ and is fetching 1000 lines in advance
- the application (3) is reading every 0.5 a second the next item from the .net adapter (2)
- so next fetch from the server will be after 1000 * 0.5 seconds = 500 seconds
- that is well beyond the 60 seconds timeout from the server (1)
 
Not sure but this **might** explain the problem.

 
Louis

PS. Note that there is another slightly related isssue. I noticed that it is impossible to refill a data adapter without implicit opening and closing the connection used. That is extremely stupid. It forces me to assign a separate connections for reader and data adapter. Which in the end is more or less the birth of the problem described in this report.

Hello,

I changed the severity back to serious, since I have a working solution now.

Louis

Hi Louis,

I'd like to discard that this is a pooling issue. Can you please add 'pooling=false;' to the connection strings you use and tell me if you have the same behaviour?

Thank you.

Hello,

I changed the connection strings and the timers in my.ini and published the project

"SERVER=localhost;Port=3307;DATABASE=<mydb>;UID=IN;PASSWORD=<mypw>;pooling=false;"
net_write_timeout = 60
net_read_timeout = 30

Than I ran the project ==> not good

For check I changed the my.ini
net_write_timeout = 99999
net_read_timeout = 99999

Ran the project again ==> works!

So the problem is probably not pooling related.

Louis
PS in the past I have been expirimenting with the CommandTimeout parameter in the application code. Changing that parameter does not solve anything.

Hello Louis Breda van,

Thank you for the bug report.
This is most likely duplicate of Bug #61807, please see Bug #61807.

Thanks,
Chiranjeevi.