MySQL Bugs: #34422: File system write failed during LogFileOperationRecord state 17. OS errno: 5

Bug #34422	File system write failed during LogFileOperationRecord state 17. OS errno: 5
Submitted:	8 Feb 2008 16:09	Modified:	27 Mar 2014 15:40
Reporter:	Cyril SCETBON	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	5.1.44-ndb-7.1.4b	OS:	Linux (debian etch)
Assigned to:		CPU Architecture:	Any
Tags:	cluster, crash, ndb

Description:
error in ndb_error.log on a node since 3 days:

Time: Monday 28 January 2008 - 10:01:38
Status: Temporary error, restart node
Message: Temporary on access to file (Internal error, programming error or missing error message, please report a bug)
Error: 2809
Error data: DBLQH: File system write failed during LogFileOperationRecord state 17. OS errno: 5
Error object: DBLQH (Line: 12697) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 312
Trace: /SPP/data/ndb_25_trace.log.2
Version: Version 5.1.22 (rc)
***EOM***
                                        
Time: Monday 4 February 2008 - 14:37:00
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: suma/Suma.cpp
Error object: SUMA (Line: 3168) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 30946
Trace: /SPP/data/ndb_25_trace.log.3
Version: Version 5.1.22 (rc)
***EOM***
                                                                                               
Time: Wednesday 6 February 2008 - 18:18:17
Status: Temporary error, restart node
Message: Temporary on access to file (Internal error, programming error or missing error message, please report a bug)
Error: 2809
Error data: DBLQH: File system write failed during LogFileOperationRecord state 17. OS errno: 5
Error object: DBLQH (Line: 12697) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 21285
Trace: /SPP/data/ndb_25_trace.log.4
Version: Version 5.1.22 (rc)
***EOM***

How to repeat:
none

ndb_trace files

Attachment: trace.zip (application/x-zip-compressed, text), 350.89 KiB.

Operating system error code 5 indicates a file
system or storage device level error:

  $  perror 5
  OS error code   5:  Input/output error

so this actually looks more like a system problem
than a MySQL bug.

To fully analyze this kind of bugs we need the cluster 
configuration file and the complete cluster log files:

- the management nodes ndb_#_cluster.log files
- all data nodes ndb_#_error.log files
- all data nodes ndb_#_out.log files
- all data nodes ndb_#_trace.log.* files

and usually also 

- all mysql nodes error log files 

Collecting these files can be automated using
the NDB error-reporter utility which is part 
of the cluster tools distribution and documented
in our online manual:

http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-utilities-ndb-error-reporter.html

> so this actually looks more like a system problem than a MySQL bug.
That's what we supposed but no error is logged by the OS (dmesg, messages,syslog or others) and the output log files (trace files and others) are on the same device and are written !!! So why did we get the error logged on the device that raised I/O errors ?

Cyril,

thank you for the feedback.

Could you please provide requested information:

- the management nodes ndb_#_cluster.log files
- all data nodes ndb_#_error.log files
- all data nodes ndb_#_out.log files
- all data nodes ndb_#_trace.log.* files
- all mysql nodes error log files

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I just hit exactly the same bug:

Time: Thursday 26 February 2009 - 21:11:35
Status: Temporary error, restart node
Message: Temporary on access to file (Internal error, programming error or missi
ng error message, please report a bug)
Error: 2809
Error data: DBLQH: File system write failed during LogFileOperationRecord state
17. OS errno: 5
Error object: DBLQH (Line: 13086) 0x0000000e
Program: /ndb/local/libexec/ndbd
Pid: 4348
Trace: /ndb/mysql-cluster/ndb_29_trace.log.1
Version: mysql-5.1.32 ndb-6.3.23-GA
***EOM***

And we solved the issue by setting "ODirect = 0" in the config.ini file.  It looks like there is a odd behavior with ODirect = 1.  The cluster has 24 ndbd nodes,  I don't have all the log files.

Yves

We have not met this bug anymore since we are in 5.1.27-ndb-6.3.17

Hi we actually hit this bug for the third time. I send you our our logs and configuration file.

any news ?

setting ODirect=0 does not solve our issue. Any other idea ?

version changed

we can't use the replicated site until this bug is solved !

we upgraded our slave cluster in our architecture (1 cluster replicated over another one). We still met the same error on a datanode:

Time: Friday 2 July 2010 - 10:02:04
Status: Temporary error, restart node
Message: Temporary on access to file (Internal error, programming error or
missing error message, please report a bug)
Error: 2809
Error data: DBLQH: File system write failed during LogFileOperationRecord
state 17. OS errno: 5
Error object: DBLQH (Line: 14516) 0x0000000e
Program: /usr/sbin/ndbd
Pid: 30951
Version: mysql-5.1.44 ndb-7.1.4b
Trace: /data/ndb_5_trace.log.2
***EOM***

Hi,

os error 5 is "Input/output error",
  this is something that we can't/don't handle
  gracefully...

do you have hw errors/bad sectors?
what os(kernel) do you run

/Jonas

Hi again,

I quite sure that another user had this...and it disappeared when they upgrade
linux kernel...

If you have more information it would be appreciated. Each time we met this error there was no information in the system log files. we checked our disks and nothing appeared to be in error.

so what os version do you use ?
what filesystem do you use ?

/Jonas

debian/etch + ext3 filesystem

Is upgrading to a newer linux kernel an option?
Google said that etch used 2.6.18 an was released around 2007...

/Jonas

not yet :(
we're using a 2.6.24 kernel but we'll try it on ubuntu lucid next year. Maybe the error will disappear, but I don't know. No one else met this error ? it's weird cause we only met it with mysql cluster and no other software.

Thank you for the feedback.

Did you update the kernel as was planned? Did it solve the error?

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Sorry I can't talk about it anymore cause I changed position and was not able to test it again ..

Thank you for the feedback.

I can only set status of this report as "Can't repeat" in this case. If you hit same issue again with newer kernel feel free to reopen it.