Bug #61575 InnoDB: Error: io_setup() failed
Submitted: 21 Jun 2011 7:24 Modified: 15 May 2012 19:39
Reporter: Marko Mäkelä Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:5.5+ OS:Linux
Assigned to: Inaam Rana CPU Architecture:Any
Tags: aio, linux

[21 Jun 2011 7:24] Marko Mäkelä
Description:
This is from http://forums.mysql.com/read.php?11,406737,406737

MySQL Forums :: Install :: Install fails with error io_setup() failed seg fault

According to the Linux manual page for io_setup(), EAGAIN means that it is useless to retry with the same parameters: "The specified nr_events exceeds the user's limit of available events." InnoDB seems to think otherwise:

110208 11:56:51 InnoDB: Using Linux native AIO
110208 11:56:51 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
110208 11:56:54 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_native_aio = off in my.cnf
110208 11:56:54 InnoDB: Initializing buffer pool, size = 128.0M
110208 11:56:54 InnoDB: Completed initialization of buffer pool
110208 11:56:54 - mysqld got signal 11 ;

How to repeat:
Compile InnoDB on a Linux system that has libaio installed. Do not disable innodb_use_native_aio.

sudo sysctl fs.aio-max-nr=1
mysqld

This will lead to array==NULL being dereferenced in os_aio_linux_handle(), called by fil_aio_wait() in an io_handler_thread().

	/* Find the array and the local segment. */
	segment = os_aio_get_array_and_local_segment(&array, global_seg);
	n = array->n_slots / array->n_segments;

Suggested fix:
In innobase_start_or_create_for_mysql(), adjust the io_limit to whatever is reported by sysctl, instead of hard-coding it at

io_limit = 8 * SRV_N_PENDING_IOS_PER_THREAD;

Consider making this a configureable start-up parameter.

Refuse to start up if os_aio_init() fails.
[22 Jun 2011 12:24] Stewart Smith
note that we have hit this with the default setting for Ubuntu kernels and running mtr (or other tests). I can quite trivially hit this on laptop/desktop.
[15 May 2012 19:39] John Russell
Adding to the descriptions of innodb_read_io_threads and 
innodb_write_io_threads: 

On Linux systems, running multiple MySQL servers (typically more than 
12) with default settings for innodb_read_io_threads, 
innodb_write_io_threads, and the Linux aio-max-nr setting can exceed 
system limits. Ideally, increase the aio-max-nr setting; as a 
workaround, you might reduce the settings for one or both of the 
MySQL configuration options.