MySQL Bugs: #61575: InnoDB: Error: io

Bug #61575	InnoDB: Error: io_setup() failed
Submitted:	21 Jun 2011 7:24	Modified:	15 May 2012 19:39
Reporter:	Marko Mäkelä	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	5.5+	OS:	Linux
Assigned to:	Inaam Rana	CPU Architecture:	Any
Tags:	aio, linux

Description:
This is from http://forums.mysql.com/read.php?11,406737,406737

MySQL Forums :: Install :: Install fails with error io_setup() failed seg fault

According to the Linux manual page for io_setup(), EAGAIN means that it is useless to retry with the same parameters: "The specified nr_events exceeds the user's limit of available events." InnoDB seems to think otherwise:

110208 11:56:51 InnoDB: Using Linux native AIO
110208 11:56:51 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
110208 11:56:54 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_native_aio = off in my.cnf
110208 11:56:54 InnoDB: Initializing buffer pool, size = 128.0M
110208 11:56:54 InnoDB: Completed initialization of buffer pool
110208 11:56:54 - mysqld got signal 11 ;

How to repeat:
Compile InnoDB on a Linux system that has libaio installed. Do not disable innodb_use_native_aio.

sudo sysctl fs.aio-max-nr=1
mysqld

This will lead to array==NULL being dereferenced in os_aio_linux_handle(), called by fil_aio_wait() in an io_handler_thread().

	/* Find the array and the local segment. */
	segment = os_aio_get_array_and_local_segment(&array, global_seg);
	n = array->n_slots / array->n_segments;

Suggested fix:
In innobase_start_or_create_for_mysql(), adjust the io_limit to whatever is reported by sysctl, instead of hard-coding it at

io_limit = 8 * SRV_N_PENDING_IOS_PER_THREAD;

Consider making this a configureable start-up parameter.

Refuse to start up if os_aio_init() fails.

note that we have hit this with the default setting for Ubuntu kernels and running mtr (or other tests). I can quite trivially hit this on laptop/desktop.

Adding to the descriptions of innodb_read_io_threads and 
innodb_write_io_threads: 

On Linux systems, running multiple MySQL servers (typically more than 
12) with default settings for innodb_read_io_threads, 
innodb_write_io_threads, and the Linux aio-max-nr setting can exceed 
system limits. Ideally, increase the aio-max-nr setting; as a 
workaround, you might reduce the settings for one or both of the 
MySQL configuration options.