MySQL Bugs: #19081: MySQL Server Assertion failure in file srv0srv.c line 2080

Bug #19081	MySQL Server Assertion failure in file srv0srv.c line 2080
Submitted:	13 Apr 2006 20:34	Modified:	8 Nov 2006 16:54
Reporter:	Brian Clark
Status:	Can't repeat
Category:	Server: InnoDB	Severity:	S1 (Critical)
Version:	5.1 beta 8	OS:	Linux (Linux 2.4.20 ppc)
Assigned to:	Marko Mäkelä	Target Version:

Description:
System crashed with "Assertion failure in thread 3677162272 in file srv0srv.c line 2080"
under heavy load.  System load was at 550 queries/sec and running a large batch insert
from mysqldump file.  Queries were mainly inserts and updates.  Large error log
available.

How to repeat:
Can provide mysql dump that was being loaded and examples of queries being run at the
time.

Thank you for the bug report.

Yes please provide the dump and queries's script. You can upload
into a file with a name identifying this bug report:

ftp://ftp.mysql.com:/pub/mysql/upload

Thanks in advance.

Brian,

please send the .err log gzipped to heikki.tuuri@oracle.com

Regards,

Heikki

The dump file and the .err log have been uploaded.

Hi!

I have now looked at the .err file. Looks like a thread has released an S-latch on an
index tree latch though the thread did not have an S-latch there.

The index tree is probably in table 'items'.

Do you use partitioning or some other new features of 5.1?

Regards,

Heikki

=====================================
060413 13:31:05 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 43 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 197413, signal count 160615
--Thread 3724700448 has waited at btr0cur.c line 359 for 252.00 seconds the semaphore:
X-lock on RW-latch at 0x502ddac0 created in file dict0dict.c line 3662
a writer (thread id 3724700448) has reserved it in mode  wait exclusive
number of readers 4294967295, waiters flag 1
Last time read locked in file btr0cur.c line 366
Last time write locked in file btr0cur.c line 359

Marko,

please check if you can find an execution path where rw_lock_s_unlock is called when it
should not.

Please change the assertion to ut_a(lock->reader_count > 0) in 5.1, so that we get a
stack trace if this reappears.

Regards,

Heikki

sync0rw.ic:

/**********************************************************************
Releases a shared mode lock. */
UNIV_INLINE
void
rw_lock_s_unlock_func(
/*==================*/
	rw_lock_t*	lock	/* in: rw-lock */
#ifdef UNIV_SYNC_DEBUG
	,ulint		pass	/* in: pass value; != 0, if the lock may have
				been passed to another thread to unlock */
#endif
	)
{
	mutex_t*	mutex	= &(lock->mutex);
	ibool		sg	= FALSE;

	/* Acquire the mutex protecting the rw-lock fields */
	mutex_enter(mutex);

	/* Reset the shared lock by decrementing the reader count */

	ut_ad(lock->reader_count > 0);
	lock->reader_count--;

Heikki,
it seems that btr_cur_search_to_nth_level() will not reacquire btr_search_latch when
inserting via the insert buffer. Here's the proposed patch to 4.1:

Index: btr/btr0cur.c
===================================================================
--- btr/btr0cur.c	(revision 32)
+++ btr/btr0cur.c	(working copy)
@@ -415,7 +415,7 @@ retry_page_get:		
 				/* Insertion to the insert buffer succeeded */
 				cursor->flag = BTR_CUR_INSERT_TO_IBUF;
 
-				return;
+				goto func_exit;
 			}
 
 			/* Insert to the insert buffer did not succeed:
@@ -523,6 +523,7 @@ retry_page_get:		
 						|| mode != PAGE_CUR_LE);
 	}
 
+func_exit:
 	if (has_search_latch) {
 		
 		rw_lock_s_lock(&btr_search_latch);

Marko,

we do not have btr_search_latch reserved in an insert. Please see the code fragment
below. It is only reserved in row0sel.c.

But your proposed patch makes the code in btr0cur.c less fragile. Please put it to 5.0
and 5.1.

Regards,

Heikki

row_search_for_mysql() in 4.1.6:

                if (trx->mysql_n_tables_locked == 0
                    && prebuilt->select_lock_type == LOCK_NONE
                    && trx->isolation_level > TRX_ISO_READ_UNCOMMITTED
                    && trx->read_view) {

                        /* This is a SELECT query done as a consistent read,
                        and the read view has already been allocated:
                        let us try a search shortcut through the hash
                        index.
                        NOTE that we must also test that
                        mysql_n_tables_locked == 0, because this might
                        also be INSERT INTO ... SELECT ... or
                        CREATE TABLE ... SELECT ... . Our algorithm is
                        NOT prepared to inserts interleaved with the SELECT,
                        and if we try that, we can deadlock on the adaptive
                        hash index semaphore! */

#ifndef UNIV_SEARCH_DEBUG
                        if (!trx->has_search_latch) {
                                rw_lock_s_lock(&btr_search_latch);
                                trx->has_search_latch = TRUE;
                        }
#endif
                        shortcut = row_sel_try_search_shortcut_for_mysql(&rec,
                                                               prebuilt, &mtr);

Marko,

please put the ut_a() also to 5.0, so that we get more testing of this.

The hang could also be due to a bug in sync0* files. If there is a 'lost update' problem
when updating the reader count in an rw_lock, it could explain both this bug report and
http://bugs.mysql.com/bug.php?id=20358

In theory, also hardware faults could cause a 'lost update'.

Regards,

Heikki

Brian,

is this a 64-bit Linux version?

Regards,

Heikki

Changing the status to Can't repeat until we get more reports of this.