Bug #19081 MySQL Server Assertion failure in file srv0srv.c line 2080
Submitted: 13 Apr 2006 18:34 Modified: 8 Nov 2006 15:54
Reporter: Brian Clark Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.1 beta 8 OS:Linux (Linux 2.4.20 ppc)
Assigned to: Marko Mäkelä CPU Architecture:Any

[13 Apr 2006 18:34] Brian Clark
Description:
System crashed with "Assertion failure in thread 3677162272 in file srv0srv.c line 2080" under heavy load.  System load was at 550 queries/sec and running a large batch insert from mysqldump file.  Queries were mainly inserts and updates.  Large error log available.

How to repeat:
Can provide mysql dump that was being loaded and examples of queries being run at the time.
[13 Apr 2006 18:45] MySQL Verification Team
Thank you for the bug report.

Yes please provide the dump and queries's script. You can upload
into a file with a name identifying this bug report:

ftp://ftp.mysql.com:/pub/mysql/upload

Thanks in advance.
[13 Apr 2006 23:49] Heikki Tuuri
Brian,

please send the .err log gzipped to heikki.tuuri@oracle.com

Regards,

Heikki
[18 Apr 2006 19:12] Brian Clark
The dump file and the .err log have been uploaded.
[9 Jun 2006 5:46] Heikki Tuuri
Hi!

I have now looked at the .err file. Looks like a thread has released an S-latch on an index tree latch though the thread did not have an S-latch there.

The index tree is probably in table 'items'.

Do you use partitioning or some other new features of 5.1?

Regards,

Heikki

=====================================
060413 13:31:05 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 43 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 197413, signal count 160615
--Thread 3724700448 has waited at btr0cur.c line 359 for 252.00 seconds the semaphore:
X-lock on RW-latch at 0x502ddac0 created in file dict0dict.c line 3662
a writer (thread id 3724700448) has reserved it in mode  wait exclusive
number of readers 4294967295, waiters flag 1
Last time read locked in file btr0cur.c line 366
Last time write locked in file btr0cur.c line 359
[9 Jun 2006 9:00] Heikki Tuuri
Marko,

please check if you can find an execution path where rw_lock_s_unlock is called when it should not.

Please change the assertion to ut_a(lock->reader_count > 0) in 5.1, so that we get a stack trace if this reappears.

Regards,

Heikki

sync0rw.ic:

/**********************************************************************
Releases a shared mode lock. */
UNIV_INLINE
void
rw_lock_s_unlock_func(
/*==================*/
	rw_lock_t*	lock	/* in: rw-lock */
#ifdef UNIV_SYNC_DEBUG
	,ulint		pass	/* in: pass value; != 0, if the lock may have
				been passed to another thread to unlock */
#endif
	)
{
	mutex_t*	mutex	= &(lock->mutex);
	ibool		sg	= FALSE;

	/* Acquire the mutex protecting the rw-lock fields */
	mutex_enter(mutex);

	/* Reset the shared lock by decrementing the reader count */

	ut_ad(lock->reader_count > 0);
	lock->reader_count--;
[9 Jun 2006 11:48] Marko Mäkelä
Heikki,
it seems that btr_cur_search_to_nth_level() will not reacquire btr_search_latch when inserting via the insert buffer. Here's the proposed patch to 4.1:

Index: btr/btr0cur.c
===================================================================
--- btr/btr0cur.c	(revision 32)
+++ btr/btr0cur.c	(working copy)
@@ -415,7 +415,7 @@ retry_page_get:		
 				/* Insertion to the insert buffer succeeded */
 				cursor->flag = BTR_CUR_INSERT_TO_IBUF;
 
-				return;
+				goto func_exit;
 			}
 
 			/* Insert to the insert buffer did not succeed:
@@ -523,6 +523,7 @@ retry_page_get:		
 						|| mode != PAGE_CUR_LE);
 	}
 
+func_exit:
 	if (has_search_latch) {
 		
 		rw_lock_s_lock(&btr_search_latch);
[9 Jun 2006 13:18] Heikki Tuuri
Marko,

we do not have btr_search_latch reserved in an insert. Please see the code fragment below. It is only reserved in row0sel.c.

But your proposed patch makes the code in btr0cur.c less fragile. Please put it to 5.0 and 5.1.

Regards,

Heikki

row_search_for_mysql() in 4.1.6:

                if (trx->mysql_n_tables_locked == 0
                    && prebuilt->select_lock_type == LOCK_NONE
                    && trx->isolation_level > TRX_ISO_READ_UNCOMMITTED
                    && trx->read_view) {

                        /* This is a SELECT query done as a consistent read,
                        and the read view has already been allocated:
                        let us try a search shortcut through the hash
                        index.
                        NOTE that we must also test that
                        mysql_n_tables_locked == 0, because this might
                        also be INSERT INTO ... SELECT ... or
                        CREATE TABLE ... SELECT ... . Our algorithm is
                        NOT prepared to inserts interleaved with the SELECT,
                        and if we try that, we can deadlock on the adaptive
                        hash index semaphore! */

#ifndef UNIV_SEARCH_DEBUG
                        if (!trx->has_search_latch) {
                                rw_lock_s_lock(&btr_search_latch);
                                trx->has_search_latch = TRUE;
                        }
#endif
                        shortcut = row_sel_try_search_shortcut_for_mysql(&rec,
                                                               prebuilt, &mtr);
[9 Jun 2006 15:23] Heikki Tuuri
Marko,

please put the ut_a() also to 5.0, so that we get more testing of this.

The hang could also be due to a bug in sync0* files. If there is a 'lost update' problem when updating the reader count in an rw_lock, it could explain both this bug report and http://bugs.mysql.com/bug.php?id=20358

In theory, also hardware faults could cause a 'lost update'.

Regards,

Heikki
[12 Jun 2006 14:35] Heikki Tuuri
Brian,

is this a 64-bit Linux version?

Regards,

Heikki
[8 Nov 2006 15:54] Heikki Tuuri
Changing the status to Can't repeat until we get more reports of this.
[5 May 2010 15:15] Bugs System
Pushed into 5.1.47 (revid:joro@sun.com-20100505145753-ivlt4hclbrjy8eye) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[28 May 2010 5:57] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100524190136-egaq7e8zgkwb9aqi) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (pib:16)
[28 May 2010 6:26] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100524190941-nuudpx60if25wsvx) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[28 May 2010 6:54] Bugs System
Pushed into 5.5.5-m3 (revid:alik@sun.com-20100524185725-c8k5q7v60i5nix3t) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 11:59] Bugs System
Pushed into 5.1.47-ndb-7.0.16 (revid:martin.skold@mysql.com-20100617114014-bva0dy24yyd67697) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 12:40] Bugs System
Pushed into 5.1.47-ndb-6.2.19 (revid:martin.skold@mysql.com-20100617115448-idrbic6gbki37h1c) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 13:25] Bugs System
Pushed into 5.1.47-ndb-6.3.35 (revid:martin.skold@mysql.com-20100617114611-61aqbb52j752y116) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)