MySQL Bugs: #59230: assert 0 row_upd_changes_ord_field_binary() in post-crash trx rollback or purge

Bug #59230	assert 0 row_upd_changes_ord_field_binary() in post-crash trx rollback or purge
Submitted:	30 Dec 2010 13:31	Modified:	10 Feb 2011 23:45
Reporter:	Mikhail Izioumtchenko	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.1,5.5,5.6	OS:	Any
Assigned to:	Marko Mäkelä	CPU Architecture:	Any

Description:
stress/recovery testing, an assert like this happens:

row0upd.c line 1264
InnoDB: Failing assertion: 0

in row_upd_changes_ord_field_binary
row_undo_mod_upd_exist_sec
...
trx_rollback_or_clean_all_recovered

May be related to the fix to bug#55284 given the first
mysql-5.5-innodb revision I saw it is r3264.

How to repeat:
internal stress/recovery testing.
Reproducibility is fair with 5.5, seen once with 5.1 plugin,
I doubt it happens in trunk.

Michael, does this always occur during the rollback of recovered transactions? If so, I would suggest this patch:

=== modified file 'storage/innodb_plugin/row/row0upd.c'
--- storage/innodb_plugin/row/row0upd.c	revid:marko.makela@oracle.com-20110125134308-cle24ig6ukwe7zw2
+++ storage/innodb_plugin/row/row0upd.c	2011-01-26 14:14:27 +0000
@@ -1263,9 +1263,12 @@ row_upd_changes_ord_field_binary(
 
 			if (UNIV_LIKELY_NULL(buf)) {
 				if (UNIV_UNLIKELY(buf == field_ref_zero)) {
-					/* This should never happen, but
-					we try to fail safe here. */
-					ut_ad(0);
+					/* The externally stored field
+					was not written yet. This
+					record should only be seen by
+					recv_recovery_rollback_active(),
+					when the server had crashed before
+					storing the field. */
 					return(TRUE);
 				}
 

I would like to assert trx->is_recovered, but unfortunately this function does not get a trx parameter.

I forgot to add that I introduced this assertion when fixing Bug #55284. This debug assertion should only be triggered on DYNAMIC or COMPRESSED tables in the Barracuda format when the server crashed just at the right time before the restart.

to the best of my knowledge the assert only happens while we are rolling back
the transactions after the recovery. iirc I also made an experiment
that showed that removing the assert allowed the recovery to complete
successfully without any damage to the database. So we could just take
the assert removal. However I'd suggest improving the assert:

* try to pass a parameter indicating the txn was recovered. In this particular stack the next level caller is row_undo_mod_upd_exist_sec() which knows 'thr',
other callers in the 5.6 tree at least also look aware of it.
* at least assert on !we_are_doing_rollback_after_recovery_still, iirc there is such a variable out there.

Michael, please remove the "or purge" from the bug title if you have not seen this in purge. The assertion failure could be serious during purge.

I have a fix that adds some extra parameters in the debug build, so that the assertion can check that the condition occurs during the rollback of a recovered transaction. I will have to ask you to test my fix, because I am afraid that I cannot reproduce this bug myself.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129997

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129998

Pushed into mysql-5.1 5.1.56 (revid:vasil.dimov@oracle.com-20110208172642-ekjin6otzuy47vn9) (version source revid:vasil.dimov@oracle.com-20110208172642-ekjin6otzuy47vn9) (merge vers: 5.1.56) (pib:24)

Pushed into mysql-trunk 5.6.2 (revid:vasil.dimov@oracle.com-20110208173442-ocy58fdcuew3xvex) (version source revid:vasil.dimov@oracle.com-20110208173331-fu0j2s14jbg915zu) (merge vers: 5.6.2) (pib:24)

Pushed into mysql-5.5 5.5.10 (revid:vasil.dimov@oracle.com-20110208173046-qsmzbrw1gppahx5o) (version source revid:vasil.dimov@oracle.com-20110208172800-tls70r2ot1i0dub7) (merge vers: 5.5.10) (pib:24)