Bug #59230 assert 0 row_upd_changes_ord_field_binary() in post-crash trx rollback or purge
Submitted: 30 Dec 2010 13:31 Modified: 10 Feb 2011 23:45
Reporter: Mikhail Izioumtchenko Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.1,5.5,5.6 OS:Any
Assigned to: Marko Mäkelä CPU Architecture:Any
Triage: Triaged: D1 (Critical) / R1 (None/Negligible) / E1 (None/Negligible)

[30 Dec 2010 13:31] Mikhail Izioumtchenko
Description:
stress/recovery testing, an assert like this happens:

row0upd.c line 1264
InnoDB: Failing assertion: 0

in row_upd_changes_ord_field_binary
row_undo_mod_upd_exist_sec
...
trx_rollback_or_clean_all_recovered

May be related to the fix to bug#55284 given the first
mysql-5.5-innodb revision I saw it is r3264.

How to repeat:
internal stress/recovery testing.
Reproducibility is fair with 5.5, seen once with 5.1 plugin,
I doubt it happens in trunk.
[26 Jan 2011 14:16] Marko Mäkelä
Michael, does this always occur during the rollback of recovered transactions? If so, I would suggest this patch:

=== modified file 'storage/innodb_plugin/row/row0upd.c'
--- storage/innodb_plugin/row/row0upd.c	revid:marko.makela@oracle.com-20110125134308-cle24ig6ukwe7zw2
+++ storage/innodb_plugin/row/row0upd.c	2011-01-26 14:14:27 +0000
@@ -1263,9 +1263,12 @@ row_upd_changes_ord_field_binary(
 
 			if (UNIV_LIKELY_NULL(buf)) {
 				if (UNIV_UNLIKELY(buf == field_ref_zero)) {
-					/* This should never happen, but
-					we try to fail safe here. */
-					ut_ad(0);
+					/* The externally stored field
+					was not written yet. This
+					record should only be seen by
+					recv_recovery_rollback_active(),
+					when the server had crashed before
+					storing the field. */
 					return(TRUE);
 				}
 

I would like to assert trx->is_recovered, but unfortunately this function does not get a trx parameter.
[26 Jan 2011 14:19] Marko Mäkelä
I forgot to add that I introduced this assertion when fixing Bug #55284. This debug assertion should only be triggered on DYNAMIC or COMPRESSED tables in the Barracuda format when the server crashed just at the right time before the restart.
[26 Jan 2011 15:47] Mikhail Izioumtchenko
to the best of my knowledge the assert only happens while we are rolling back
the transactions after the recovery. iirc I also made an experiment
that showed that removing the assert allowed the recovery to complete
successfully without any damage to the database. So we could just take
the assert removal. However I'd suggest improving the assert:

* try to pass a parameter indicating the txn was recovered. In this particular stack the next level caller is row_undo_mod_upd_exist_sec() which knows 'thr',
other callers in the 5.6 tree at least also look aware of it.
* at least assert on !we_are_doing_rollback_after_recovery_still, iirc there is such a variable out there.
[27 Jan 2011 9:52] Marko Mäkelä
Michael, please remove the "or purge" from the bug title if you have not seen this in purge. The assertion failure could be serious during purge.

I have a fix that adds some extra parameters in the debug build, so that the assertion can check that the condition occurs during the rollback of a recovered transaction. I will have to ask you to test my fix, because I am afraid that I cannot reproduce this bug myself.
[31 Jan 2011 7:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129997
[31 Jan 2011 7:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129998
[8 Feb 2011 17:36] Bugs System
Pushed into mysql-5.1 5.1.56 (revid:vasil.dimov@oracle.com-20110208172642-ekjin6otzuy47vn9) (version source revid:vasil.dimov@oracle.com-20110208172642-ekjin6otzuy47vn9) (merge vers: 5.1.56) (pib:24)
[8 Feb 2011 17:37] Bugs System
Pushed into mysql-trunk 5.6.2 (revid:vasil.dimov@oracle.com-20110208173442-ocy58fdcuew3xvex) (version source revid:vasil.dimov@oracle.com-20110208173331-fu0j2s14jbg915zu) (merge vers: 5.6.2) (pib:24)
[8 Feb 2011 17:38] Bugs System
Pushed into mysql-5.5 5.5.10 (revid:vasil.dimov@oracle.com-20110208173046-qsmzbrw1gppahx5o) (version source revid:vasil.dimov@oracle.com-20110208172800-tls70r2ot1i0dub7) (merge vers: 5.5.10) (pib:24)