Bug #54166 ha_rnd_init() result code not checked
Submitted: 2 Jun 2010 5:02 Modified: 29 Jul 2012 22:43
Reporter: Stewart Smith Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Storage Engine API Severity:S2 (Serious)
Version:5.1 OS:Any
Assigned to: CPU Architecture:Any
Triage: Triaged: D3 (Medium)

[2 Jun 2010 5:02] Stewart Smith
Description:
This is about 10-15 bugs.

If your storage engine has the nerve to return an error code from rnd_init() (as all documentation suggests you can, as well as the 'int' return type) a lot of places in the code don't check the return type, continue, and you can end up in a world of pain.

It's also the case for rnd_end and other handler functions, but this is the one that bit me first.

How to repeat:
This patch produces a nice list of where they are (gcc only of course, but easy to wrap in a macro).

--- ../mysql-5.1.46/sql/handler.h	2010-04-07 00:06:06.000000000 +1000
+++ sql/handler.h	2010-06-01 17:37:48.230727373 +1000
@@ -1188,7 +1188,7 @@
     inited=NONE;
     DBUG_RETURN(index_end());
   }
-  int ha_rnd_init(bool scan)
+  int ha_rnd_init(bool scan) __attribute__ ((warn_unused_result))
   {
     int result;
     DBUG_ENTER("ha_rnd_init");

Suggested fix:
-Werror and the above patch.
[2 Jun 2010 18:47] Sveta Smirnova
Thank you for the report.

Verified as described:

$cat bug54166.log  | grep ha_rnd_init
item_subselect.cc:2044: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_select.cc:10854: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_select.cc:13944: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_select.cc:14072: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_update.cc:1976: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
log_event.cc:9063: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
log_event_old.cc:843: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
log_event_old.cc:2487: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
records.cc:208: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
records.cc:245: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
records.cc:256: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
filesort.cc:538: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
handler.cc:2150: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
filesort.cc:538: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
handler.cc:2150: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
item_subselect.cc:2044: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
records.cc:208: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
records.cc:245: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
records.cc:256: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_select.cc:10854: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_select.cc:13944: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_select.cc:14072: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
sql_update.cc:1976: warning: ignoring return value of ‘int handler::ha_rnd_init(bool)’, declared with attribute warn_unused_result
[5 Jan 2011 16:16] Mark Callaghan
Are there plans to fix this?
[11 Jan 2011 11:53] Stewart Smith
MariaDB and Drizzle have implemented fixes (also for other handler calls).
[17 Feb 2011 7:08] Stewart Smith
index_init is also affected.

InnoDB has various workarounds (I've only audited parts of the codepath in Drizzle though, but it *should* be the same in MySQL 5.5 and thus okay) - see https://bugs.launchpad.net/drizzle/+bug/720552

MyISAM and NDB just set local vars, no problem.

Other engines may have issues if they can possibly return an error from index_init and then not preserve it for subsequent index calls.
[29 Jul 2012 22:43] Paul Dubois
Noted in 5.6.6 changelog. 

Code for the storage engine API did not check the return value from 
the ha_rnd_init() and ha_index_init() functions.