MySQL Bugs: #37958: test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.

Bug #37958	test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.
Submitted:	8 Jul 2008 9:04	Modified:	17 Oct 2008 17:48
Reporter:	Mattias Jonsson	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	5.1, 6.0 bzr	OS:	Any (Linux, Mac OS X)
Assigned to:	Ingo Strüwing	CPU Architecture:	Any

Description:
When running test plugin on my MacBook with Mac OS X 10.5.4 it crashes on the main.plugin.

Here is the backtrace at the crash:
Program received signal EXC_ARITHMETIC, Arithmetic exception.
0x91d07466 in hash_search ()
(gdb) bt
#0  0x91d07466 in hash_search ()
#1  0x00ce30b6 in get_share (table_name=0x10724a8 "./test/t1", table=0x1071a18) at ha_example.cc:177
#2  0x00ce343e in ha_example::open (this=0x1074628, name=0x10724a8 "./test/t1", mode=2, test_if_locked=2) at ha_example.cc:294
#3  0x00224c55 in handler::ha_open (this=0x1074628, table_arg=0x1071a18, name=0x10724a8 "./test/t1", mode=2, test_if_locked=2) at handler.cc:1988
#4  0x0015e7ff in open_table_from_share (thd=0x1026418, share=0x1072228, alias=0x1069b50 "t1", db_stat=39, prgflag=44, ha_open_flags=0, outparam=0x1071a18, is_create_table=false) at table.cc:1858
#5  0x0015115f in open_unireg_entry (thd=0x1026418, entry=0x1071a18, table_list=0x1069b58, alias=0x1069b50 "t1", cache_key=0xb00616fd "test", cache_key_length=8, mem_root=0xb0061924, flags=0) at sql_base.cc:3879
#6  0x00153df3 in open_table (thd=0x1026418, table_list=0x1069b58, mem_root=0xb0061924, refresh=0xb0061967, flags=0) at sql_base.cc:2902
#7  0x00154974 in open_tables (thd=0x1026418, start=0xb00619f4, counter=0xb00619d8, flags=0) at sql_base.cc:4513
#8  0x001551c7 in open_and_lock_tables_derived (thd=0x1026418, tables=0x1069b58, derived=true) at sql_base.cc:4913
#9  0x00113946 in open_and_lock_tables (thd=0x1026418, tables=0x1069b58) at mysql_priv.h:1529
#10 0x00100729 in execute_sqlcom_select (thd=0x1026418, all_tables=0x1069b58) at sql_parse.cc:4730
#11 0x001077d2 in mysql_execute_command (thd=0x1026418) at sql_parse.cc:2073
#12 0x00110c1e in mysql_parse (thd=0x1026418, inBuf=0x1069a28 "SELECT * FROM t1", length=16, found_semicolon=0xb0062df4) at sql_parse.cc:5650
#13 0x00111e87 in dispatch_command (command=COM_QUERY, thd=0x1026418, packet=0xcd3019 "SELECT * FROM t1", packet_length=16) at sql_parse.cc:1137
#14 0x001130fd in do_command (thd=0x1026418) at sql_parse.cc:794
#15 0x000fec11 in handle_one_connection (arg=0x1026418) at sql_connect.cc:1115
#16 0x91c9b6f5 in _pthread_start ()
#17 0x91c9b5b2 in thread_start ()
(gdb) 

and the test output:
main.plugin                    [ fail ]

mysqltest: At line 17: query 'SELECT * FROM t1' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
CREATE TABLE t1(a int) ENGINE=EXAMPLE;
Warnings:
Warning	1286	Unknown table engine 'EXAMPLE'
Warning	1266	Using storage engine MyISAM for table 't1'
DROP TABLE t1;
INSTALL PLUGIN example SONAME 'ha_example.so';
INSTALL PLUGIN EXAMPLE SONAME 'ha_example.so';
ERROR HY000: Function 'EXAMPLE' already exists
UNINSTALL PLUGIN example;
INSTALL PLUGIN example SONAME 'ha_example.so';
CREATE TABLE t1(a int) ENGINE=EXAMPLE;
SELECT * FROM t1;

More results from queries before failure can be found in /Users/mattiasj/clones/bzrroot/topush2-51-bugteam/mysql-test/var/log/plugin.log

How to repeat:
Run test main.plugin on an Intel Mac OS X machine

Suggested fix:
Have not investigated further than the backtrace.

Thank you for the report.

I can not repeat described behavior. Please indicate which package you ues and if you built server yourself provide your configure options.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

It exists in both mysql-5.1-bugteam and mysql-6.0-bugteam on my machine:
Darwin witty 9.4.0 Darwin Kernel Version 9.4.0: Mon Jun  9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386 i386

i.e. Mac OSX 10.5.4 macbook 2.2 GHz 4 GB RAM...

Thank you for the feedback.

Verified as described. Bug is repeatable on Linux with main tree too.

To repeat run test several times:

./mtr  plugin plugin plugin plugin plugin plugin plugin plugin

The problem lies in the implementation of the handlerton-to-plugin mapping. We use a fixed-size array to hold a plugin reference for each handlerton. On install of a handler plugin, we allocate a new slot of the array. On uninstall we do not free it. After some uninstall/install cycles the array overflows. We do not check for overflow.

Implementing a check for overflow stopped the crashes, but kept the test case failing after every fourth repetition as we ran into the limit. We need to free the array slot at uninstall to be able do do arbitrary numbers of uninstall/install cycles.

Manually switching to "Patch pending" as the bug database missed my commit email:

http://lists.mysql.com/commits/53535

 2735 Ingo Struewing	2008-09-08
      Bug#37958 - test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.
      
      The problem was improper implementation of the handlerton-to-plugin mapping.
      We use a fixed-size array to hold a plugin reference for each handlerton.
      On every install of a handler plugin, we allocated a new slot of the array.
      On uninstall we did not free it. After some uninstall/install cycles
      the array overflowed. We did not check for overflow.
      
      One fix is to check for overflow to stop the crashes.
      
      Another fix is to free the array slot at uninstall and search for a free slot
      at plugin install.

As Sergey rightfully suspected, the former patch did not fix the original problem, reported by Mattias. It fixed only the problem reported by Sveta, which turned out to be independent. So here is the second patch.
Please review.

Again BugDB did not pick up my commit email:

http://lists.mysql.com/commits/53722

2736 Ingo Struewing	2008-09-10
      Bug#37958 - test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.
      
      The server crashed on Mac OS X when trying to access an EXAMPLE table
      after the EXAMPLE plugin was installed.
      
      It turned out that the dynamically loaded EXAMPLE plugin called the
      function hash_earch() from a Mac OS X system library, instead of
      hash_earch() from MySQL's mysys library. Makefile.am in storage/example
      does not include libmysys. So the Mac OS X linker arranged the hash_search()
      function to be linked to the system library when the shared object is
      loaded.
      
      One possible solution would be to include libmysys into the linkage of
      dynamic plugins. But then we must have a libmysys.so, which must be
      used by the server too. This could have a minimal performance impact,
      but foremost the change seems to bee too risky at the current state of
      MySQL 5.1.
      
      The selected solution is to rename MySQL's hash_search() to my_hash_search()
      like it has been done before with hash_insert() and hash_reset().
      
      Since this is the third time, we need to rename a hash_*() function,
      I did renamed all hash_*() functions to my_hash_*().
      
      To avoid changing a zillion calls to these functions, and announcing
      this to hundreds of developers, I added defines that map the old names
      to the new names.

I approve the second patch.

Patch approved, but BEFORE pushing, get clearance from the Storage Engine API team.

(I have verified that the patch works on Mac OS X 10.5.4 intel).

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54649

2746 Ingo Struewing	2008-09-29
      Bug#37958 - test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.
      
      This patch contains fixes for two problems:
      
      1. As originally reported, the server crashed on Mac OS X when trying to access
         an EXAMPLE table after the EXAMPLE plugin was installed.
            
         It turned out that the dynamically loaded EXAMPLE plugin called the
         function hash_earch() from a Mac OS X system library, instead of
         hash_earch() from MySQL's mysys library. Makefile.am in storage/example
         does not include libmysys. So the Mac OS X linker arranged the hash_search()
         function to be linked to the system library when the shared object is
         loaded.
            
         One possible solution would be to include libmysys into the linkage of
         dynamic plugins. But then we must have a libmysys.so, which must be
         used by the server too. This could have a minimal performance impact,
         but foremost the change seems to bee too risky at the current state of
         MySQL 5.1.
            
         The selected solution is to rename MySQL's hash_search() to my_hash_search()
         like it has been done before with hash_insert() and hash_reset().
            
         Since this is the third time, we need to rename a hash_*() function,
         I did renamed all hash_*() functions to my_hash_*().
            
         To avoid changing a zillion calls to these functions, and announcing
         this to hundreds of developers, I added defines that map the old names
         to the new names.
      
         This change is in hash.h and hash.c.
      
      2. The other problem was improper implementation of the handlerton-to-plugin
         mapping. We use a fixed-size array to hold a plugin reference for each
         handlerton. On every install of a handler plugin, we allocated a new slot
         of the array. On uninstall we did not free it. After some uninstall/install
         cycles the array overflowed. We did not check for overflow.
            
         One fix is to check for overflow to stop the crashes.
            
         Another fix is to free the array slot at uninstall and search for a free slot
         at plugin install.
      
         This change is in handler.cc.

Queued to 6.0-bugteam and 5.1-bugteam.

Noted in 5.1.29 changelog.

The handlerton-to-plugin mapping implementation did not free handler
plugin references when the plugin was uninstalled, resulting in a
server crash after several install/uninstall cycles. Also, on Mac OS
X, the server crashed when trying to access an EXAMPLE table after
the EXAMPLE plugin was installed.  

Leaving report status unchanged; this is early documentation of an upcoming push into 5.1.29.

Pushed into 5.1.30  (revid:ingo.struewing@sun.com-20081001102115-q66y9ylp3psrtqd5) (version source revid:mats@sun.com-20081008113713-2vxny72m5w1tywoi) (pib:4)

Setting report to NDI pending push into 6.0.x.

Pushed into 6.0.8-alpha  (revid:ingo.struewing@sun.com-20081001102115-q66y9ylp3psrtqd5) (version source revid:sergey.glukhov@sun.com-20081002112911-2ao596le24bpawx5) (pib:5)

Noted in 6.0.8 changelog.

Pushed into 5.1.29-ndb-6.2.17  (revid:ingo.struewing@sun.com-20081001102115-q66y9ylp3psrtqd5) (version source revid:tomas.ulin@sun.com-20081028140209-u4emkk1xphi5tkfb) (pib:5)

Pushed into 5.1.29-ndb-6.3.19  (revid:ingo.struewing@sun.com-20081001102115-q66y9ylp3psrtqd5) (version source revid:tomas.ulin@sun.com-20081028194045-0353yg8cvd2c7dd1) (pib:5)

Pushed into 5.1.29-ndb-6.4.0  (revid:ingo.struewing@sun.com-20081001102115-q66y9ylp3psrtqd5) (version source revid:jonas@mysql.com-20081101082305-qx5a1bj0z7i8ueys) (pib:5)