Bug #29870 Server crashes during sqlbench tests execution

Description:
During execution of sqlbench test suite against Falcon storage engine from Distrib 6.0.1-alpha server crashes, sometimes even without error message.

How to repeat:
Install Distrib 6.0.1-alpha, make falcon the default storage engine
Start sqlbench with following command:  ./run-all-tests --server=mysql --fast --log --user=root --hires
Wait for some time, usually server crashes on test-insert, but I couldn't prove that.

server version added

Could you please try to verify your failure with latest sources from our bk repository. 6.0.1-alpha is quite old already and we fixed quite some bugs in meantime.

Best regards,

Hakan

Hakan,

Actually we tried with fresh bk-tree build

Verified using latest Falcon code. Looks like that mysqld got killed by OOM from OS.

hakan@lu0011:~/work/mysql/mysql-5.1-falcon/sql-bench$ ./run-all-tests --server=mysql --fast --log --user=root --hires

Warning: Can't execute insert.  Check the file 'output/insert-mysql_fast-Linux_2.6.18_4_amd64_x86_64'
Warning: Can't execute select.  Check the file 'output/select-mysql_fast-Linux_2.6.18_4_amd64_x86_64'
Warning: Can't execute transactions.  Check the file 'output/transactions-mysql_fast-Linux_2.6.18_4_amd64_x86_64'
Warning: Can't execute wisconsin.  Check the file 'output/wisconsin-mysql_fast-Linux_2.6.18_4_amd64_x86_64'
Test finished. You can find the result in:
output/RUN-mysql_fast-Linux_2.6.18_4_amd64_x86_64

...
...
Estimated time for select_range_key2 (20000:36279): 721.823 wallclock secs ( 1.75 usr  0.11 sys +  0.00 cusr  0.00 csys =  1.86 CPU
)
Error occured with execute(select max(id) from bench1)
 -> Lost connection to MySQL server during query

Now failing at

Warning: Can't execute insert.  Check the file 'output/insert-mysql_fast-Linux_2.6.21_2_amd64_x86_64'

with

Time for update_with_key_prefix (100000):  74 wallclock secs ( 2.86 usr  0.83 sys +  0.00 cusr  0.00 csys =  3.69 CPU)

Testing update of all rows
Got error 305 'record memory is exhausted' from Falcon at ./test-insert line 1015.

Will try with bigger memory settings now.

With Falcon settings of
   falcon_record_memory_max = 768MB
   falcon_page_cache_size = 160MB

we see a crash now

lu0011:/data/mysql# resolve_stack_dump -s /tmp/mysqld.sym -n /tmp/mysqld.stack | c++filt
0x6a1c80 handle_segfault + 704
0x8bf1f9 Transaction::releaseSavepoint(int) + 231
0x975f58 Nfs::Statement::start(NNode*) + 262
0x92c686 PreparedStatement::executeUpdate() + 144
0x905537 Index::optimize(unsigned long long, Connection*) + 813
0x8b7782 Table::optimize(Connection*) + 176
0x8a91cd StorageTable::optimize() + 41
0x89c6a5 StorageInterface::optimize(THD*, st_ha_check_opt*) + 105
0x7ea3eb mysql_admin_table(THD*, TABLE_LIST*, st_ha_check_opt*, char const*, thr_lock_type, bool, bool, unsigned int, int (*)(THD*, TABLE_LIST*, st_ha_check_opt*), int (handler::*)(THD*, st_ha_check_opt*), int (*)(THD*, TABLE_LIST*)) + 2921
0x7eb50c mysql_optimize_table(THD*, TABLE_LIST*, st_ha_check_opt*) + 192
0x6b44c6 mysql_execute_command(THD*) + 9956
0x6ba85d mysql_parse(THD*, char const*, unsigned int, char const**) + 583
0x6bb597 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 2987
0x6bc93e do_command(THD*) + 552
0x6aa8c7 handle_one_connection + 313
0x2abd2c477317 _end + 724868143

Christoffer,  Can you reproduce this crash with a debug engine?

Hakan, Can you see if this problem still occurs?  The last comment indicated an out-of-memory problem, which may be fixed.

sqlbench does not crash Falcon anymore. I tried it with latest mysql-6.0-falcon-team tree.

We see two failures, but no crashes anymore. I will report the failures as new bug.

hakan@lu0011:~/work/mysql/mysql-6.0-falcon-team/sql-bench$ ./run-all-tests --server=mysql --fast --log --user=root --hires --socket=/tmp/mysql.sock
Warning: Can't execute alter-table.  Check the file 'output/alter-table-mysql_fast-Linux_2.6.22_2_amd64_x86_64'
Warning: Can't execute insert.  Check the file 'output/insert-mysql_fast-Linux_2.6.22_2_amd64_x86_64'
Test finished. You can find the result in:
output/RUN-mysql_fast-Linux_2.6.22_2_amd64_x86_64

A note has been added to the 6.0.4 changelog: 

Running the sqlbench test suite against Falcon would cause a crash.