Bug #79986 Many row_sel_store_mysql_field Valgrind client request errors on 32 bit builds
Submitted: 14 Jan 2016 14:18 Modified: 22 Jan 2016 14:05
Reporter: Laurynas Biveinis Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:5.7.10 OS:Ubuntu (15.10 32-bit)
Assigned to: CPU Architecture:Any
Tags: 32 bit, valgrind

[14 Jan 2016 14:18] Laurynas Biveinis
Description:
A 32 bit build is likely to produce the following Valgrind error while running a big part of MTR tests:

==22696== Uninitialised byte(s) found during client check request
==22696==    at 0x970BFE9: row_sel_store_mysql_field_func(unsigned char*, row_prebuilt_t*, unsigned char const*, dict_index_t const*, unsigned long const*, unsigned long, mysql_row_templ_t const*) (row0sel.cc:3135)
==22696==    by 0x970C5B9: row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, unsigned char const*, dtuple_t const*, unsigned long, dict_index_t const*, unsigned long const*) (row0sel.cc:3280)
==22696==    by 0x970EB0B: row_search_no_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long) (row0sel.cc:4340)
==22696==    by 0x957182D: ha_innobase::index_read(unsigned char*, unsigned char const*, unsigned int, ha_rkey_function) (ha_innodb.cc:8304)
==22696==    by 0x957276D: ha_innobase::index_first(unsigned char*) (ha_innodb.cc:8709)
==22696==    by 0x9572A73: ha_innobase::rnd_next(unsigned char*) (ha_innodb.cc:8814)
==22696==    by 0x8AF5E13: handler::ha_rnd_next(unsigned char*) (handler.cc:2817)
==22696==    by 0x908C3DC: rr_sequential(READ_RECORD*) (records.cc:510)
==22696==    by 0x9113E30: join_init_read_record(QEP_TAB*) (sql_executor.cc:2464)
==22696==    by 0x9110FB2: sub_select(JOIN*, QEP_TAB*, bool) (sql_executor.cc:1271)
==22696==    by 0x91108E5: do_select(JOIN*) (sql_executor.cc:944)
==22696==    by 0x910E846: JOIN::exec() (sql_executor.cc:199)
==22696==    by 0x91AE2FE: handle_query(THD*, LEX*, Query_result*, unsigned long long, unsigned long long) (sql_select.cc:184)
==22696==    by 0x91608A1: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5098)
==22696==    by 0x9159A0B: mysql_execute_command(THD*, bool) (sql_parse.cc:2748)
==22696==    by 0x91616EB: mysql_parse(THD*, Parser_state*) (sql_parse.cc:5508)

The one instance I have with Valgrind --track-origins=yes, it has the following extra info:

==22696==  Address 0x14f19e3a is 2,314 bytes inside a block of size 2,328 alloc'd
==22696==    at 0x402C19C: malloc (vg_replace_malloc.c:299)
==22696==    by 0x9485376: my_raw_malloc (my_malloc.c:191)
==22696==    by 0x9484ED6: my_malloc (my_malloc.c:54)
==22696==    by 0x947CF50: alloc_root (my_alloc.c:217)
==22696==    by 0x92070DE: create_tmp_table(THD*, Temp_table_param*, List<Item>&, st_order*, bool, bool, unsigned long long, unsigned long long, char const*) (sql_tmp_table.cc:1278)
==22696==    by 0x91D65B7: create_schema_table(THD*, TABLE_LIST*) (sql_show.cc:7554)
==22696==    by 0x91D7383: mysql_schema_table(THD*, LEX*, TABLE_LIST*) (sql_show.cc:7783)
==22696==    by 0x90D3B4A: open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) (sql_base.cc:4914)
==22696==    by 0x90D4DF2: open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) (sql_base.cc:5633)
==22696==    by 0x90D5F00: open_tables_for_query(THD*, TABLE_LIST*, unsigned int) (sql_base.cc:6373)
==22696==    by 0x9160741: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5069)
==22696==    by 0x9159A0B: mysql_execute_command(THD*, bool) (sql_parse.cc:2748)
==22696==    by 0x91616EB: mysql_parse(THD*, Parser_state*) (sql_parse.cc:5508)
==22696==    by 0x915687A: dispatch_command(THD*, COM_DATA const*, enum_server_command) (sql_parse.cc:1425)
==22696==    by 0x91557DD: do_command(THD*) (sql_parse.cc:993)
==22696==    by 0x9288188: handle_connection (connection_handler_per_thread.cc:301)

So it could be related to InnoDB disk temp tables.

How to repeat:
For example,

$ ./mtr --debug-server --valgrind rpl_semi_sync --big-test
[22 Jan 2016 9:31] MySQL Verification Team
Hello Laurynas,

Thank you for the report.
Could you please share exact cmake command used for the build?

I tried this way:

--

ushastry@ubuntu32:~/Downloads/mysql-5.7.10 cmake . -DWITH_DEBUG=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=../boost
ushastry@ubuntu32:~/Downloads/mysql-5.7.10 make
ushastry@ubuntu32:~/Downloads/mysql-5.7.10$ cd mysql-test
 
 ushastry@ubuntu32:~/Downloads/mysql-5.7.10/mysql-test$ ./mtr --debug-server --valgrind  rpl_semi_sync --big-test
 Logging: ./mtr  --debug-server --valgrind rpl_semi_sync --big-test
 MySQL Version 5.7.10
 Turning on valgrind for all executables
 Running valgrind with options " --show-reachable=yes "
 Turning off --check-testcases to save time when valgrinding
 Checking supported features...
  - SSL connections supported
  - binaries are debug compiled
 Collecting tests...
  - adding combinations for rpl
 Checking leftover processes...
 Removing old var directory...
 Creating var directory '/home/ushastry/Downloads/mysql-5.7.10/mysql-test/var'...
 Installing system database...
 
 ==============================================================================
 
 TEST                                      RESULT   TIME (ms) or COMMENT
 --------------------------------------------------------------------------
 
 worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009
 worker[1] mysql-test-run: WARNING: Waited 60 seconds for /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/run/mysqld.1.pid to be created, still waiting for 1740 seconds...
 rpl.rpl_semi_sync 'mix'                  [ pass ]  349849
 worker[1] mysql-test-run: WARNING: Waited 60 seconds for /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/run/mysqld.1.pid to be created, still waiting for 1740 seconds...
 rpl.rpl_semi_sync 'row'                  [ pass ]  331910
 worker[1] mysql-test-run: WARNING: Waited 60 seconds for /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/run/mysqld.1.pid to be created, still waiting for 1740 seconds...
 worker[1] mysql-test-run: WARNING: Waited 120 seconds for /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/run/mysqld.1.pid to be created, still waiting for 1680 seconds...
 rpl.rpl_semi_sync 'stmt'                 [ pass ]  356707
 worker[1] Valgrind report from /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/log/mysqld.1.err after tests:
  rpl.rpl_semi_sync
 --------------------------------------------------------------------------
 HEAP SUMMARY:
     in use at exit: 18,944 bytes in 1 blocks
   total heap usage: 77,981 allocs, 77,980 frees, 365,598,044 bytes allocated
 
 18,944 bytes in 1 blocks are still reachable in loss record 1 of 1
    at 0x402C19C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
    by 0x411874A: ??? (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.21)
    by 0x400EDCF: call_init.part.0 (dl-init.c:72)
    by 0x400EEDF: call_init (dl-init.c:30)
    by 0x400EEDF: _dl_init (dl-init.c:120)
    by 0x4000ACE: ??? (in /lib/i386-linux-gnu/ld-2.21.so)
 
 LEAK SUMMARY:
    definitely lost: 0 bytes in 0 blocks
    indirectly lost: 0 bytes in 0 blocks
      possibly lost: 0 bytes in 0 blocks
    still reachable: 18,944 bytes in 1 blocks
         suppressed: 0 bytes in 0 blocks
 
 For counts of detected and suppressed errors, rerun with: -v
 ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
 
 worker[1] Valgrind report from /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/log/mysqld.1.err after tests:
  rpl.rpl_semi_sync
 --------------------------------------------------------------------------
 HEAP SUMMARY:
     in use at exit: 18,944 bytes in 1 blocks
   total heap usage: 80,363 allocs, 80,362 frees, 366,699,548 bytes allocated
 
 18,944 bytes in 1 blocks are still reachable in loss record 1 of 1
    at 0x402C19C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
    by 0x411874A: ??? (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.21)
    by 0x400EDCF: call_init.part.0 (dl-init.c:72)
    by 0x400EEDF: call_init (dl-init.c:30)
    by 0x400EEDF: _dl_init (dl-init.c:120)
    by 0x4000ACE: ??? (in /lib/i386-linux-gnu/ld-2.21.so)
 
 LEAK SUMMARY:
    definitely lost: 0 bytes in 0 blocks
    indirectly lost: 0 bytes in 0 blocks
      possibly lost: 0 bytes in 0 blocks
    still reachable: 18,944 bytes in 1 blocks
         suppressed: 0 bytes in 0 blocks
 
 For counts of detected and suppressed errors, rerun with: -v
 ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
 
 worker[1] Valgrind report from /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/log/mysqld.1.err after tests:
  rpl.rpl_semi_sync
 --------------------------------------------------------------------------
 HEAP SUMMARY:
     in use at exit: 18,944 bytes in 1 blocks
   total heap usage: 79,107 allocs, 79,106 frees, 366,182,014 bytes allocated
 
 18,944 bytes in 1 blocks are still reachable in loss record 1 of 1
    at 0x402C19C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
    by 0x411874A: ??? (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.21)
    by 0x400EDCF: call_init.part.0 (dl-init.c:72)
    by 0x400EEDF: call_init (dl-init.c:30)
    by 0x400EEDF: _dl_init (dl-init.c:120)
    by 0x4000ACE: ??? (in /lib/i386-linux-gnu/ld-2.21.so)
 
 LEAK SUMMARY:
    definitely lost: 0 bytes in 0 blocks
    indirectly lost: 0 bytes in 0 blocks
      possibly lost: 0 bytes in 0 blocks
    still reachable: 18,944 bytes in 1 blocks
         suppressed: 0 bytes in 0 blocks
 
 For counts of detected and suppressed errors, rerun with: -v
 ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
 
 valgrind_report                          [ fail ]
         Test ended at 2016-01-22 13:52:46
 
 Valgrind reported failures at shutdown, see above
 
 --------------------------------------------------------------------------
 The servers were restarted 2 times
 Spent 1038.466 of 1788 seconds executing testcases
 
 Completed: Failed 1/4 tests, 75.00% were successful.
 
 Failing test(s): valgrind_report
 
 The log files in var/log may give you some hint of what went wrong.
 
 If you want to report this error, please read first the documentation
at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html

OS and other details:

ushastry@ubuntu32:~/Downloads/mysql-5.7.10/mysql-test$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=15.10
DISTRIB_CODENAME=wily
DISTRIB_DESCRIPTION="Ubuntu 15.10"
NAME="Ubuntu"
VERSION="15.10 (Wily Werewolf)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 15.10"
VERSION_ID="15.10"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
ushastry@ubuntu32:~/Downloads/mysql-5.7.10/mysql-test$ valgrind --version
valgrind-3.11.0
ushastry@ubuntu32:~/Downloads/mysql-5.7.10/mysql-test$ cmake --version
cmake version 3.2.2

Thanks,
Umesh
[22 Jan 2016 10:12] Laurynas Biveinis
Umesh -

Please add -DWITH_VALGRIND=ON to your CMake args. Note that the reported Valgrind errors are coming from Valgrind client requests, and those need that CMake option to be compiled in.
[22 Jan 2016 11:00] MySQL Verification Team
Thank you for the feedback.

Regards,
Umesh
[22 Jan 2016 14:05] MySQL Verification Team
-- with 5.7.10 on Ubuntu 32bit

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009
worker[1] mysql-test-run: WARNING: Waited 60 seconds for /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/run/mysqld.1.pid to be created, still waiting for 1740 seconds...
rpl.rpl_semi_sync 'mix'                  [ fail ]  Found warnings/errors in server log file!
        Test ended at 2016-01-22 19:32:02
line
==10846== Thread 28:
==10846== Uninitialised byte(s) found during client check request
==10846==    at 0x96E54E3: row_sel_store_mysql_field_func(unsigned char*, row_prebuilt_t*, unsigned char const*, dict_index_t const*, unsigned long const*, unsigned long, mysql_row_templ_t const*) (row0sel.cc:3135)
==10846==    by 0x96E5AB3: row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, unsigned char const*, dtuple_t const*, unsigned long, dict_index_t const*, unsigned long const*) (row0sel.cc:3280)
==10846==    by 0x96E8005: row_search_no_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long) (row0sel.cc:4340)
==10846==    by 0x954D691: ha_innobase::index_read(unsigned char*, unsigned char const*, unsigned int, ha_rkey_function) (ha_innodb.cc:8304)
==10846==    by 0x954E5D1: ha_innobase::index_first(unsigned char*) (ha_innodb.cc:8709)
==10846==    by 0x954E8D7: ha_innobase::rnd_next(unsigned char*) (ha_innodb.cc:8814)
==10846==    by 0x8AF4C43: handler::ha_rnd_next(unsigned char*) (handler.cc:2817)
==10846==    by 0x908B20C: rr_sequential(READ_RECORD*) (records.cc:510)
==10846==    by 0x9112C60: join_init_read_record(QEP_TAB*) (sql_executor.cc:2464)
==10846==    by 0x910FDE2: sub_select(JOIN*, QEP_TAB*, bool) (sql_executor.cc:1271)
==10846==    by 0x910F715: do_select(JOIN*) (sql_executor.cc:944)
==10846==    by 0x910D676: JOIN::exec() (sql_executor.cc:199)
==10846==    by 0x91AD12E: handle_query(THD*, LEX*, Query_result*, unsigned long long, unsigned long long) (sql_select.cc:184)
==10846==    by 0x915F6D1: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5098)
==10846==    by 0x915883B: mysql_execute_command(THD*, bool) (sql_parse.cc:2748)
==10846==    by 0x916051B: mysql_parse(THD*, Parser_state*) (sql_parse.cc:5508)
==10846==  Address 0x1440d762 is 2,314 bytes inside a block of size 2,328 alloc'd
==10846==    at 0x402C19C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==10846==    by 0x94841B4: my_raw_malloc (my_malloc.c:191)
==10846==    by 0x9483D14: my_malloc (my_malloc.c:54)
==10846==    by 0x947BD8E: alloc_root (my_alloc.c:217)
==10846==    by 0x9205F0E: create_tmp_table(THD*, Temp_table_param*, List<Item>&, st_order*, bool, bool, unsigned long long, unsigned long long, char const*) (sql_tmp_table.cc:1278)
==10846==    by 0x91D53E7: create_schema_table(THD*, TABLE_LIST*) (sql_show.cc:7554)
==10846==    by 0x91D61B3: mysql_schema_table(THD*, LEX*, TABLE_LIST*) (sql_show.cc:7783)
==10846==    by 0x90D297A: open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) (sql_base.cc:4914)
==10846==    by 0x90D3C22: open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) (sql_base.cc:5633)
==10846==    by 0x90D4D30: open_tables_for_query(THD*, TABLE_LIST*, unsigned int) (sql_base.cc:6373)
==10846==    by 0x915F571: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5069)
==10846==    by 0x915883B: mysql_execute_command(THD*, bool) (sql_parse.cc:2748)
==10846==    by 0x916051B: mysql_parse(THD*, Parser_state*) (sql_parse.cc:5508)
==10846==    by 0x91556AA: dispatch_command(THD*, COM_DATA const*, enum_server_command) (sql_parse.cc:1425)
==10846==    by 0x915460D: do_command(THD*) (sql_parse.cc:993)
==10846==    by 0x9286FB8: handle_connection (connection_handler_per_thread.cc:301)
==10846== Uninitialised byte(s) found during client check request
==10846==    at 0x96E54E3: row_sel_store_mysql_field_func(unsigned char*, row_prebuilt_t*, unsigned char const*, dict_index_t const*, unsigned long const*, unsigned long, mysql_row_templ_t const*) (row0sel.cc:3135)
==10846==    by 0x96E5AB3: row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, unsigned char const*, dtuple_t const*, unsigned long, dict_index_t const*, unsigned long const*) (row0sel.cc:3280)
==10846==    by 0x96E8005: row_search_no_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long) (row0sel.cc:4340)
==10846==    by 0x954E2F1: ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int) (ha_innodb.cc:8597)
==10846==    by 0x954E905: ha_innobase::rnd_next(unsigned char*) (ha_innodb.cc:8822)
==10846==    by 0x8AF4C43: handler::ha_rnd_next(unsigned char*) (handler.cc:2817)
==10846==    by 0x908B20C: rr_sequential(READ_RECORD*) (records.cc:510)
==10846==    by 0x910FDF8: sub_select(JOIN*, QEP_TAB*, bool) (sql_executor.cc:1274)
==10846==    by 0x910F715: do_select(JOIN*) (sql_executor.cc:944)
==10846==    by 0x910D676: JOIN::exec() (sql_executor.cc:199)
==10846==    by 0x91AD12E: handle_query(THD*, LEX*, Query_result*, unsigned long long, unsigned long long) (sql_select.cc:184)
==10846==    by 0x915F6D1: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5098)
==10846==    by 0x915883B: mysql_execute_command(THD*, bool) (sql_parse.cc:2748)
==10846==    by 0x916051B: mysql_parse(THD*, Parser_state*) (sql_parse.cc:5508)
==10846==    by 0x91556AA: dispatch_command(THD*, COM_DATA const*, enum_server_command) (sql_parse.cc:1425)
==10846==    by 0x915460D: do_command(THD*) (sql_parse.cc:993)
==10846==  Address 0x1440d762 is 2,314 bytes inside a block of size 2,328 alloc'd
==10846==    at 0x402C19C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==10846==    by 0x94841B4: my_raw_malloc (my_malloc.c:191)
==10846==    by 0x9483D14: my_malloc (my_malloc.c:54)
==10846==    by 0x947BD8E: alloc_root (my_alloc.c:217)
==10846==    by 0x9205F0E: create_tmp_table(THD*, Temp_table_param*, List<Item>&, st_order*, bool, bool, unsigned long long, unsigned long long, char const*) (sql_tmp_table.cc:1278)
==10846==    by 0x91D53E7: create_schema_table(THD*, TABLE_LIST*) (sql_show.cc:7554)
==10846==    by 0x91D61B3: mysql_schema_table(THD*, LEX*, TABLE_LIST*) (sql_show.cc:7783)
==10846==    by 0x90D297A: open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) (sql_base.cc:4914)
==10846==    by 0x90D3C22: open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) (sql_base.cc:5633)
==10846==    by 0x90D4D30: open_tables_for_query(THD*, TABLE_LIST*, unsigned int) (sql_base.cc:6373)
==10846==    by 0x915F571: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5069)
==10846==    by 0x915883B: mysql_execute_command(THD*, bool) (sql_parse.cc:2748)
==10846==    by 0x916051B: mysql_parse(THD*, Parser_state*) (sql_parse.cc:5508)
==10846==    by 0x91556AA: dispatch_command(THD*, COM_DATA const*, enum_server_command) (sql_parse.cc:1425)
==10846==    by 0x915460D: do_command(THD*) (sql_parse.cc:993)
==10846==    by 0x9286FB8: handle_connection (connection_handler_per_thread.cc:301)
^ Found warnings in /home/ushastry/Downloads/mysql-5.7.10/mysql-test/var/log/mysqld.1.err
ok