Bug #89418 Possible null pointer crash with btr_cur_ins_lock_and_undo
Submitted: 25 Jan 2018 15:56 Modified: 2 Feb 2018 8:29
Reporter: Zsolt Parragi (OCA) Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.6 OS:Ubuntu (artful)
Assigned to: CPU Architecture:Any

[25 Jan 2018 15:56] Zsolt Parragi
Description:
Loading the world database causes a crash with some compilers - I encountered it with gcc 7.

The crash is caused because the 4th parameter of trx_undo_report_row_operation is marked as nonnull in storage/innobase/include/trx0rec.h. The nonnull attribute was removed in 5.7.

This attribute results in gcc optimizing out a few if(variable!=NULL) checks in release builds.

How to repeat:
Build MySQL 5.6 on artful / with gcc 7, in release mode
Try to load the world database
[29 Jan 2018 11:36] MySQL Verification Team
Hello Zsolt,

Thank you for the report and feedback.
I built 5.6.39 release built on Ubuntu 17.10(Artful Aardvark) and imported world database without any issues. Could you please provide exact cmake options used in your environment and other steps to reproduce this issue at our end?

--
-- Artful Aardvark
gcc 7.2, cmake 3.9.1

ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ cmake  .  -DENABLE_DOWNLOADS=1  -DCMAKE_INSTALL_PREFIX=/home/ushastry/Downloads/mysql-5639
ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ make
ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ make install

rm -rf 89441
scripts/mysql_install_db --basedir=$PWD --datadir=$PWD/89441
bin/mysqld --no-defaults --basedir=$PWD --datadir=$PWD/89441 --core-file --socket=/tmp/mysql_ushastry.sock  --port=3306 --log-error=$PWD/89441/log.err 2>&1 &

Thanks,
Umesh
[29 Jan 2018 11:40] MySQL Verification Team
test results

Attachment: 89418_5.6.39.build (application/octet-stream, text), 268.55 KiB.

[29 Jan 2018 11:40] Zsolt Parragi
Hello

As this is caused by the optimizer, it only happens in release/relwithdebinfo mode - add -DCMAKE_BUILD_TYPE=Release to the cmake command
[29 Jan 2018 11:43] MySQL Verification Team
Thank you, let me rebuild again and get back to you.

Thanks,
Umesh
[29 Jan 2018 13:10] MySQL Verification Team
-- No issues, even with the exact build.
Could you please confirm in which exact 5.6/5.7 versions you are seeing the issue? Thanks..

ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ cmake  .  -DCMAKE_BUILD_TYPE=Release  -DCMAKE_INSTALL_PREFIX=/home/ushastry/Downloads/mysql-5639
ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ make
ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ make install

ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ gcc --version
gcc (Ubuntu 7.2.0-8ubuntu3) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ cmake --version
cmake version 3.9.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.10
DISTRIB_CODENAME=artful
DISTRIB_DESCRIPTION="Ubuntu 17.10"
NAME="Ubuntu"
VERSION="17.10 (Artful Aardvark)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.10"
VERSION_ID="17.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=artful
UBUNTU_CODENAME=artful
ushastry@Ubunut1710:~/Downloads/mysql-5.6.39$ 
-- cd to ~/Downloads/mysql-5639

-- Start server

rm -rf 89441
scripts/mysql_install_db --basedir=$PWD --datadir=$PWD/89441
bin/mysqld --no-defaults --basedir=$PWD --datadir=$PWD/89441 --core-file --socket=/tmp/mysql_ushastry.sock  --port=3306 --log-error=$PWD/89441/log.err 2>&1 &

-- import/extract http://downloads.mysql.com/docs/world.sql.zip

ushastry@Ubunut1710:~/Downloads/mysql-5639$ bin/mysql -uroot -S /tmp/mysql_ushastry.sock < world.sql 
ushastry@Ubunut1710:~/Downloads/mysql-5639$ bin/mysql -uroot -S /tmp/mysql_ushastry.sock
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.39 Source distribution

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
| world              |
+--------------------+
5 rows in set (0.00 sec)

mysql> source world.sql;
Query OK, 0 rows affected (0.00 sec)
.
Query OK, 0 rows affected (0.00 sec)

mysql> 

ushastry@Ubunut1710:~/Downloads/mysql-5639$ cat docs/INFO_SRC
commit: a91263a00f8d59c8b5b4775b3b035f03c76f7c34
date: 2017-12-06 13:19:57 +0530
build-date: 2017-12-09 08:33:45 +0100
short: a91263a
branch: mysql-5.6.39-release

MySQL source 5.6.39
[30 Jan 2018 8:56] Zsolt Parragi
Hello

Sorry, looks like even I can't reproduce the crash. I originally found it in percona server, and assumed that if the relevant code pieces are the same, it should be present in mysql too, but looks like I was wrong.

While I can't reproduce the crash itself, the possibility is there in the code - and since the attribute I mentioned was removed in 5.7, I assume it would be possible to reproduce this somehow.

The problem, based on the code, which is present in 5.6:

* the thr parameter for trx_undo_report_row_operation is marked as non null
* the thr parameter for btr_cur_ins_lock_and_undo isn't non null, a comment even mentions that in can be null. But this function calls trx_undo_report_row_operation without additional null checks.

For some reason, this doesn't result in a compilation warning/error.
The compilation reaches the optimization phase, where based on the call chain, it deduces that some if(thr) non null checks can be optimized away, because based on the attribute, thr can't be null. This could lead to crashes.
[2 Feb 2018 8:29] MySQL Verification Team
Thank you for confirming that issue is not reproducible at your end as well.
If you can provide exact repeatable test case, feel free to add it to this bug and change the status back to 'Open'.

Thanks,
Umesh