MySQL Bugs: #57822: mysqld process crashes in check_access(THD*, unsigned long, char const*...) call

Bug #57822	mysqld process crashes in check_access(THD, unsigned long, char const...) call
Submitted:	28 Oct 2010 18:28	Modified:	16 Dec 2010 10:54
Reporter:	Gökcen Eraslan	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.1.52	OS:	Linux
Assigned to:		CPU Architecture:	Any

Description:
When I try to start akonadiserver in KDE, mysqld process crashes and prints a stacktrace. Here is the mysql log file:

101028 20:49:53  InnoDB: Started; log sequence number 0 430083318
101028 20:49:53 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.51'  socket: '/home/gokcen/.local/share/akonadi/db_misc/mysql.socket'  port: 0  Pardus Linux
101028 20:49:53 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=2
max_threads=256
threads_connected=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 566891 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x90c7668
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xaf3cf36c thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2a) [0x84abc3a]
/usr/sbin/mysqld(handle_segfault+0x4fc) [0x81dbeec]
[0xffffe400]
/usr/sbin/mysqld(check_access(THD*, unsigned long, char const*, unsigned long*, bool, bool, bool)+0xa5) [0x81e6c75]
/usr/sbin/mysqld(check_table_access(THD*, unsigned long, TABLE_LIST*, unsigned int, bool)+0x295) [0x81e77a5]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x39fb) [0x81eeb0b]
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x30a) [0x81f490a]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x610) [0x81f4f20]
/usr/sbin/mysqld(do_command(THD*)+0x102) [0x81f62b2]
/usr/sbin/mysqld(handle_one_connection+0x281) [0x81e5261]
/lib/libpthread.so.0(+0x5e81) [0xb76f8e81]
/lib/libc.so.6(clone+0x5e) [0xb72d5aee]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x90d1060 = CHECK TABLE `collectionattributetable`  FOR UPGRADE
thd->thread_id=2
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
I start mysqld manually with command:

mysqld --defaults-file=/home/gokcen/.local/share/akonadi/mysql.conf --datadir=/home/gokcen/.local/share/akonadi/db_data/ --socket=/home/gokcen/.local/share/akonadi/db_misc/mysql.socket

after that I start akonadi server using command:

akonadictl start

mysql log file

Attachment: mysql.err (application/octet-stream, text), 2.30 KiB.

mysql conf file

Attachment: mysql.conf (application/octet-stream, text), 2.10 KiB.

Thank you for the report.

Server crashes on statement CHECK TABLE `collectionattributetable`  FOR UPGRADE. Does it crash if run this statement alone? Please also send us output of SHOW CREATE TABLE collectionattributetable

Here are the outputs of commands you've asked:

mysql> SHOW CREATE TABLE collectionattributetable;
+--------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table                    | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+--------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| collectionattributetable | CREATE TABLE `collectionattributetable` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `collectionId` bigint(20) NOT NULL,
  `type` longblob NOT NULL,
  `value` longblob,
  PRIMARY KEY (`id`),
  KEY `CollectionAttributeTable_collectionIndex` (`collectionId`),
  CONSTRAINT `collectionattributetable_ibfk_1` FOREIGN KEY (`collectionId`) REFERENCES `collectiontable` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=266 DEFAULT CHARSET=utf8 |
+--------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> CHECK TABLE `collectionattributetable`  FOR UPGRADE
    -> ;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql>

Thank you for the feedback.

I could not repeat described behavior with test data. Could you please send us shared InnoDB tablespace and ibd, frm files for table collectionattributetable and table it references to? Or at lease output of SHOW TABLE STATUS LIKE 'collectionattributetable'

Here is the output:

mysql> SHOW TABLE STATUS LIKE 'collectionattributetable';
+--------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+
| Name                     | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time | Check_time | Collation       | Checksum | Create_options | Comment |
+--------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+                                                                                                                                           
| collectionattributetable | InnoDB |      10 | Compact    |  296 |            221 |       65536 |               0 |        16384 |         0 |            266 | 2010-08-27 19:33:44 | NULL        | NULL       | utf8_general_ci |     NULL |                |         |                                                                                                                                           
+--------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+                                                                                                                                           
1 row in set (0.02 sec)

Here is the frm file for collectionattributetable

Attachment: collectionattributetable.frm (application/x-maker, text), 8.46 KiB.

Here is the ibd file for collectionattributetable

Attachment: collectionattributetable.ibd (application/octet-stream, text), 144.00 KiB.

Same with 5.1.52

GDB trace of the crash is as follows:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xafca1b70 (LWP 20604)]

0x081e6c65 in check_access (thd=0x8cc6b48, want_access=536870913, db=0x14 <Address 0x14 out of bounds>, save_priv=0x8cc43e4, dont_check_global_grants=false, no_errors=false, schema_db=true)
    at sql_parse.cc:5275
5275      if ((!db || !db[0]) && !thd->db && !dont_check_global_grants)

(gdb) thread apply all bt

Thread 13 (Thread 0xafca1b70 (LWP 20604)):
#0  0x081e6c65 in check_access (thd=0x8cc6b48, want_access=536870913, db=0x14 <Address 0x14 out of bounds>, save_priv=0x8cc43e4, dont_check_global_grants=false, no_errors=false, schema_db=true)
    at sql_parse.cc:5275
#1  0x081e7795 in check_table_access (thd=0x8cc6b48, want_access=536870913, tables=0x8cc42c0, number=4294967295, no_errors=false) at sql_parse.cc:5503
#2  0x081eeafb in mysql_execute_command (thd=0x8cc6b48) at sql_parse.cc:3065
#3  0x081f48fa in mysql_parse (thd=0x8cc6b48, rawbuf=0x8cc41b0 "CHECK TABLE `collectionattributetable`  FOR UPGRADE", length=51, found_semicolon=0xafca106c) at sql_parse.cc:6051
#4  0x081f4f10 in dispatch_command (command=<value optimized out>, thd=0x8cc6b48, packet=0x8cce639 "CHECK TABLE `collectionattributetable`  FOR UPGRADE", packet_length=51) at sql_parse.cc:1260
#5  0x081f62a2 in do_command (thd=0x8cc6b48) at sql_parse.cc:888
#6  0x081e5251 in handle_one_connection (arg=0x8cc6b48) at sql_connect.cc:1136
#7  0xb7f9ee51 in start_thread (arg=0xafca1b70) at pthread_create.c:301
#8  0xb7b78cde in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

When I try to print db and db[0] just before the crash, output is:

(gdb) p db
$1 = 0x14 <Address 0x14 out of bounds>
(gdb) p db[0]
Cannot access memory at address 0x14

Thank you for the feedback.

I still can not repeat the problem. Please also send us information about collectiontable table: SHOW CREATE TABLE, SHOW TABLE STATUS and ibd/frm files if possible.

mysql> SHOW CREATE TABLE collectiontable;
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table           | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| collectiontable | CREATE TABLE `collectiontable` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `remoteId` varbinary(255) DEFAULT NULL,
  `remoteRevision` varbinary(255) DEFAULT NULL,
  `name` varbinary(255) NOT NULL,
  `parentId` bigint(20) DEFAULT NULL,
  `resourceId` bigint(20) NOT NULL,
  `subscribed` tinyint(1) NOT NULL DEFAULT '1',
  `cachePolicyInherit` tinyint(1) NOT NULL DEFAULT '1',
  `cachePolicyCheckInterval` int(11) NOT NULL DEFAULT '-1',
  `cachePolicyCacheTimeout` int(11) NOT NULL DEFAULT '-1',
  `cachePolicySyncOnDemand` tinyint(1) NOT NULL DEFAULT '0',
  `cachePolicyLocalParts` varbinary(255) DEFAULT NULL,
  `queryString` varbinary(255) DEFAULT NULL,
  `queryLanguage` varbinary(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `CollectionTable_parentAndNameIndex` (`parentId`,`name`),
  KEY `resourceId` (`resourceId`),
  CONSTRAINT `collectiontable_ibfk_1` FOREIGN KEY (`parentId`) REFERENCES `collectiontable` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
  CONSTRAINT `collectiontable_ibfk_2` FOREIGN KEY (`resourceId`) REFERENCES `resourcetable` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=100 DEFAULT CHARSET=utf8 |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SHOW TABLE STATUS LIKE 'collectiontable';
+-----------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+
| Name            | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time | Check_time | Collation       | Checksum | Create_options | Comment |
+-----------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+
| collectiontable | InnoDB |      10 | Compact    |   99 |            165 |       16384 |               0 |        32768 |         0 |            100 | 2010-08-27 19:33:41 | NULL        | NULL       | utf8_general_ci |     NULL |                |         |
+-----------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+
1 row in set (0.21 sec)

mysql>

Here is the frm file for collectiontable

Attachment: collectiontable.frm (application/x-maker, text), 9.00 KiB.

Here is the ibd file for collectiontable

Attachment: collectiontable.ibd (application/octet-stream, text), 128.00 KiB.

I've found the issue, if I add -fPIC to CXXFLAGS in i686 architecture, I can reproduce the bug, and when I remove everything works OK.

Sorry for the noise, I think this is a GCC bug.

Thank you for the feedback.

Now I can repeat crash if run check table collectionattributetable;, but with different trace and check table collectiontable ; also fails. Does it crashes in your environment too?

Could you also please upload full core file and mysqld binary which created it to our FTP server: I want to check table-related data in it.

Backtrace in my environment:

stack_bottom = 0x450890f8 thread_stack 0x40000
./libexec/mysqld(my_print_stacktrace+0x35)[0xb2a211]
./libexec/mysqld(handle_segfault+0x288)[0x697e5a]
/lib64/libpthread.so.0[0x3429e0dd40]
./libexec/mysqld(mem_area_get_free+0xc)[0x986883]
./libexec/mysqld(mem_area_alloc+0x157)[0x9872e3]
./libexec/mysqld(mem_heap_create_block+0xa7)[0x986367]
./libexec/mysqld(mem_heap_add_block+0xdd)[0x9865e0]
./libexec/mysqld(mem_heap_alloc+0x6a)[0x985338]
./libexec/mysqld(trx_undo_rec_copy+0x57)[0x9bfc8a]
./libexec/mysqld(trx_undo_get_undo_rec_low+0xb4)[0x9c1a62]
./libexec/mysqld(trx_undo_get_undo_rec+0x5d)[0x9c1ae0]
./libexec/mysqld(trx_undo_prev_version_build+0x20f)[0x9c1d02]
./libexec/mysqld(row_vers_build_for_consistent_read+0x2a4)[0x9b48bf]
./libexec/mysqld[0x9ad2e6]
./libexec/mysqld[0x9ad67f]
./libexec/mysqld(row_search_for_mysql+0x1a48)[0x9af70d]
./libexec/mysqld[0x9a6e9c]
./libexec/mysqld(row_check_table_for_mysql+0x13a)[0x9a6fdf]
./libexec/mysqld(_ZN11ha_innobase5checkEP3THDP15st_ha_check_opt+0x145)[0x93a913]
./libexec/mysqld(_ZN7handler8ha_checkEP3THDP15st_ha_check_opt+0xe9)[0x7f3a7d]
./libexec/mysqld[0x81ca13]
./libexec/mysqld(_Z17mysql_check_tableP3THDP10TABLE_LISTP15st_ha_check_opt+0xec)[0x81f147]
./libexec/mysqld(_Z21mysql_execute_commandP3THD+0x2b75)[0x6ac85a]
./libexec/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x2ef)[0x6b59d2]
./libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xda6)[0x6a7e3b]
./libexec/mysqld(_Z10do_commandP3THD+0x27e)[0x6a6d9e]
./libexec/mysqld(handle_one_connection+0x14c)[0x6a4fdb]
/lib64/libpthread.so.0[0x3429e061b5]
/lib64/libc.so.6(clone+0x6d)[0x34292cd39d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x10ac1fc8 = check table collectiontable
thd->thread_id=1
thd->killed=NOT_KILLED

When I remove -fPIC flag from CXXFLAGS:

mysql> check table collectionattributetable;
+----------------------------------+-------+----------+----------+
| Table                            | Op    | Msg_type | Msg_text |
+----------------------------------+-------+----------+----------+
| akonadi.collectionattributetable | check | status   | OK       |
+----------------------------------+-------+----------+----------+
1 row in set (0.01 sec)

But if I don't remove it:

mysql> check table collectionattributetable;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Core file of crash is 176M and both mysqld and core file have distro specific bits. For example we unpack source tarball to /var/pisi/mysql-5.1.52-53/work/mysql-5.1.52/ directory and debug symbols of mysqld is in /usr/lib/debug/usr/sbin/mysqld.debug etc. 

But if you really want core file, I can put it in my web page and you can download from there.

somewhat of a wild guess: what happens if you use 
-fPIC both in CXXFLAGS and CFLAGS? Will the problem go away?

We already use -fPIC for CFLAGS in both i686 and x86_64 architectures, that is still the same. But using -fPIC as CXXFLAG in i686 causes this bug, so removed -fPIC from just CXXFLAGS.

Thank you for the feedback.

Still interesting why MySQL server crashes in my case and how were created such faulty tables. So having core file would be good. And just to confirm: do you compile and use MySQL on Pardus?

Yes, I compile mysql on Pardus and here is the core file: 

http://cekirdek.pardus.org.tr/~gokcen/files/core-mysqld.12549

Thank you for the feedback.

Please send us mysqld binary also.

Please send ibdata* file[s] also: I want to be sure I got crash not because created ibdata in wrong way.

Please be sure ibdata* is same age like *ibd files, probably best way is to resend them.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".