Bug #72285 Failing assertion: ret || !assert_on_error
Submitted: 9 Apr 2014 8:45 Modified: 3 Apr 2018 22:40
Reporter: Ahmed Abbas Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Connection Handling Severity:S2 (Serious)
Version:5.5.31 OS:Linux (Ubuntu 12.04)
Assigned to: CPU Architecture:Any
Tags: assertion failure

[9 Apr 2014 8:45] Ahmed Abbas
Description:
Not sure what causes this bug. 

The error raised is 'Lost connection to MySQL server during query' intermittently. Then few 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)' follows before mysql auto restarts.
The mysql error log is below.

140404  5:00:25  InnoDB: Assertion failure in thread 270793536 in file ut0mem.c line 103
InnoDB: Failing assertion: ret || !assert_on_error
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
05:00:25 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=29
max_threads=151
thread_count=23
connection_count=23
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362448 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb72ab003]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb7157f24]
[0xb6e0e400]
/usr/sbin/mysqld(+0x5f8442)[0xb7429442]
/usr/sbin/mysqld(+0x5b65a5)[0xb73e75a5]
/usr/sbin/mysqld(+0x53a500)[0xb736b500]
/lib/i386-linux-gnu/tls/i686/nosegneg/libpthread.so.0(+0x6d4c)[0xb6d8cd4c]
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(clone+0x5e)[0xb6b9bdde]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140404  5:00:29 [Note] Plugin 'FEDERATED' is disabled.
140404  5:00:29 InnoDB: The InnoDB memory heap is disabled
140404  5:00:29 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140404  5:00:29 InnoDB: Compressed tables use zlib 1.2.3.4
140404  5:00:30 InnoDB: Initializing buffer pool, size = 2.4G
140404  5:00:32 InnoDB: Completed initialization of buffer pool
140404  5:00:32 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 37669442846
140404  5:00:32  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 37669608057
140404  5:00:33  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
140404  5:00:34  InnoDB: Waiting for the background threads to start
140404  5:00:35 InnoDB: 5.5.31 started; log sequence number 37669608057
140404  5:00:35 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
140404  5:00:35 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
140404  5:00:35 [Note] Server socket created on IP: '0.0.0.0'.
140404  5:00:36 [Note] Event Scheduler: Loaded 0 events
140404  5:00:36 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.31-0ubuntu0.12.04.2-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)

How to repeat:
Most of the times an error was raised while running this.

select DATE(FROM_UNIXTIME(`timestamp`)) as date, AVG(IF(terminal_type = 'false', url, NULL)) as avg_without_tunnel, AVG(IF(terminal_type = 'true', url, NULL)) as avg_with_tunnel from statistics where stats_type = 'start-request-time' AND timestamp >= 1396396800 AND timestamp < 1397001600 GROUP BY date ORDER BY date desc

Now sure however if the error is related to it.

Fyi I also do bulk inserts of 50 rows. (not large data).. with an interval of 5 seconds.
[9 Apr 2014 10:41] MySQL Verification Team
Looks like server ran out of memory.  Had you checked that my.cnf options aren't too large and that enough free memory available when you run that query?
[10 May 2014 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[26 Oct 2015 7:40] Shahriyar Rzayev
The most easy way to reproduce this is running innobackupex with --parallel=1000000000000

Description:

[root@node1 mysql]# mysqld --version
mysqld  Ver 5.6.26-74.0 for Linux on x86_64 (Percona Server (GPL), Release 74.0, Revision 32f8dfd)

[root@node1 mysql]# xtrabackup --version
xtrabackup version 2.3.2 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 306a2e0)

[root@node1 mysql]# innobackupex --defaults-file=/etc/my.cnf --user=root --password=12345 --port=3306 --socket=/var/lib/mysql/mysql.sock --parallel=1000000000000 /home/MySQL-AutoXtraBackup/backup_dir/full/
Warning: option 'parallel': signed value 1000000000000 adjusted to 2147483647
151026 11:13:36 innobackupex: Starting the backup operation
xtrabackup: Generating a list of tablespaces
xtrabackup: Starting 2147483647 threads for parallel data files transfer
2015-10-26 11:13:37 7f938241e740  InnoDB: Assertion failure in thread 140271522277184 in file ut0mem.cc line 105
InnoDB: Failing assertion: ret || !assert_on_error
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
07:13:37 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x927d2e]
innobackupex(handle_fatal_signal+0x262) [0x7214b2]
/lib64/libpthread.so.0(+0xf130) [0x7f9381ffe130]
/lib64/libc.so.6(gsignal+0x37) [0x7f938076b5d7]
/lib64/libc.so.6(abort+0x148) [0x7f938076ccc8]
innobackupex(ut_malloc_low(unsigned long, unsigned long)+0x13c) [0x6d285c]
innobackupex(xtrabackup_backup_func()+0xa8d) [0x599ccd]
innobackupex(main+0xa95) [0x57de65]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9380757af5]
innobackupex() [0x591bc9]
[3 Mar 2018 22:40] MySQL Verification Team
Please check the comments done by Shane, try latest released version too. Thanks.
[4 Apr 2018 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".