Bug #72236 MySQL Crashes while enabling slave_parallel_workers....
Submitted: 4 Apr 2014 11:44 Modified: 11 Jan 2015 9:34
Reporter: Kalpan Naik Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Replication Severity:S1 (Critical)
Version:5.6.16 & 5.6.17 OS:Linux (CentOS 6.5 Final)
Assigned to: CPU Architecture:Any

[4 Apr 2014 11:44] Kalpan Naik
Description:
Our solution consists of multiple similar databases.

We wanted to use parallel replication being offered by Version 5.6.*.

But in certain scenario mysql crashes.

Scenario is : 

When one of the queries fail to execute on slave due to foreign_key check failure, MySQL server crashes with "mysql got signal 11" printed in the log file.

How to repeat:
Execute a query on master server's DB table which does not check for any foreign key.

But the same query when gets replicated & run onto slave should be checked for foreign key constraints.

Both versions 5.6.16 & 5.6.17, have this issue.

Previously we were using 5.5 version.
[4 Apr 2014 12:36] MySQL Verification Team
Please provide both my.cnf files (private if you wish). Thanks.
[4 Apr 2014 12:50] Kalpan Naik
Master server's config file

Attachment: master_my.cnf (application/octet-stream, text), 2.16 KiB.

[4 Apr 2014 12:51] Kalpan Naik
Slave server's config file

Attachment: slave_my.cnf (application/octet-stream, text), 2.35 KiB.

[4 Apr 2014 12:51] Kalpan Naik
I have added both the config files.

Thanks for consideration.
[4 Apr 2014 20:07] Sveta Smirnova
Thank you for the report.

I cannot repeat described behavior with generic data. Please provide query which crashes server and table definitions from both master and slave of all involved tables.
[7 Apr 2014 16:43] Kalpan Naik
SQL File to be executed on Master Server

Attachment: Master_Scenario.sql (application/octet-stream, text), 1.48 KiB.

[7 Apr 2014 16:43] Kalpan Naik
SQL File to be executed on Slave Server

Attachment: Slave_Scenario.sql (application/octet-stream, text), 2.14 KiB.

[7 Apr 2014 16:44] Kalpan Naik
--- Execute following query on Master Server and check for crash on Slave Server

INSERT INTO Table_Master (Column_1, Column_2, Column_3, Column_4, Column_5, Column_6, Column_7, Column_8, Column_9, Column_10, Column_11, Column_12, Column_13, Column_14, Column_15, Column_16, Column_17, Column_18, Column_19, Column_20, Column_21, Column_22, Column_23, Column_24, Column_25, Column_26, Column_27, Column_28, Column_29, Column_30, Column_31) VALUES

('2014-04-04 17:58:09', '2014-04-04 18:00:00', '192.168.2.152', 7000, 'c44a7c79-8e76-4753-8349-65988a1ba616', '121917987680805', '917987680805', '232917987680805', '192.168.2.101', '986422-19071@192.168.2.101', 'ABC', 'ABC', 'ABC', '917987680599', NULL, 1, 1, 9, 2, 1, 1, 7412, 1, 436, 111, 0, 0, 0, 0, 0, 0);
[7 Apr 2014 16:50] Kalpan Naik
Above scenario should be executed with slave_parallel_workers > 0 value on Slave Server.
[7 Apr 2014 19:46] Sveta Smirnova
Thank you for the feedback.

I still cannot repeat described behavior. Please provide configuration files from both master and slave.
[8 Apr 2014 3:23] Kalpan Naik
I have already submitted the conf files.
Can you give us conf files you are trying with?
[8 Apr 2014 3:25] Kalpan Naik
As per our analysis, when we try with table fields more than 30, it certainly crashes.
[8 Apr 2014 9:01] Kalpan Naik
The crash details from the log file....

Attachment: CrashDump.txt (text/plain), 2.67 KiB.

[8 Apr 2014 16:44] Sveta Smirnova
Thank you for the feedback.

I am sorry: I missed option files you provided earlier. But I still can not repeat described behavior even if use same options as you. Please also indicate exact MySQL package you use: this can be package-related issue.
[8 Apr 2014 17:31] Kalpan Naik
Downloaded from URL http://dev.mysql.com/downloads/mysql/

In select platform we selected "Source Code" option and then we selected

Generic Linux (Architecture Independent), Compressed TAR Archive

Our Server is CentOS 6.5 64 bit edition.
[8 Apr 2014 17:41] Sveta Smirnova
I tested with mysql-5.6.16-linux-glibc2.5-x86_64.tar.gz on Oracle Linux 6
[10 Apr 2014 3:59] Kalpan Naik
Any Update?
[15 Apr 2014 11:54] Sveta Smirnova
Thank you for the feedback.

I am sorry: I misinterpreted your message and missed you use source code package (I tested with Generic Linux binaries, this is differennt package).

Please provide cmake options you used to build MySQL.
[15 Apr 2014 14:02] Kalpan Naik
Thank you for the reply.

Our TestBed is as follows:

Operating System : CentOS 6.5 Final 64 bit Edition
MySQL Server     : mysql-5.6.17.tar.gz 
                  (Downloaded from http://dev.mysql.com/downloads/mysql/
                   Platform: Source Code
                   Generic Linux (Architecture Independent), 
                   Compressed TAR Archive)

cmake command we ran from source directory as follows:

cmake -DCMAKE_INSTALL_PREFIX=/opt/mysql-5.6.17

We cannot enable slave_parallel_workers option in our Production Environment due to this issue.

Thanks in advance.
[15 Apr 2014 17:29] Sveta Smirnova
Thank you for the feedback.

I still can not repeat described behavior. I analyzed crash information and found that this is most likely memory failure. You have quite huge memory settings. Please try to decrease innodb_buffer_pool_size on slave and inform us if problem still exists.
[15 Apr 2014 18:41] Kalpan Naik
Did you try with CentOS 6.5?
Bcoz when we try with CentOS 5.8 there is no issue.

Also in our lab environment even with less memory assigned to innodb buffer pool it crashes with CentOS 6.5
[6 May 2014 11:39] Kalpan Naik
Can you update?
[8 Jul 2014 19:50] Sveta Smirnova
Unfortunately not: I don't have Cent OS 6.5 machine available. I tried on Oracle Linux 6. I will try to find Cent OS 6.5 machine or create virtual one and try with it.
[11 Dec 2014 9:34] MySQL Verification Team
Imho this seems to be duplicate/related to internal Bug 18563480 , which is fixed in 5.6.20, and in 5.7.5.. Could you please try with latest MySQL GA version 5.6.22 and let us know if you are still hitting the issue?

// With 5.6.17 I see:

09:22:22 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=4
max_threads=151
thread_count=10
connection_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61077 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f1f88000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f1fbd4d7ea8 thread_stack 0x40000
/export/umesh/mysql-5.6.17/bin/mysqld(my_print_stacktrace+0x35)[0x904cb5]
/export/umesh/mysql-5.6.17/bin/mysqld(handle_fatal_signal+0x3e8)[0x6718c8]
/lib64/libpthread.so.0(+0xf130)[0x7f1fd10eb130]
/lib64/libc.so.6(+0x86901)[0x7f1fcfd3f901]
/export/umesh/mysql-5.6.17/bin/mysqld[0xb6e8c9]
/export/umesh/mysql-5.6.17/bin/mysqld(my_vsnprintf_ex+0x2f8)[0xb6ee88]
/export/umesh/mysql-5.6.17/bin/mysqld(_Z17vprint_msg_to_log8loglevelPKcP13__va_list_tag+0x43)[0x661b23]
/export/umesh/mysql-5.6.17/bin/mysqld(_ZN25Log_to_file_event_handler9log_errorE8loglevelPKcP13__va_list_tag+0x11)[0x661c91]
/export/umesh/mysql-5.6.17/bin/mysqld(_ZN6LOGGER15error_log_printE8loglevelPKcP13__va_list_tag+0x32)[0x65c262]
/export/umesh/mysql-5.6.17/bin/mysqld(_Z15error_log_print8loglevelPKcP13__va_list_tag+0x18)[0x65c448]
/export/umesh/mysql-5.6.17/bin/mysqld(_Z15sql_print_errorPKcz+0xa7)[0x65c7f7]
/export/umesh/mysql-5.6.17/bin/mysqld(_ZNK26Slave_reporting_capability9va_reportE8logleveliPKcS2_P13__va_list_tag+0x1f0)[0x8dec70]
/export/umesh/mysql-5.6.17/bin/mysqld(_ZNK12Slave_worker9do_reportE8logleveliPKcP13__va_list_tag+0xca)[0x8e749a]
/export/umesh/mysql-5.6.17/bin/mysqld(_ZNK26Slave_reporting_capability6reportE8logleveliPKcz+0x90)[0x8de950]
/export/umesh/mysql-5.6.17/bin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xda2)[0x89fc62]
/export/umesh/mysql-5.6.17/bin/mysqld(_Z21slave_worker_exec_jobP12Slave_workerP14Relay_log_info+0x203)[0x8ea0f3]
/export/umesh/mysql-5.6.17/bin/mysqld(handle_slave_worker+0x1e7)[0x8d7ec7]
/export/umesh/mysql-5.6.17/bin/mysqld(pfs_spawn_thread+0x126)[0x97e406]
/lib64/libpthread.so.0(+0x7df3)[0x7f1fd10e3df3]
/lib64/libc.so.6(clone+0x6d)[0x7f1fcfdaf47d]
[12 Jan 2015 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".