Bug #72236 | MySQL Crashes while enabling slave_parallel_workers.... | ||
---|---|---|---|
Submitted: | 4 Apr 2014 11:44 | Modified: | 11 Jan 2015 9:34 |
Reporter: | Kalpan Naik | Email Updates: | |
Status: | No Feedback | Impact on me: | |
Category: | MySQL Server: Replication | Severity: | S1 (Critical) |
Version: | 5.6.16 & 5.6.17 | OS: | Linux (CentOS 6.5 Final) |
Assigned to: | CPU Architecture: | Any |
[4 Apr 2014 11:44]
Kalpan Naik
[4 Apr 2014 12:36]
MySQL Verification Team
Please provide both my.cnf files (private if you wish). Thanks.
[4 Apr 2014 12:50]
Kalpan Naik
Master server's config file
Attachment: master_my.cnf (application/octet-stream, text), 2.16 KiB.
[4 Apr 2014 12:51]
Kalpan Naik
Slave server's config file
Attachment: slave_my.cnf (application/octet-stream, text), 2.35 KiB.
[4 Apr 2014 12:51]
Kalpan Naik
I have added both the config files. Thanks for consideration.
[4 Apr 2014 20:07]
Sveta Smirnova
Thank you for the report. I cannot repeat described behavior with generic data. Please provide query which crashes server and table definitions from both master and slave of all involved tables.
[7 Apr 2014 16:43]
Kalpan Naik
SQL File to be executed on Master Server
Attachment: Master_Scenario.sql (application/octet-stream, text), 1.48 KiB.
[7 Apr 2014 16:43]
Kalpan Naik
SQL File to be executed on Slave Server
Attachment: Slave_Scenario.sql (application/octet-stream, text), 2.14 KiB.
[7 Apr 2014 16:44]
Kalpan Naik
--- Execute following query on Master Server and check for crash on Slave Server INSERT INTO Table_Master (Column_1, Column_2, Column_3, Column_4, Column_5, Column_6, Column_7, Column_8, Column_9, Column_10, Column_11, Column_12, Column_13, Column_14, Column_15, Column_16, Column_17, Column_18, Column_19, Column_20, Column_21, Column_22, Column_23, Column_24, Column_25, Column_26, Column_27, Column_28, Column_29, Column_30, Column_31) VALUES ('2014-04-04 17:58:09', '2014-04-04 18:00:00', '192.168.2.152', 7000, 'c44a7c79-8e76-4753-8349-65988a1ba616', '121917987680805', '917987680805', '232917987680805', '192.168.2.101', '986422-19071@192.168.2.101', 'ABC', 'ABC', 'ABC', '917987680599', NULL, 1, 1, 9, 2, 1, 1, 7412, 1, 436, 111, 0, 0, 0, 0, 0, 0);
[7 Apr 2014 16:50]
Kalpan Naik
Above scenario should be executed with slave_parallel_workers > 0 value on Slave Server.
[7 Apr 2014 19:46]
Sveta Smirnova
Thank you for the feedback. I still cannot repeat described behavior. Please provide configuration files from both master and slave.
[8 Apr 2014 3:23]
Kalpan Naik
I have already submitted the conf files. Can you give us conf files you are trying with?
[8 Apr 2014 3:25]
Kalpan Naik
As per our analysis, when we try with table fields more than 30, it certainly crashes.
[8 Apr 2014 9:01]
Kalpan Naik
The crash details from the log file....
Attachment: CrashDump.txt (text/plain), 2.67 KiB.
[8 Apr 2014 16:44]
Sveta Smirnova
Thank you for the feedback. I am sorry: I missed option files you provided earlier. But I still can not repeat described behavior even if use same options as you. Please also indicate exact MySQL package you use: this can be package-related issue.
[8 Apr 2014 17:31]
Kalpan Naik
Downloaded from URL http://dev.mysql.com/downloads/mysql/ In select platform we selected "Source Code" option and then we selected Generic Linux (Architecture Independent), Compressed TAR Archive Our Server is CentOS 6.5 64 bit edition.
[8 Apr 2014 17:41]
Sveta Smirnova
I tested with mysql-5.6.16-linux-glibc2.5-x86_64.tar.gz on Oracle Linux 6
[10 Apr 2014 3:59]
Kalpan Naik
Any Update?
[15 Apr 2014 11:54]
Sveta Smirnova
Thank you for the feedback. I am sorry: I misinterpreted your message and missed you use source code package (I tested with Generic Linux binaries, this is differennt package). Please provide cmake options you used to build MySQL.
[15 Apr 2014 14:02]
Kalpan Naik
Thank you for the reply. Our TestBed is as follows: Operating System : CentOS 6.5 Final 64 bit Edition MySQL Server : mysql-5.6.17.tar.gz (Downloaded from http://dev.mysql.com/downloads/mysql/ Platform: Source Code Generic Linux (Architecture Independent), Compressed TAR Archive) cmake command we ran from source directory as follows: cmake -DCMAKE_INSTALL_PREFIX=/opt/mysql-5.6.17 We cannot enable slave_parallel_workers option in our Production Environment due to this issue. Thanks in advance.
[15 Apr 2014 17:29]
Sveta Smirnova
Thank you for the feedback. I still can not repeat described behavior. I analyzed crash information and found that this is most likely memory failure. You have quite huge memory settings. Please try to decrease innodb_buffer_pool_size on slave and inform us if problem still exists.
[15 Apr 2014 18:41]
Kalpan Naik
Did you try with CentOS 6.5? Bcoz when we try with CentOS 5.8 there is no issue. Also in our lab environment even with less memory assigned to innodb buffer pool it crashes with CentOS 6.5
[6 May 2014 11:39]
Kalpan Naik
Can you update?
[8 Jul 2014 19:50]
Sveta Smirnova
Unfortunately not: I don't have Cent OS 6.5 machine available. I tried on Oracle Linux 6. I will try to find Cent OS 6.5 machine or create virtual one and try with it.
[11 Dec 2014 9:34]
MySQL Verification Team
Imho this seems to be duplicate/related to internal Bug 18563480 , which is fixed in 5.6.20, and in 5.7.5.. Could you please try with latest MySQL GA version 5.6.22 and let us know if you are still hitting the issue? // With 5.6.17 I see: 09:22:22 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=4 max_threads=151 thread_count=10 connection_count=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61077 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1f88000990 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f1fbd4d7ea8 thread_stack 0x40000 /export/umesh/mysql-5.6.17/bin/mysqld(my_print_stacktrace+0x35)[0x904cb5] /export/umesh/mysql-5.6.17/bin/mysqld(handle_fatal_signal+0x3e8)[0x6718c8] /lib64/libpthread.so.0(+0xf130)[0x7f1fd10eb130] /lib64/libc.so.6(+0x86901)[0x7f1fcfd3f901] /export/umesh/mysql-5.6.17/bin/mysqld[0xb6e8c9] /export/umesh/mysql-5.6.17/bin/mysqld(my_vsnprintf_ex+0x2f8)[0xb6ee88] /export/umesh/mysql-5.6.17/bin/mysqld(_Z17vprint_msg_to_log8loglevelPKcP13__va_list_tag+0x43)[0x661b23] /export/umesh/mysql-5.6.17/bin/mysqld(_ZN25Log_to_file_event_handler9log_errorE8loglevelPKcP13__va_list_tag+0x11)[0x661c91] /export/umesh/mysql-5.6.17/bin/mysqld(_ZN6LOGGER15error_log_printE8loglevelPKcP13__va_list_tag+0x32)[0x65c262] /export/umesh/mysql-5.6.17/bin/mysqld(_Z15error_log_print8loglevelPKcP13__va_list_tag+0x18)[0x65c448] /export/umesh/mysql-5.6.17/bin/mysqld(_Z15sql_print_errorPKcz+0xa7)[0x65c7f7] /export/umesh/mysql-5.6.17/bin/mysqld(_ZNK26Slave_reporting_capability9va_reportE8logleveliPKcS2_P13__va_list_tag+0x1f0)[0x8dec70] /export/umesh/mysql-5.6.17/bin/mysqld(_ZNK12Slave_worker9do_reportE8logleveliPKcP13__va_list_tag+0xca)[0x8e749a] /export/umesh/mysql-5.6.17/bin/mysqld(_ZNK26Slave_reporting_capability6reportE8logleveliPKcz+0x90)[0x8de950] /export/umesh/mysql-5.6.17/bin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xda2)[0x89fc62] /export/umesh/mysql-5.6.17/bin/mysqld(_Z21slave_worker_exec_jobP12Slave_workerP14Relay_log_info+0x203)[0x8ea0f3] /export/umesh/mysql-5.6.17/bin/mysqld(handle_slave_worker+0x1e7)[0x8d7ec7] /export/umesh/mysql-5.6.17/bin/mysqld(pfs_spawn_thread+0x126)[0x97e406] /lib64/libpthread.so.0(+0x7df3)[0x7f1fd10e3df3] /lib64/libc.so.6(clone+0x6d)[0x7f1fcfdaf47d]
[12 Jan 2015 1:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".