| Bug #79586 | MCM1.4 - autotune --dryrun Internal error: Unable to open dumpfile | ||
|---|---|---|---|
| Submitted: | 10 Dec 2015 7:34 | Modified: | 27 Feb 2016 2:35 |
| Reporter: | Ivan Ma | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Cluster Manager | Severity: | S3 (Non-critical) |
| Version: | 1.4 | OS: | Oracle Linux (7.0) |
| Assigned to: | MySQL Verification Team | CPU Architecture: | Any |
[10 Dec 2015 7:34]
Ivan Ma
[10 Dec 2015 7:35]
Ivan Ma
Information about the execution
Attachment: MCM 1-4Autotune-bug.ppt (application/vnd.ms-powerpoint, text), 444.50 KiB.
[10 Dec 2015 7:37]
Ivan Ma
strace execution on mcmd (strace -f -v ....) when the autotune was run
Attachment: my3.zip (application/zip, text), 126.11 KiB.
[23 Dec 2015 16:59]
MySQL Verification Team
Hi, I just tested exactly the same setup you did and it worked flawlessly?! Single cluster, 2 clusters, 3 clusters, 1 host, 2 hosts, 4 hosts .. I was not able to reproduce the problem in any configuration. Is this the only time you reproduced this bug or you set the dual cluster on 2 hosts more then once with same problem? all best Bogdan Kecman
[24 Dec 2015 1:50]
Ivan Ma
The following are the exactly the command execution. where mcmd started on primary and secondary (/etc/hosts has entries about primary and secondary).
***********************
Primary
***********************
[mysql@virtual-31 rep]$ mcmd &
[1] 1989
[mysql@virtual-31 rep]$ MySQL Cluster Manager 1.4.0 (64bit) started
Connect to MySQL Cluster Manager by running "/home/mysql/mcm/mcm1.4.0/bin/mcm" - a virtual-31.localhost:1862
***********************
Secondary
***********************
[mysql@virtual-32 rep]$ mcmd &
[1] 1970
[mysql@virtual-32 rep]$ MySQL Cluster Manager 1.4.0 (64bit) started
Connect to MySQL Cluster Manager by running "/home/mysql/mcm/mcm1.4.0/bin/mcm" -a virtual-32.localhost:1862
***********************
On Primary running mcm
***********************
mcm> list sites;
Empty set (0.01 sec)
mcm> create site --hosts=primary,secondary mysite;
+---------------------------+
| Command result |
+---------------------------+
| Site created successfully |
+---------------------------+
1 row in set (1.63 sec)
mcm> add package -b /home/mysql/mcm/cluster-latest cluster.pkg
-> ;
+----------------------------+
| Command result |
+----------------------------+
| Package added successfully |
+----------------------------+
1 row in set (0.57 sec)
mcm> create cluster -P cluster.pkg --processhosts=ndbd@primary,ndbd@primary,ndb_ mgmd@primary,ndbapi@*,ndbapi@*,ndbapi@* mycluster1;
+------------------------------+
| Command result |
+------------------------------+
| Cluster created successfully |
+------------------------------+
1 row in set (0.38 sec)
mcm> create cluster -P cluster.pkg --processhosts=ndbd@secondary,ndbd@secondary, ndb_mgmd@secondary,ndbapi@*,ndbapi@*,ndbapi@* mycluster2;
+------------------------------+
| Command result |
+------------------------------+
| Cluster created successfully |
+------------------------------+
1 row in set (0.50 sec)
mcm> start cluster mycluster1;
+------------------------------+
| Command result |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (10.22 sec)
mcm> start cluster mycluster2;
+------------------------------+
| Command result |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (11.18 sec)
mcm> autotune --dryrun --writeload=load realtime mycluster1
-> ;
ERROR 12 (00MGR): Illegal value for option writeload
mcm> autotune --dryrun --writeload=low realtime mycluster1
-> ;
ERROR 1003 (00MGR): Internal error: Unable to open dumpfile : 2 No such file or directory
mcm>
*****************************
The mcmd.log on Primary
*****************************
2015-12-24 09:42:54.478: (message) [T0x168a450 CMGR ]: Got new message x_trans {ccc61121 89 1} prepared_trans pc=7
2015-12-24 09:42:54.489: (message) [T0x168a450 CMGR ]: Got new message x_trans {ccc61121 90 0} prepared_trans pc=7
2015-12-24 09:42:54.586: (message) [T0x168a450 CMGR ]: Got new message mgr_autotune_configvalues {ccc61121 91 0}
2015-12-24 09:42:54.586: (message) [T0x168a450 CMGR ]: Got new message x_trans {ccc61121 92 0} prepared_trans pc=11
2015-12-24 09:42:54.587: (message) [T0x168aad0 RECFG]: Settings to autotune dumped to /home/mysql/mcm/mcm_data/clusters/mycluster1/tmp/autotune.ccc61121_88_0.mcm
2015-12-24 09:42:54.677: (message) [T0x168a450 CMGR ]: Got new message x_trans {ccc61121 92 1} abort_trans pc=14
2015-12-24 09:42:54.689: (message) [T0x168a450 CMGR ]: Got new message x_trans {ccc61121 93 0} prepared_trans pc=15
2015-12-24 09:42:54.691: (message) [T0x168a450 CMGR ]: Got new message mgr_process_operationstatus {0 0 0} agent secondary failed
2015-12-24 09:42:54.692: (message) [T0x168a450 CMGR ]: req_id 8 Operation finished with failure for configversion {ccc61121 88 0}
2015-12-24 09:42:54.694: (warning) [T0x168a450 CMGR ]: Operation failed : 1003 Internal error: Unable to open dumpfile : 2 No such file or directory
2015-12-24 09:42:54.695: (message) [T0x168a450 CMGR ]: Got new message x_trans {ccc61121 94 0} abort_trans pc=16
2015-12-24 09:42:54.698: (message) [T0x168a450 CMGR ]: Got new message x_trans {ccc61121 95 0} abort_trans pc=16
2015-12-24 09:42:54.741: (message) [T0x1613320 chass]: Returning error to client : 1003 Internal error: Unable to open dumpfile : 2 No such file or directory
**************************
The mcmd.log on Secondary
***************************
2015-12-24 09:42:54.045: (message) [T0x25b1450 CMGR ]: Got new message mgr_process_operationstatus {0 0 0} 52 finished
2015-12-24 09:42:54.127: (message) [T0x25b1450 CMGR ]: Got new message x_trans {ccc61121 89 1} prepared_trans pc=7
2015-12-24 09:42:54.141: (message) [T0x25b1450 CMGR ]: Got new message x_trans {ccc61121 90 0} prepared_trans pc=7
2015-12-24 09:42:54.237: (message) [T0x25b1450 CMGR ]: Got new message mgr_autotune_configvalues {ccc61121 91 0}
2015-12-24 09:42:54.237: (message) [T0x25b1450 CMGR ]: Got new message x_trans {ccc61121 92 0} prepared_trans pc=11
2015-12-24 09:42:54.327: (message) [T0x25b1450 CMGR ]: Got new message x_trans {ccc61121 92 1} abort_trans pc=14
2015-12-24 09:42:54.342: (message) [T0x25b1450 CMGR ]: Got new message x_trans {ccc61121 93 0} prepared_trans pc=15
2015-12-24 09:42:54.343: (message) [T0x25b1450 CMGR ]: Got new message x_trans {ccc61121 94 0} abort_trans pc=16
2015-12-24 09:42:54.344: (message) [T0x25b1450 CMGR ]: Got new message mgr_process_operationstatus {0 0 0} agent secondary failed
2015-12-24 09:42:54.344: (message) [T0x25b1450 CMGR ]: req_id 8 Operation finished with failure for configversion {ccc61121 88 0}
2015-12-24 09:42:54.346: (warning) [T0x25b1450 CMGR ]: Operation failed : 1003 Internal error: Unable to open dumpfile : 2 No such file or directory
2015-12-24 09:42:54.349: (message) [T0x25b1450 CMGR ]: Got new message x_trans {ccc61121 95 0} abort_trans pc=16
[24 Dec 2015 3:23]
Ivan Ma
Tried to run independently adding 2 clusters (mycluster1, mycluster2) on same machine, it has no problem at all. (Tried both on Primary and Secondary). The issue only happens when there the site has 2 hosts. One cluster on primary and One cluster on secondary.
mcm> list sites;
Empty set (0.02 sec)
mcm> show status -r mycluster1;
ERROR 3003 (00MGR): Command requires a site to be defined
mcm> create site --hosts=secondary mysite;
+---------------------------+
| Command result |
+---------------------------+
| Site created successfully |
+---------------------------+
1 row in set (1.63 sec)
mcm> add package -b /home/mysql/mcm/cluster-latest cluster.pkg;
+----------------------------+
| Command result |
+----------------------------+
| Package added successfully |
+----------------------------+
1 row in set (0.66 sec)
mcm> create cluster -P cluster.pkg --processhosts=ndbd@secondary,ndbd@secondary,ndb_mgmd@primary,ndbapi@*,ndbapi@*,ndbapi@* mycluster1
-> ;
ERROR 2004 (00MGR): Host primary is not a member of site mysite
mcm> create cluster -P cluster.pkg --processhosts=ndbd@secondary,ndbd@secondary,ndb_mgmd@secondary,ndbapi@*,ndbapi@*,ndbapi@* mycluster1
-> ;
+------------------------------+
| Command result |
+------------------------------+
| Cluster created successfully |
+------------------------------+
1 row in set (0.42 sec)
mcm> create cluster -P cluster.pkg --processhosts=ndbd@secondary,ndbd@secondary,ndb_mgmd@secondary,ndbapi@*,ndbapi@*,ndbapi@* mycluster2;
+------------------------------+
| Command result |
+------------------------------+
| Cluster created successfully |
+------------------------------+
1 row in set (0.39 sec)
mcm> set PortNumber:ndb_mgmd:49=1286 mycluster1;
+-----------------------------------+
| Command result |
+-----------------------------------+
| Cluster reconfigured successfully |
+-----------------------------------+
1 row in set (0.42 sec)
mcm> set PortNumber:ndb_mgmd:49=1386 mycluster2;
+-----------------------------------+
| Command result |
+-----------------------------------+
| Cluster reconfigured successfully |
+-----------------------------------+
1 row in set (0.47 sec)
mcm> start cluster mycluster1;
+------------------------------+
| Command result |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (9.36 sec)
mcm> start cluster mycluster2;
+------------------------------+
| Command result |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (21.55 sec)
mcm> autotune --dryrun --writeload=low realtime mycluster1;
+--------------------------------------------------------------------------------------+
| Command result |
+--------------------------------------------------------------------------------------+
| Autotuning calculation complete. Please check log for settings that will be applied. |
+--------------------------------------------------------------------------------------+
1 row in set (0.69 sec)
mcm> autotune --dryrun --writeload=low realtime mycluster2;
+--------------------------------------------------------------------------------------+
| Command result |
+--------------------------------------------------------------------------------------+
| Autotuning calculation complete. Please check log for settings that will be applied. |
+--------------------------------------------------------------------------------------+
1 row in set (0.57 sec)
[24 Dec 2015 3:29]
Ivan Ma
A more simple test is ********************************* As long as the site has 2 hosts, and creating a cluster on a single node ONLY, the other hosts may look for the dump file which it is not there and produce INTERNAL ERROR. ********************************* Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mcm> list sites; Empty set (0.01 sec) mcm> create site --hosts=primary,secondary mysite; +---------------------------+ | Command result | +---------------------------+ | Site created successfully | +---------------------------+ 1 row in set (1.72 sec) mcm> add package -b /home/mysql/mcm/cluster-latest cluster.pkg; +----------------------------+ | Command result | +----------------------------+ | Package added successfully | +----------------------------+ 1 row in set (0.64 sec) mcm> create cluster -P cluster.pkg --processhosts=ndbd@primary,ndbd@primary,ndb_mgmd@primary,ndbapi@*,ndbapi@*,ndbapi@* mycluster1; +------------------------------+ | Command result | +------------------------------+ | Cluster created successfully | +------------------------------+ 1 row in set (0.43 sec) mcm> start cluster mycluster1; +------------------------------+ | Command result | +------------------------------+ | Cluster started successfully | +------------------------------+ 1 row in set (10.28 sec) mcm> autotune --dryrun --writeload=low realtime mycluster1; ERROR 1003 (00MGR): Internal error: Unable to open dumpfile : 2 No such file or directory mcm>
[24 Dec 2015 4:40]
MySQL Verification Team
Hi, Thanks for detailed report, I managed to verify the problem as described. all best Bogdan
[27 Feb 2016 2:35]
Daniel So
Added the following entry to the MySQL Cluster Manager 1.4.1 changelog: "With two clusters running separately on two different hosts in the same site, the autotune --dryrun command for a cluster failed with an internal error, complaining that a dump file [for the other cluster running on the other host] could not be opened because it did not exist on the host—which was to be expected."
