MySQL Bugs: #44775: MTR fails to bootstrap mysqld on Windows in Pushbuild 2

Bug #44775	MTR fails to bootstrap mysqld on Windows in Pushbuild 2
Submitted:	11 May 2009 7:47	Modified:	18 Jun 2009 15:33
Reporter:	John Embretsen	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: MTR / mysql-test-run	Severity:	S7 (Test Cases)
Version:		OS:	Windows
Assigned to:	Vladislav Vaintroub	CPU Architecture:	Any

Description:
mysql-test-run.pl (MTR) version 2 (in mysql-6.0-*) fails to bootstrap mysqld - and hence start the server and run tests - roughly 50% of the time on Windows platforms in Pushbuild 2 (PB2).

On the PB2 "show_pushes" page in the columns for platforms "win x86 max" and "win x86_64 max" you will notice that quite often there are no mysql-test tests run (0/0) for a given push. Test logs show the following error:

---

mysql-test-run: *** ERROR: Error executing mysqld --bootstrap
Could not install system database from G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test/mysql-test/var/tmp/bootstrap.sql
see G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test/mysql-test/var/log/bootstrap.log for errors

---

var/log/bootstrap.log for a given attempted test run shows:

---

G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test/sql/relwithdebinfo/mysqld.exe --no-defaults --bootstrap --basedir=G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test --datadir=G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test/mysql-test/var/install.db --loose-skip-innodb --loose-skip-falcon --loose-skip-ndbcluster --tmpdir=G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test/mysql-test/var/tmp/ --core-file --language=G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test/sql/share/english --character-sets-dir=G:/pb2/test/sb_1-326761-1234614478.93/mysql-6.0.10-alpha-win-x86_64-test/sql/share/charsets
safe_process[3028]: FATAL ERROR, CreateProcess failed
error: 5, Access is denied.

---

Of the last 20 pushes to mysql-6.0-mtr, no tests were run 9 times on 'win x86 max' and 9 times on 'win x86_64'. It is probably safe to assume that all those failures are caused by this issue, though I have only checked some of the vardirs, as far back as 2009-02-14 and as recent as 2009-05-08.

It is not certain what is causing these errors, but is has been suggested that "The cause is that PB2 runs MTR in a job group that lets it shut down the entire group. MTR2 also tries to create a job group and this seems to fail."
(See Bug#42641)

How to repeat:
See Pushbuild 2 test results for platforms "win x86 max" and "win x86_64 max".

Link to mysql-6.0 PB2 overview:
http://pb2.norway.sun.com/web.py?branch=mysql-6.0&template=show_pushes

Link to test log from failing run on 2009-05-06, platform "win x86_64 max":
http://pb2.norway.sun.com/web.py?action=archive_download&archive_id=496905&pretty=please

Link to vardir from same attempted test run:
http://trollheim.norway.sun.com/archive/496909.vardirs.zip

Suggested fix:
If related, see suggestions in Bug#42641 (mtr.pl fails to run within JobObject).

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/74105

2791 Vladislav Vaintroub	2009-05-14
      Bug #44775 MTR fails to bootstrap mysqld on Windows in Pushbuild 2.
      
      Suspected reason for the failure is that safe_process.exe already runs in a job that does not allow breakaways. 
      The fix is to use a fallback -  make newly created process the root of the new process group. This allows to kill process together with descendants via GenerateConsoleCtrlEvent (CTRL_BREAK_EVENT, pid)

OK let's try this!

Pushed into 5.1.36 (revid:joro@sun.com-20090528073639-yohsb4q1jzg7ycws) (version source revid:bjorn.munch@sun.com-20090525205831-zitq9aysxy5zgf5r) (merge vers: 5.1.36) (pib:6)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/75184

Pushed into 5.1.34-ndb-6.2.18 (revid:magnus.blaudd@sun.com-20090528190554-a9ujwjizp9uauayk) (version source revid:magnus.blaudd@sun.com-20090528190554-a9ujwjizp9uauayk) (merge vers: 5.1.34-ndb-6.2.18) (pib:6)

Pushed into 5.1.34-ndb-6.3.26 (revid:magnus.blaudd@sun.com-20090528190722-ppqpp5qp3b1d6qpb) (version source revid:magnus.blaudd@sun.com-20090528190722-ppqpp5qp3b1d6qpb) (merge vers: 5.1.34-ndb-6.3.26) (pib:6)

Pushed into 5.1.34-ndb-7.0.7 (revid:magnus.blaudd@sun.com-20090528191035-2jvz2zbg6w876bsw) (version source revid:magnus.blaudd@sun.com-20090528191035-2jvz2zbg6w876bsw) (merge vers: 5.1.34-ndb-7.0.7) (pib:6)

Test suite change. No changelog entry needed.

Setting report to NDI pending push into 6.0.x.

Pushed into 5.4.4-alpha (revid:alik@sun.com-20090616183122-chjzbaa30qopdra9) (version source revid:bjorn.munch@sun.com-20090525205945-phe3om6ow3v4snwr) (merge vers: 6.0.12-alpha) (pib:11)

Test suite change. No changelog entry needed.

Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)

Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)

Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)