Description:
By looking at the service file (/lib/systemd/system/mysql.service) I see that these options are specified:
Restart=on-failure
RestartPreventExitStatus=1
so my understanding is that mysqld should not be restarted repeatedly if it exits with status 1.
I have put a wrong option in my.cnf and when I try to start the service I do get that it fails, but in the background systemd is still trying repeatedly to start mysqld and fills my error log.
Eventually it did finish with fail and stop restarting but for some other reason "Start request repeated too quickly."
My understanding is that if there's a wrongly set option in my.cnf mysqld shouldn't be started again until this is fixed but maybe I'm missing something here.
Here's how it looks for me:
It restarted 175 times before stopping for repeating too quickly.
root@t-ubuntu1604-64:/var/log/mysql# grep -c "invalid-option" error.log
175
root@t-ubuntu1604-64:/var/log/mysql# sudo -u mysql /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
Initialization of mysqld failed: 0
root@t-ubuntu1604-64:/var/log/mysql# echo $?
1
In the "systemctl status mysql" output you can see it is activating/deactivating repeatedly:
ubuntu@t-ubuntu1604-64:~$ sudo systemctl status mysql
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: deactivating (final-sigterm) (Result: exit-code)
Process: 13479 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 13438 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Tasks: 14
Memory: 171.8M
CPU: 188ms
CGroup: /system.slice/mysql.service
└─13482 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
Jul 26 14:50:15 t-ubuntu1604-64 systemd[1]: Starting MySQL Community Server...
Jul 26 14:50:15 t-ubuntu1604-64 mysqld[13479]: Initialization of mysqld failed: 0
Jul 26 14:50:15 t-ubuntu1604-64 systemd[1]: mysql.service: Control process exited, code=exited status=1
ubuntu@t-ubuntu1604-64:~$ sudo systemctl status mysql
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-pre) since Wed 2017-07-26 14:50:17 UTC; 3ms ago
Process: 13479 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Control: 13515 (mysql-systemd-s)
Tasks: 2
Memory: 640.0K
CPU: 2ms
CGroup: /system.slice/mysql.service
└─control
├─13515 /bin/bash /usr/share/mysql/mysql-systemd-start pre
├─13520 /bin/bash /usr/share/mysql/mysql-systemd-start pre
├─13521 /bin/bash /usr/share/mysql/mysql-systemd-start pre
├─13522 /bin/bash /usr/share/mysql/mysql-systemd-start pre
└─13523 /bin/bash /usr/share/mysql/mysql-systemd-start pre
Jul 26 14:50:17 t-ubuntu1604-64 systemd[1]: Starting MySQL Community Server...
ubuntu@t-ubuntu1604-64:~$ sudo systemctl status mysql
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: deactivating (final-sigterm) (Result: exit-code)
Process: 13559 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 13515 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Tasks: 13
Memory: 170.3M
CPU: 197ms
CGroup: /system.slice/mysql.service
└─13563 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
And in the journalctl output this:
Jul 26 14:51:28 t-ubuntu1604-64 systemd[1]: mysql.service: Control process exited, code=exited status=1
Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: Failed to start MySQL Community Server.
Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Unit entered failed state.
Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Failed with result 'exit-code'.
Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: Stopped MySQL Community Server.
Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: Starting MySQL Community Server...
Jul 26 14:51:30 t-ubuntu1604-64 audit[15996]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=15996 comm="apparmor_parser"
Jul 26 14:51:30 t-ubuntu1604-64 kernel: audit: type=1400 audit(1501080690.552:216): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=15996 comm="apparmor_pars
Jul 26 14:51:30 t-ubuntu1604-64 mysqld[15998]: Initialization of mysqld failed: 0
Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Control process exited, code=exited status=1
Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: Failed to start MySQL Community Server.
Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Unit entered failed state.
Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Failed with result 'exit-code'.
Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: Stopped MySQL Community Server.
Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Start request repeated too quickly.
Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: Failed to start MySQL Community Server.
Thanks!
How to repeat:
Use ubuntu xenial with mysql 5.7.19 and add invalid option to /etc/mysql/mysql.conf.d/my.cnf
Check error log, systemctl and journalctl for repeated start of mysqld.
Suggested fix:
If mysqld can't start because of wrong options (this might apply to some other circumstances) no need to restart it repeatedly.
Description: By looking at the service file (/lib/systemd/system/mysql.service) I see that these options are specified: Restart=on-failure RestartPreventExitStatus=1 so my understanding is that mysqld should not be restarted repeatedly if it exits with status 1. I have put a wrong option in my.cnf and when I try to start the service I do get that it fails, but in the background systemd is still trying repeatedly to start mysqld and fills my error log. Eventually it did finish with fail and stop restarting but for some other reason "Start request repeated too quickly." My understanding is that if there's a wrongly set option in my.cnf mysqld shouldn't be started again until this is fixed but maybe I'm missing something here. Here's how it looks for me: It restarted 175 times before stopping for repeating too quickly. root@t-ubuntu1604-64:/var/log/mysql# grep -c "invalid-option" error.log 175 root@t-ubuntu1604-64:/var/log/mysql# sudo -u mysql /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid Initialization of mysqld failed: 0 root@t-ubuntu1604-64:/var/log/mysql# echo $? 1 In the "systemctl status mysql" output you can see it is activating/deactivating repeatedly: ubuntu@t-ubuntu1604-64:~$ sudo systemctl status mysql ● mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: deactivating (final-sigterm) (Result: exit-code) Process: 13479 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE) Process: 13438 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) Tasks: 14 Memory: 171.8M CPU: 188ms CGroup: /system.slice/mysql.service └─13482 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid Jul 26 14:50:15 t-ubuntu1604-64 systemd[1]: Starting MySQL Community Server... Jul 26 14:50:15 t-ubuntu1604-64 mysqld[13479]: Initialization of mysqld failed: 0 Jul 26 14:50:15 t-ubuntu1604-64 systemd[1]: mysql.service: Control process exited, code=exited status=1 ubuntu@t-ubuntu1604-64:~$ sudo systemctl status mysql ● mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: activating (start-pre) since Wed 2017-07-26 14:50:17 UTC; 3ms ago Process: 13479 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE) Control: 13515 (mysql-systemd-s) Tasks: 2 Memory: 640.0K CPU: 2ms CGroup: /system.slice/mysql.service └─control ├─13515 /bin/bash /usr/share/mysql/mysql-systemd-start pre ├─13520 /bin/bash /usr/share/mysql/mysql-systemd-start pre ├─13521 /bin/bash /usr/share/mysql/mysql-systemd-start pre ├─13522 /bin/bash /usr/share/mysql/mysql-systemd-start pre └─13523 /bin/bash /usr/share/mysql/mysql-systemd-start pre Jul 26 14:50:17 t-ubuntu1604-64 systemd[1]: Starting MySQL Community Server... ubuntu@t-ubuntu1604-64:~$ sudo systemctl status mysql ● mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: deactivating (final-sigterm) (Result: exit-code) Process: 13559 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE) Process: 13515 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) Tasks: 13 Memory: 170.3M CPU: 197ms CGroup: /system.slice/mysql.service └─13563 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid And in the journalctl output this: Jul 26 14:51:28 t-ubuntu1604-64 systemd[1]: mysql.service: Control process exited, code=exited status=1 Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: Failed to start MySQL Community Server. Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Unit entered failed state. Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Failed with result 'exit-code'. Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Service hold-off time over, scheduling restart. Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: Stopped MySQL Community Server. Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: Starting MySQL Community Server... Jul 26 14:51:30 t-ubuntu1604-64 audit[15996]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=15996 comm="apparmor_parser" Jul 26 14:51:30 t-ubuntu1604-64 kernel: audit: type=1400 audit(1501080690.552:216): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=15996 comm="apparmor_pars Jul 26 14:51:30 t-ubuntu1604-64 mysqld[15998]: Initialization of mysqld failed: 0 Jul 26 14:51:30 t-ubuntu1604-64 systemd[1]: mysql.service: Control process exited, code=exited status=1 Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: Failed to start MySQL Community Server. Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Unit entered failed state. Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Failed with result 'exit-code'. Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Service hold-off time over, scheduling restart. Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: Stopped MySQL Community Server. Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: mysql.service: Start request repeated too quickly. Jul 26 14:51:32 t-ubuntu1604-64 systemd[1]: Failed to start MySQL Community Server. Thanks! How to repeat: Use ubuntu xenial with mysql 5.7.19 and add invalid option to /etc/mysql/mysql.conf.d/my.cnf Check error log, systemctl and journalctl for repeated start of mysqld. Suggested fix: If mysqld can't start because of wrong options (this might apply to some other circumstances) no need to restart it repeatedly.