MySQL keeps restarting on its own. The alert fires at 06:37, the database is back before anyone opens a laptop, the log holds a wall of Lost connection to MySQL server during query, and uptime says 41 days. The next morning it is 06:12, the morning after that nothing.
On a default Ubuntu or Debian server, unattended-upgrades installs security updates on a daily systemd timer, and when one touches MySQL, dpkg stops mysqld, upgrades the data directory, and starts it again. Nothing rebooted. The database was replaced underneath a host that never stopped running. We measured it on an Amazon Elastic Compute Cloud (EC2) instance.
Is this a MySQL problem?
No. It comes from the Debian package’s install script, generated by the standard packaging tooling and marked as such in the file. Every Debian package carrying a daemon ships the same block: the packaging can’t tell a database from a print spooler.
What schedules it is unattended-upgrades, a Python program on a timer whose whole policy is two small config files. On a stock Ubuntu 24.04 image they say run daily and treat the security pocket as fair game. The blacklist is empty, so there is no exception for databases. Nobody configured that; it’s what the image ships.
Ubuntu also ships needrestart, wired into apt, which restarts any service whose libraries were replaced on disk. In one run an OpenSSL update arrived, MySQL was held back and never upgraded, and needrestart restarted it anyway, along with sshd and the journal.
How long does it take the database away?
A systemctl restart mysql costs about four seconds. One automatic security upgrade costs 22.98 seconds, because dpkg does not restart MySQL, it replaces it: mysqld starts and stops three times in those 23 seconds, two of them listening on nothing while the data directory is upgraded. Turn on Automatic-Reboot, which Ubuntu ships off, and a kernel update sets /var/run/reboot-required: our client lost the database for 19.37 seconds, less than the upgrade.
And 23 seconds is the floor, because stopping a real database means flushing every dirty page first and the pool on that server is the 128 MB default. On a second instance with an 8 GB buffer pool, a 5.5 GB table and 2.8 GB of pages dirty, an ordinary systemctl stop mysql took 1 minute 59.8 seconds. The MySQL unit Ubuntu ships gives the stop no timeout at all.
The minute it lands on is chosen at random, by design: the timer runs at 06:00 with RandomizedDelaySec=60m, which spreads the start across the following hour independently on every host. So it arrives at a different minute every morning and on every machine, and the repetition that would make it obvious never forms. Persistent=true fires a missed run as soon as the machine comes back.
How do I tell which one happened?
First, did the machine reboot? The two have different fixes and the same application error. Compare uptime -s against systemctl show -p ActiveEnterTimestamp --value mysql.
Then one line decides it. MY-013381, the “Server upgrade from” line in the MySQL error log, appears only when MySQL finds an older data directory and upgrades it. A crash, an OOM kill, an operator restart and a host reboot can’t produce it. /var/log/apt/history.log and journalctl -u mysql give you the rest.
How do I disable or remove it?
On a database server, remove it. Replacing a running mysqld is a maintenance action, and maintenance belongs to whoever answers for the outage. On a web tier behind a load balancer, leave it on. Two commands:
sudo systemctl disable --now apt-daily-upgrade.timer sudo apt-get purge -y unattended-upgrades
The first stops tonight’s run. The second takes /usr/bin/unattended-upgrade and both config files, and leaves the MySQL update pending until you install it yourself. The timers belong to apt and survive the purge with nothing left to run: we fired apt-daily-upgrade.service by hand afterwards and it installed nothing.
To keep the package and only stop it installing, set one line in /etc/apt/apt.conf.d/20auto-upgrades instead:
APT::Periodic::Unattended-Upgrade "0";
Neither the purge nor that setting touches needrestart, which runs on every apt transaction, including yours: on our purged host a plain apt-get upgrade of OpenSSL restarted MySQL and cost 4.10 seconds. Exclude the database:
echo '$nrconf{override_rc} = { qr(^mysql) => 0 };' \
| sudo tee /etc/needrestart/conf.d/99-mysql.conf
We ran all of this with the 8.0.46 upgrade pending and a client reading and writing throughout; mysqld never moved.
On a web tier, letting the machine patch itself at a minute of its own choosing is a good trade. On a database it should be someone’s decision, in a window they chose. The update still has to happen; purging unattended-upgrades only moves it to that window.
The floor is yours.
0 comments · Moderated · civil & on-topicFirst comment appears here once approved. Questions, corrections, and counterpoints welcome — just no self-promotion.