Something restarts my MySQL every morning. It isn’t MySQL.

The alert fires at 06:37 and by the time anyone opens a laptop the database is back. The application log holds a wall of Lost connection to MySQL server during query, uptime says 41 days, and the error log records a clean shutdown: no crash, no signal, no out-of-memory kill. The next morning it happens at 06:12, and the morning after that not at all.

On a default Ubuntu or Debian server, unattended-upgrades installs security updates on a daily systemd timer, and when one touches MySQL, dpkg stops mysqld, upgrades the data directory to match the new binary, and starts it again while the machine stays up. Nothing rebooted. The database was replaced underneath a host that never stopped running.

That is deliberate, and the part nobody signed up for is that “a service” includes the database. We put a client in front of MySQL on an Amazon Elastic Compute Cloud (EC2) instance and let the timer do what it does every morning.

Is this a MySQL problem?

No. The restart was not written by anyone at MySQL. It comes from a block in the Debian package’s install script, generated by the standard packaging tooling and marked as such in the file, and every Debian package carrying a daemon ships the same one. It starts the service on first install and restarts it on every upgrade after, because the packaging cannot tell a database from a print spooler.

The second mechanism is not MySQL-specific either. Ubuntu ships needrestart, wired into apt, which restarts any service whose libraries were replaced on disk. In one run an OpenSSL update arrived, MySQL was held back and never upgraded, and needrestart restarted it anyway, along with sshd and the journal. Postgres, Redis and nginx would have been on that list too.

What does unattended-upgrades actually do?

It is a Python program plus two systemd units that install updates without a human: refresh the package lists, take what is upgradable from origins it is allowed to touch, install with dpkg, and reboot afterwards if a package asked for one and the operator turned that on. Its whole policy is two small config files, which on a stock Ubuntu 24.04 image say run daily and treat the security pocket as fair game. The blacklist is empty, so there is no exception for databases. Nobody configured that; it is what the image ships.

Does the RHEL family do this too?

Partly, and the difference is the half that matters. We ran the same checks on a stock Amazon Linux 2023 image:

Every row was read off the image rather than from documentation.

The last row matters: the RPM restarts the daemon on upgrade exactly as the Debian package does, and a dnf upgrade of MariaDB stopped and started the database. What Ubuntu adds is not the restart, it is the schedule.

How long does it take the database away?

A systemctl restart mysql costs about four seconds. One automatic security upgrade costs 22.98 seconds, because dpkg does not restart MySQL, it replaces it: mysqld starts and stops three times in those 23 seconds, and two of the three are listening on nothing while the data directory is upgraded. The machine does sometimes reboot on its own, when a kernel update sets /var/run/reboot-required and Automatic-Reboot is on, which Ubuntu ships off: we turned it on and the client lost the database for 19.37 seconds, less than the package upgrade costs.

And 23 seconds is the floor, because our lab server runs the 128 MB default buffer pool with almost nothing in it. Stopping a real database means flushing every dirty page first. On a second instance with an 8 GB buffer pool, a 5.5 GB table and roughly 2.8 GB of pages dirty, an ordinary systemctl stop mysql took 1 minute 59.8 seconds, and nothing bounds it: the MySQL unit Ubuntu ships gives the stop no timeout at all. Scale that pool to the terabyte a real server has and this stops being a blip.

Why does it feel random?

Because it is scheduled to be. The timer runs at 06:00 with RandomizedDelaySec=60m, spreading the start across the following hour independently on every host, which is what the setting is for: it keeps a fleet off the mirrors at the same second. So the restart lands at a different minute every morning and on every machine, and the repetition that would make it obvious never forms. Persistent=true fires a missed run as soon as the machine comes back, which is how one lands ten minutes after a maintenance window rather than during it.

How do I tell which one happened?

First settle whether the machine rebooted, because the two have different fixes and the same application error. Compare uptime -s, when the kernel booted, against systemctl show -p ActiveEnterTimestamp --value mysql, when this mysqld started.

Then three logs and one line. /var/log/apt/history.log names what apt installed, the unattended-upgrades log says what it decided, and journalctl -u mysql shows systemd stopping and starting the service with a gap of tens of seconds where dpkg works. The MySQL error log settles it: MY-013381, the “Server upgrade from” line, appears only when MySQL finds a data directory from an older version and upgrades it, so a crash, an out-of-memory kill, an operator restart and a host reboot cannot produce it.

How do I disable or remove it?

My opinion: on a production database server, automatic installation of database packages should be off. Replacing a running mysqld is a maintenance action, and maintenance actions belong to whoever answers for the outage. On a web tier behind a load balancer, leave it on.

To disable it, set APT::Periodic::Unattended-Upgrade "0" in /etc/apt/apt.conf.d/20auto-upgrades, which is what dpkg-reconfigure -plow unattended-upgrades writes. The daily unit then runs and installs nothing. To stop the timer entirely, systemctl mask it rather than disable it: a disabled unit stays startable and a later upgrade can restore the symlink.

To remove it, apt-get purge unattended-upgrades, which takes the binary and both config files but keeps the logs. It does not take the timers: apt-daily.timer and apt-daily-upgrade.timer ship with apt itself and stay enabled, firing into a binary that is gone. We ran all of this with the upgrade pending and a client probing; MySQL never restarted.

Two caveats. needrestart runs on every apt transaction including yours, so on a purged host a hand-typed apt-get upgrade of OpenSSL still restarted MySQL and cost 4.10 seconds; excluding the database there takes a one-line override in /etc/needrestart/conf.d/. And this moves the outage to a moment you chose rather than removing the need for it.

Conclusion

A database that disappears for twenty seconds at night, on a machine whose uptime says weeks, is unattended-upgrades doing what it was configured to do on the day the machine was built, at a minute systemd picked at random. It is not a MySQL bug, and not really an Ubuntu bug either. What Ubuntu adds is that nobody had to ask, and the machine picks the minute. On a web tier that is a good trade. On a database it should be a decision a person makes, in a window they chose.

Written by

Vinicius Grippa

Writes this blog. Mostly about databases. Boring on purpose.

More about me →