[Bug 2034656] Re: ESM archive getting DoSed with legitimate traffic every day at 06:25 (cron.daily time)

[Julian Andres Klode]

Unfortunately as far as trusty is concerned, it uses the legacy cron job
and not the new systemd timers which spread the load over 24h, so it
only is able to randomize over 30 minutes starting from the cron.daily
run time (6am IIRC, probably UTC as everyone runs servers in UTC).

I don't think anyone remembers why we didn't backport the switch to
timers to it (did we not have [working] timers in 18.04), but also they
shouldn't then see more load than the archive mirror, this shouldn't be
a new problem in that sense.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/2034656

Title:
  ESM archive getting DoSed with legitimate traffic every day at 06:25
  (cron.daily time)

Status in cloud-images:
  New
Status in apt package in Ubuntu:
  New
Status in ubuntu-advantage-tools package in Ubuntu:
  Invalid

Bug description:
  Hi,

  We're seeing frequent alerts on the Ubuntu ESM archive servers due to
  surges in requests. On two systems, I'm seeing this:

  | Sep  6 05:47:16 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 05:47:17 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 10:49:35 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 10:49:35 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 17:17:16 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 17:17:17 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 23:47:16 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 23:47:17 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  7 01:55:02 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  7 01:55:02 machine-2 systemd[1]: Finished Update the local ESM caches.

  On another:

  | Sep  6 02:41:02 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 02:41:03 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 09:02:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 09:02:41 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 15:32:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 15:32:41 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 22:02:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 22:02:41 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  7 04:32:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  7 04:32:42 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.

  This is all from `/usr/lib/systemd/system/esm-cache.service` which
  calls `/usr/lib/ubuntu-advantage/esm_cache.py`.

  Can we please have this run less frequent? Perhaps only once daily
  which aligns with APT and apt-daily-upgrade.service / unattended-
  upgrades?

  Perhaps check existence of a file and run if not, then age of that
  same file and only run if it's older than a day?

  I think, from what I can see, this may be triggered from
  /lib/systemd/system/ua-timer.timer and /etc/apt/apt.conf.d/20apt-esm-
  hook.conf?

  See also LP:1554848 which was for APT.

  On Trusty and Xenial clients we only seem to update daily, but the
  problem is worse as it's a cron.daily job, so all clients fire
  simultaneously - could we get this changed to a cron.d job with a
  randomised firing time instead?

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/2034656/+subscriptions