[Bug 2034656] Re: ESM archive getting DoSed with legitimate traffic every day at 06:25 (cron.daily time)

Fri Mar 8 03:23:00 UTC 2024

Maybe it is this as Grant (orndorffgrant) mentioned earlier?

| APT::Periodic::Download-Upgradeable-Packages "0";

During the time of a surge, we saw this:

| Hits      h%  Vis.     v%   Bandwidth Mtd  Proto    Data
| ----- ------ ----- ------ ----------- ---- -------- ----
| 3772   0.82%  2461  0.85%   56.62 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-modules-extra-5.4.0-1116-azure_5.4.0-1116.123~18.04.1_amd64.deb
| 3246   0.71%  2147  0.74%   41.55 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-modules-5.4.0-1116-azure_5.4.0-1116.123~18.04.1_amd64.deb
| 3850   0.84%  2585  0.89%   41.15 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-azure-5.4-headers-5.4.0-1116_5.4.0-1116.123~18.04.1_all.deb
| 3580   0.78%  2364  0.81%   32.93 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-signed-azure-5.4/linux-image-5.4.0-1116-azure_5.4.0-1116.123~18.04.1_amd64.deb
| 3607   0.78%  2478  0.85%   18.01 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-azure-5.4-tools-5.4.0-1116_5.4.0-1116.123~18.04.1_amd64.deb
| 3523   0.77%  2413  0.83%    5.33 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-headers-5.4.0-1116-azure_5.4.0-1116.123~18.04.1_amd64.deb
| 3635   0.79%  2510  0.86%    2.00 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-azure-5.4-cloud-tools-5.4.0-1116_5.4.0-1116.123~18.04.1_amd64.deb
| 3604   0.78%  2508  0.86%    1.93 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-cloud-tools-5.4.0-1116-azure_5.4.0-1116.123~18.04.1_amd64.deb
| 3431   0.75%  2421  0.83%    1.83 GiB GET  HTTP/1.1 /infra/ubuntu/pool/main/l/linux-azure-5.4/linux-tools-5.4.0-1116-azure_5.4.0-1116.123~18.04.1_amd64.deb

This kernel would be for Bionic VMs in Azure - as Junien (axino) has
pointed out earlier.

`APT::Periodic::Download-Upgradeable-Packages` is defined in
`/etc/apt/apt.conf.d/10periodic` and shipped out via `update-notifier-
common`.

** Also affects: update-notifier (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/2034656

Title:
  ESM archive getting DoSed with legitimate traffic every day at 06:25
  (cron.daily time)

Status in cloud-images:
  New
Status in apt package in Ubuntu:
  New
Status in ubuntu-advantage-tools package in Ubuntu:
  Invalid
Status in update-notifier package in Ubuntu:
  New

Bug description:
  Hi,

  We're seeing frequent alerts on the Ubuntu ESM archive servers due to
  surges in requests. On two systems, I'm seeing this:

  | Sep  6 05:47:16 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 05:47:17 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 10:49:35 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 10:49:35 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 17:17:16 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 17:17:17 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 23:47:16 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 23:47:17 machine-2 systemd[1]: Finished Update the local ESM caches.
  | Sep  7 01:55:02 machine-2 systemd[1]: Starting Update the local ESM caches...
  | Sep  7 01:55:02 machine-2 systemd[1]: Finished Update the local ESM caches.

  On another:

  | Sep  6 02:41:02 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 02:41:03 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 09:02:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 09:02:41 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 15:32:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 15:32:41 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  6 22:02:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  6 22:02:41 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.
  | Sep  7 04:32:40 is-bastion-ps5 systemd[1]: Starting Update the local ESM caches...
  | Sep  7 04:32:42 is-bastion-ps5 systemd[1]: Finished Update the local ESM caches.

  This is all from `/usr/lib/systemd/system/esm-cache.service` which
  calls `/usr/lib/ubuntu-advantage/esm_cache.py`.

  Can we please have this run less frequent? Perhaps only once daily
  which aligns with APT and apt-daily-upgrade.service / unattended-
  upgrades?

  Perhaps check existence of a file and run if not, then age of that
  same file and only run if it's older than a day?

  I think, from what I can see, this may be triggered from
  /lib/systemd/system/ua-timer.timer and /etc/apt/apt.conf.d/20apt-esm-
  hook.conf?

  See also LP:1554848 which was for APT.

  On Trusty and Xenial clients we only seem to update daily, but the
  problem is worse as it's a cron.daily job, so all clients fire
  simultaneously - could we get this changed to a cron.d job with a
  randomised firing time instead?

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/2034656/+subscriptions