[Bug 2116119] Re: autopkgtests fail on PS7 due to mandatory proxy

Tue Jul 15 16:52:12 UTC 2025

I suspect that's also the case of livecd-rootfs:
https://autopkgtest.ubuntu.com/packages/l/livecd-rootfs

I'm seeing autopkgtest failing to download small text files from
addresses like people.canonical.com, ports.ubuntu.com, ubuntu-archive-
team.ubuntu.com to name a few.

One can see it's failing for most architectures across different Ubuntu
releases. Not all cases are equal, but I found many logs like the ones
below from jammy and later and I observed the tests that fail like that
ran on STO (PS7) and the ones that didn't fail ran on BOS3 (PS6) for the
same release, version and architecture:

Either this
```
248s autopkgtest [07:29:15]: test default-bootstraps: [-----------------------
248s Building rootfs for project: 'ubuntu-base' subproject: '' template: '' in /adt/tmp.SVp4PQFSeL/ubuntu-base::
248s [2025-07-07 07:29:14] lb_config 
248s [1;37mP[0m: Executing auto/config script.
248s Building on autopkgtest.local
248s Running germinate...
248s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky/main/binary-amd64/Packages.xz file ...
248s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky/main/binary-amd64/Packages.xz file ...
248s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky/main/source/Sources.xz file ...
249s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky/main/source/Sources.xz file ...
249s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky/restricted/binary-amd64/Packages.xz file ...
249s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky/restricted/binary-amd64/Packages.xz file ...
249s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky/restricted/source/Sources.xz file ...
249s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky/restricted/source/Sources.xz file ...
249s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky-updates/main/binary-amd64/Packages.xz file ...
249s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky-updates/main/binary-amd64/Packages.xz file ...
249s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky-updates/main/source/Sources.xz file ...
249s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky-updates/main/source/Sources.xz file ...
249s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky-updates/restricted/binary-amd64/Packages.xz file ...
249s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky-updates/restricted/binary-amd64/Packages.xz file ...
249s Downloading http://archive.ubuntu.com/ubuntu/dists/plucky-updates/restricted/source/Sources.xz file ...
249s Decompressing http://archive.ubuntu.com/ubuntu/dists/plucky-updates/restricted/source/Sources.xz file ...
249s * Downloading https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.plucky/STRUCTURE
249s ! Could not open (any of):
249s !   https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.plucky/STRUCTURE
```

or this

```
88s [1;37mP[0m: Executing auto/config script.
 88s Building on autopkgtest.local
 88s Running germinate...
 88s Downloading http://ports.ubuntu.com/ubuntu-ports/dists/plucky/main/binary-s390x/Packages.xz file ...
 88s Downloading http://ports.ubuntu.com/ubuntu-ports/dists/plucky/main/binary-s390x/Packages.bz2 file ...
 88s Downloading http://ports.ubuntu.com/ubuntu-ports/dists/plucky/main/binary-s390x/Packages.gz file ...
 88s Downloading http://ports.ubuntu.com/ubuntu-ports/dists/plucky/main/binary-s390x/Packages file ...
 89s Traceback (most recent call last):
 89s   File "/usr/bin/germinate", line 33, in <module>
 89s     main(sys.argv)
 89s     ~~~~^^^^^^^^^^
 89s   File "/usr/lib/python3/dist-packages/germinate/scripts/germinate_main.py", line 227, in main
 89s     g.parse_archive(archive)
 89s     ~~~~~~~~~~~~~~~^^^^^^^^^
 89s   File "/usr/lib/python3/dist-packages/germinate/germinator.py", line 628, in parse_archive
 89s     for indextype, section in archive.sections():
 89s                               ~~~~~~~~~~~~~~~~^^
 89s   File "/usr/lib/python3/dist-packages/germinate/archive.py", line 229, in sections
 89s     packages = self._open_tag_files(
 89s         self._mirrors,
 89s     ...<4 lines>...
 89s         "binary-" + self._arch + "/Packages",
 89s     )
 89s   File "/usr/lib/python3/dist-packages/germinate/archive.py", line 208, in _open_tag_files
 89s     raise OSError("no %s files found" % tagfile_type)
 89s OSError: no Packages files found
```

Here are some examples (there are many more):

Plucky version 25.04.26

amd64:
 - BOS03 (pass) on 2025-06-24: https://autopkgtest.ubuntu.com/results/autopkgtest-plucky/plucky/amd64/l/livecd-rootfs/20250624_070159_75d25@/log.gz
 - STO01 (fail) on 2025-07-07: https://autopkgtest.ubuntu.com/results/autopkgtest-plucky/plucky/amd64/l/livecd-rootfs/20250707_073404_6cd58@/log.gz

s390x:
  - BOS03 (pass) on 2025-06-19: https://autopkgtest.ubuntu.com/results/autopkgtest-plucky/plucky/s390x/l/livecd-rootfs/20250619_032036_c3858@/log.gz
  - STO01 (fail) on 2025-07-07: 

Questing version 25.10.9
amd64:
  - BOS03 (pass) on 2025-07-03: https://autopkgtest.ubuntu.com/results/autopkgtest-questing/questing/amd64/l/livecd-rootfs/20250703_170337_23a24@/log.gz
  - STO01 (fail) on 2025-07-02: https://autopkgtest.ubuntu.com/results/autopkgtest-questing/questing/amd64/l/livecd-rootfs/20250702_181836_3d6e2@/log.gz

arm64:
  - BOS03 (pass) on 2025-07-03: https://autopkgtest.ubuntu.com/results/autopkgtest-questing/questing/arm64/l/livecd-rootfs/20250703_180652_de91b@/log.gz
  - STO01 (pass) on 2025-07-03: https://autopkgtest.ubuntu.com/results/autopkgtest-questing/questing/arm64/l/livecd-rootfs/20250703_152853_f055c@/log.gz

I'm not entirely sure but I suspect that the restrictions used in the
livecd-rootfs autopkgtests would be the reason for failure as it they
require a clean environment (most likely proxy settings are not passed
through from the host, except for apt settings as suggested by
autopkgtest-build-qemu man page for example). The tests themselves are
quite simple, they essentially run the live-build/auto/* shell scripts
which don't touch proxy settings AFAICT.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to update-manager in Ubuntu.
https://bugs.launchpad.net/bugs/2116119

Title:
  autopkgtests fail on PS7 due to mandatory proxy

Status in apport package in Ubuntu:
  Confirmed
Status in livecd-rootfs package in Ubuntu:
  New
Status in ubuntu-boot-test package in Ubuntu:
  Confirmed
Status in update-manager package in Ubuntu:
  Confirmed

Bug description:
  Canonical PS7 (prodstack 7) now requires[1] a proxy for any outgoing
  connection, even to archive.ubuntu.com. This breaks assumptions made
  by some of the update-manager tests[2], which expect a direct
  connection.

  Long thread on internal mattermost here[3].

  For example:
  1320s ERROR: testOriginMatcherSimple (test_update_origin.TestOriginMatcher.testOriginMatcherSimple)
  1320s ----------------------------------------------------------------------
  1320s Traceback (most recent call last):
  1320s   File "/usr/lib/python3/dist-packages/apt/cache.py", line 565, in update
  1320s     res = self._cache.update(fetch_progress, slist, pulse_interval)
  1320s apt_pkg.Error: E:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/noble/InRelease  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::101). - connect (101: Network is unreachable)

  Curiously, these tests seem to have passed in non-amd64 architectures,
  but closer inspection showed that these ran in good-old-PS6 (bos03),
  where direct connections to archive.ubuntu.com (and
  changelogs.ubuntu.com) are allowed.

  This bug might be invalid for update-manager in the end, and we could
  decide to allow such direct connections in PS7, but for the meantime,
  we need a place to discuss, and reference, because we are about to
  have to skip these tests for SRUs and migrations that this is
  blocking.

  Keep in mind that forcing all these tests to go through a proxy will
  also probably mean we will lose test coverage, since we are losing the
  directly-connected test scenario.

  Some troubleshooting was done already, here is some info.

  a) provisioning of proxy settings for PS7 autopkgtest vms:

    - echo http_proxy=http://egress.ps7.internal:3128/ >> /etc/environment
    - echo https_proxy=http://egress.ps7.internal:3128/ >> /etc/environment
    - echo no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain >> /etc/environment
    - echo 'Acquire::http::Proxy \"http://egress.ps7.internal:3128/\";' > /etc/apt/apt.conf.d/99proxy

  1. No clue where this decision was made
  2. https://autopkgtest.ubuntu.com/results/autopkgtest-questing/questing/amd64/u/update-manager/20250707_093622_8d0d0@/log.gz
  3. https://chat.canonical.com/canonical/pl/g7hugd5p53nwmmibntd359z3gr

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apport/+bug/2116119/+subscriptions