Considering using --needs-internet=try in autopkgtest

[Balint Reczey]

Hi Laney,

On Thu, Jan 14, 2021 at 10:36 AM Iain Lane <laney at ubuntu.com> wrote:
>
> As background, our autopkgtest workers can only access the wider
> internet via a http/s proxy. DNS resolution also works, but other
> services most likely will not.
>
> A while ago I advised some folks to make a test 'skippable', where it
> was trying to access resources that couldn't be accessed due to this.
>
> This patch was duly forwarded to the Debian maintainer - thanks - as
>
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969057
>
> you can read that log for the full discussion, but the tl;dr is that
> they feel that the "needs-internet" restriction which is already present
> in the package should cover this case.
>
> I've got to say that I am kind of convinced by that.
> README.package-tests.rst says:
>
>   The test needs unrestricted internet access
>
> and also
>
>   If a test does use the internet outside of the pre-configured apt
>   domain, the test must be marked with the needs-internet restriction.
>   Using the internet usually makes tests less reliable, so this should
>   be kept to a minimum. But for many packages their main purpose is to
>   interact with remote web services and thus their testing should
>   actually cover those too, to ensure that the distribution package
>   keeps working with their corresponding web service.
>
> Given this, I think we're on dodgy ground asking the maintainer to make
> changes. We have two choices that I can see:
>
>   * Accept that some tests which need to access external services will
>     fail, and we might have to hint (ignore) those failures.
>   * Run autopkgtest with "--needs-internet=try", which means that we run
>     the test but treat it as flaky if it fails.
>
> I prefer the latter since it will result in less work for the release
> team, but it does come with a caveat which is that needs-internet tests
> will essentially no longer be able to fail. Given that we don't have a
> restriction which expresses our particular flavour of internet access,
> tests which only use http/s, and so are totally fine for us to run fully
> currently, will regress in coverage. We won't be able to detect legit
> failures any more.
>
> In the medium term, if scheduled, we could try to work on a way to
> express our type of internet access and then have packages use that*,
> but here I'd like to limit the discussion to solutions we can implement
> now, please.

Thank you for moving this recurring [1] topic forward. The status quo
is roughly the first option, filing hints or even worse growing delta
or poking Debian maintainers which is even more labour intensive and
less pleasant for them.
I'd very much support at least moving to the latter option, from the
ones to be discussed now.

Also I'd oppose trying to add a new restriction with our particular
proxy/DNS configuration, because this would be even more labour
intensive on both Ubuntu's and Debian's side and would result even
more friction between the participants of the projects. IMO there is a
good, frictionless solution for the problem, but I keep that for a
different thread/discussion.

Cheers,
Balint

[1] https://bugs.launchpad.net/auto-package-testing/+bug/1892903