[Bug 1892358] Re: autopkgtest success rate dropped inhibiting proposed migration

[Łukasz Zemczak]

I am a bit confused on this bug. I know it's a rather 'vague' bug
describing overall flaky/troubling tests, but for instance it has been
marked as Fix Released for all the series already (groovy as well, as
this was devel back then). Now it's again present in the changelog
entries for at least the groovy systemd SRU (probably for others as
well). But because of this, I don't know what's the state of the fixes
in say, hirsute? Are those backported from there? Would be nice if an
explanation why this bug is being re-opened was stated before submitting
the SRU.

Not saying we shouldn't re-use existing bugs, no. Just saying that this
way not much information about the current situation is given.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1892358

Title:
  autopkgtest success rate dropped inhibiting proposed migration

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Bionic:
  Fix Released
Status in systemd source package in Focal:
  Fix Released

Bug description:
  [impact]

  autopkgtests are failing/flaky and prevent other packages from
  migrating to -updates

  [test case]

  check autopkgtest history

  [regression potential]

  in regard to the changed test cases, any regression would likely
  result in either an incorrectly passed test, or an incorrectly failed
  test.

  [scope]

  for systemd, this is needed for x, b, and f.

  tests in g appear to be mostly stable, but I've opened MR (linked from
  this bug) to update the tests there as well.

  i don't plan to update x, as it's reaching ESM in ~6 months, and
  backporting the test fixes is more work than just a simple code copy,
  since there are additional differences/changes needed in the older
  version of systemd (and python3). the failing/flaky tests in x have
  been like that forever, and people have just retried them; we can keep
  retrying them until x moves into ESM next year.

  [original description]

  Hi,
  we had such cases in the past like bug 1817721 for bionic and maybe bug 1892130 is about the same as well. There were more but I didn't want to search for all of them - what I checked is that there are no open ones clearly pointing out the recent further drop in already flaky subtests.

  In particular the tests "tests-in-lxd" and "systemd-fsckd" were known
  to be flaky before, but got even worse.

  Here stats of the last 40 runs, it might be a coincidences that this
  is after 246-2ubuntu1 landed. Could as well be any other change

  groovy
    amd64
  tests-in-lxd                   (F 42% S  0% B 10% => P 45%/) FFFFBFFFFFFFB....FF.B.....F.....F...FBF
  build-login                    (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  unit-config                    (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  networkd-testpy                (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  boot-and-services              (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  boot-smoke                     (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  logind                         (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  storage                        (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  upstream                       (F 35% S  0% B 10% => P 52%/) ..FFB.FFF.FFB....FF.B.....F.F..F....FBF
  udev                           (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  systemd-fsckd                  (F 37% S  0% B 10% => P 50%/) FFFFBFFFFFFFB.FF...FB.....F..........B.
  root-unittests                 (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
    ppc64el
  tests-in-lxd                   (F 25% S  0% B  0% => P 75%/) FFFF....FF............FFF.....F.........
  systemd-fsckd                  (F 35% S  0% B  0% => P 65%/) FFFFFFF...FF........F....FF.F..F........
  root-unittests                 (F  2% S  0% B  0% => P 97%/) ..............................F.........
    s390x
  tests-in-lxd                   (F 52% S  0% B  0% => P 47%/) FFFFFFF.FFFFFFF.FF.........FFFF...F.....
  timedated                      (F  2% S  0% B  0% => P 97%/) ...........F............................
  upstream                       (F 17% S  0% B  0% => P 82%/) .....F......F.F.............FFF...F.....
  systemd-fsckd                  (F 32% S  0% B  0% => P 67%/) FFFFFFF..FF..F.................FF..F....
  root-unittests                 (F 10% S  0% B  0% => P 90%/) ............................FFF...F.....
    arm64
  tests-in-lxd                   (F 40% S  0% B  2% => P 57%/) FFFFF.B...FFF.FF..F..F.........FFF.F....
  logind                         (F  2% S  0% B  2% => P 95%/) ......B...................F.............
  upstream                       (F 22% S  0% B  2% => P 75%/) ...F.FB.....F.F.............F..FFF.F....
  root-unittests                 (F 12% S  0% B  2% => P 85%/) ......B.F...........F.F........F...F....

  (I'm sure LP will make this unreadable, but is is nice in monospace)

  Whatever the root cause is - the success rate of these has reduced so
  much that the (even formerly questionable) practice of retry-until-
  success won't work anymore.

  I have run the two tests in a local VM and systemd-fsckd works there
  while tests-in-lxd seems to trip over the old flaky fellow being
  "boot-and-services".

  We had the discussion in the past, but I think I need to again bring
  up the suggestion to skip "tests-in-lxd" and "systemd-fsckd" until
  they are on reasonable success rates.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1892358/+subscriptions