[Bug 1935617] Re: systemd autopkgtest broken on ppc64el with qemu 6.0

[Christian Ehrhardt ]

After some setup and preparation I was bisecting through v5.2.0..v6.0.0
to see which change exactly added this new behavior.

Summary:
 - build reduced qemu
 - build, configure and log-check errors will be skips
 - run the extracted systemd test
 - redirect serial to file
 - check log if the service failed
 - unlikely to be related, but to be sure I as using the same slof.bin on all runs
 - verified with self-built v5.2.0 (food) and v6.0.0 (bad)

The full log is here:
$ git bisect log
git bisect start
# new: [609d7596524ab204ccd71ef42c9eee4c7c338ea4] Update version for v6.0.0 release
git bisect new 609d7596524ab204ccd71ef42c9eee4c7c338ea4
# old: [553032db17440f8de011390e5a1cfddd13751b0b] Update version for v5.2.0 release
git bisect old 553032db17440f8de011390e5a1cfddd13751b0b
# new: [57206696058e3a8b1ee6ec8a129722baa6e7ec9f] hw/block/nvme: merge implicitly/explicitly opened processing masks
git bisect new 57206696058e3a8b1ee6ec8a129722baa6e7ec9f
# new: [4663b72a48fd540cbe16053b01d6839a95656440] intc/arm_gic: Fix gic_irq_signaling_enabled() for vCPUs
git bisect new 4663b72a48fd540cbe16053b01d6839a95656440
# new: [55810e90cc9a5ca18289c25aa5c1e0e2dc77eadb] ppc/spapr: cleanup -machine pseries,nvdimm=X handling
git bisect new 55810e90cc9a5ca18289c25aa5c1e0e2dc77eadb
# old: [cd08b80952657ab9f9ca5b8f374982d9c3159f4a] Replace HTTP links with HTTPS ones: documentation
git bisect old cd08b80952657ab9f9ca5b8f374982d9c3159f4a
# old: [5f3013654e879bb4b22876617fdb235aa22568d3] target/mips: Remove unused headers from op_helper.c
git bisect old 5f3013654e879bb4b22876617fdb235aa22568d3
# new: [ab32b78cd1b3b31950c4332f0fa8b192295d77fd] audio: Simplify audio_bug() removing old code
git bisect new ab32b78cd1b3b31950c4332f0fa8b192295d77fd
# new: [711dfb2423a85fba91884b5a90a8c0712c152ce2] hw/ppc: Do not re-read the clock on pre_save if doing savevm
git bisect new 711dfb2423a85fba91884b5a90a8c0712c152ce2
# old: [cf77e27f1e16d8eca761e5c966ea80291254a32c] target/ppc: replaced the TODO with LOG_UNIMP and add break for silence warnings
git bisect old cf77e27f1e16d8eca761e5c966ea80291254a32c
# new: [728aa6f6ffd8cf21d9485eff7ac1926b00155ed2] hw/ppc/spapr_tpm_proxy: Fix hexadecimal format string specifier
git bisect new 728aa6f6ffd8cf21d9485eff7ac1926b00155ed2
# old: [132954a8307635c08c8e3c4d63d4af9fa865a459] ppc/translate: Turn the helper macros into functions
git bisect old 132954a8307635c08c8e3c4d63d4af9fa865a459
# old: [91699dbf30a94dea2575ae193412c364c7f3a5fd] ppc/translate: Raise exceptions after setting the cc
git bisect old 91699dbf30a94dea2575ae193412c364c7f3a5fd
# new: [bcb0b7b1a1c05707304f80ca6f523d557816f85c] ppc/translate: Rewrite gen_lxvdsx to use gvec primitives
git bisect new bcb0b7b1a1c05707304f80ca6f523d557816f85c
# first new commit: [bcb0b7b1a1c05707304f80ca6f523d557816f85c] ppc/translate: Rewrite gen_lxvdsx to use gvec primitives


** Bug watch added: gitlab.com/qemu-project/qemu/-/issues #212
   https://gitlab.com/qemu-project/qemu/-/issues/212

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1935617

Title:
  systemd autopkgtest broken on ppc64el with qemu 6.0

Status in The Ubuntu-power-systems project:
  New
Status in qemu package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  New

Bug description:
  I'm not sure yet if this is flaky or a real issue, but I'm filing it
  to avoid multiple people analyzing the same.

  The Qemu 6.0 upload https://launchpad.net/ubuntu/+source/qemu/1:6.0+dfsg-1~ubuntu2 triggers a test failure like
  https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/ppc64el/s/systemd/20210708_223311_e3bbb@/log.gz

  I have tested the new qemu on ppc64 and it worked fine for device emulation and migration cases.
  But this is suspicious.
  Of the last tests exactly and only those with the new qemu failed.

  
  impish
    ppc64el
      tests-in-lxd                   (F  5% f  0% S  0% B  0% => P 95%/) F.........F.............................
      systemd-fsckd                  (F  0% f  0% S 100% B  0% => P  0%/) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
      upstream-1                     (F 15% f  0% S  0% B  0% => P 85%/) F..FFFF....F............................
      upstream-2                     (F 12% f  0% S  0% B  0% => P 87%/) F..FFFF.................................

  
  For an insight in flakyness/reproducibility I've retriggered the missing qemu and the non-qemu cases a few times. If those reproduce all-bad vs all-good again this would further indicate a real issue.

  Unfortunately the ppc maas seems down right now and canonistack also
  isn't too nice this week - overall that inhibits the testing a bit :-/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1935617/+subscriptions