[Bug 1935617] Re: systemd autopkgtest broken on ppc64el with qemu 6.0

Mon Jul 12 12:04:16 UTC 2021

As root:

$ mkdir /tmp/debugtest
$ export ARTIFACT_DIRECTORY=/tmp/debugtest
$ export TEST_SHOW_JOURNAL=warning
$ export TEST_REQUIRE_INSTALL_TESTS=0
$ export TEST_PREFER_NSPAWN=1
$ export NO_BUILD=1
$ export SELECTED_TESTS="TEST-08-*"
./test/run-integration-tests.sh

Reproduces the issue, it will keep the journal in /tmp/debugtest and run
in emulation.

The crash looks like:
[   15.988730] kernel: ext4 filesystem being mounted at /root supports timestamps until 2038 (0x7fffffff)
[   18.235447] systemd[1]: Failed to find module 'autofs4'
[   21.938779] kernel: ext4 filesystem being remounted at / supports timestamps until 2038 (0x7fffffff)
[   25.404984] systemd-coredump[371]: [🡕] Process 369 (rm) of user 0 dumped core.

                                      Stack trace of thread 369:
                                      #0  0x00000b1dcc8c61d0 n/a (rm + 0x61d0)
                                      #1  0x00000b1dcc8c8f64 n/a (rm + 0x8f64)
                                      #2  0x00000b1dcc8c3538 n/a (rm + 0x3538)
                                      #3  0x00000b1dcc8c4424 n/a (rm + 0x4424)
                                      #4  0x00000b1dcc8c2b70 n/a (rm + 0x2b70)
                                      #5  0x00007ee0760aa0b4 n/a (libc.so.6 + 0x2a0b4)
                                      #6  0x00007ee0760aa2a0 __libc_start_main (libc.so.6 + 0x2a2a0)
[   25.480260] systemd[1]: testsuite-08.service: Control process exited, code=dumped, status=11/SEGV

It seems always "rm" that breaks, here from a fill log:

$ grep -e "of user 0 dumped core" systemd-ppc-qemu6.0.txt
[   47.691335] systemd-coredump[286]: Process 279 (rm) of user 0 dumped core.
[   44.911447] systemd-coredump[284]: Process 279 (rm) of user 0 dumped core.
[   45.089287] systemd-coredump[284]: Process 282 (rm) of user 0 dumped core.
[   44.799519] systemd-coredump[284]: Process 278 (rm) of user 0 dumped core.
[   44.666997] systemd-coredump[285]: Process 283 (rm) of user 0 dumped core.
[   46.306449] systemd-coredump[282]: Process 279 (rm) of user 0 dumped core.
[   43.898582] systemd-coredump[283]: Process 280 (rm) of user 0 dumped core.
[   53.298322] systemd-coredump[338]: Process 335 (rm) of user 0 dumped core.
[   45.731700] systemd-coredump[284]: Process 282 (rm) of user 0 dumped core.
[   45.354492] systemd-coredump[283]: Process 281 (rm) of user 0 dumped core.
[   43.579587] systemd-coredump[281]: Process 278 (rm) of user 0 dumped core.
[   45.047475] systemd-coredump[284]: Process 278 (rm) of user 0 dumped core.
[   50.662839] systemd-coredump[281]: Process 278 (rm) of user 0 dumped core.
[   45.634806] systemd-coredump[283]: Process 278 (rm) of user 0 dumped core.

It is interesting to see such a crash as all other services report a
valid green "ok".

If I start the very same guest, but without the pre-defined
systemd.wants=end.service (as it is meant to shut it down again) I can
log in as root without a PW.

$ /bin/qemu-system-ppc64 -smp 4 -net none -m 512M -nographic -vga none -kernel /boot/vmlinux-5.11.0-22-generic -drive format=raw,cache=unsafe,file=/var/tmp/systemd-test.zx7kOu/test08.img -initrd /boot/initrd.img-5.11.0-22-generic -append ' root=/dev/sda1 rw raid=noautodetect rd.luks=0 loglevel=2 init=/lib/systemd/systemd console=hvc0 selinux=0  SYSTEMD_UNIT_PATH=/usr/lib/systemd/tests/testdata/testsuite-08.units:/usr/lib/systemd/tests/testdata/units: systemd.unit=testsuite.target systemd.wants=testsuite-08.service

In this boot mode the console is far from perfect, but I can find
testsuite-08.service to be failed on start with that SEGFAULT ... trying
to isolate the failing command.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1935617

Title:
  systemd autopkgtest broken on ppc64el with qemu 6.0

Status in qemu package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  New

Bug description:
  I'm not sure yet if this is flaky or a real issue, but I'm filing it
  to avoid multiple people analyzing the same.

  The Qemu 6.0 upload https://launchpad.net/ubuntu/+source/qemu/1:6.0+dfsg-1~ubuntu2 triggers a test failure like
  https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/ppc64el/s/systemd/20210708_223311_e3bbb@/log.gz

  I have tested the new qemu on ppc64 and it worked fine for device emulation and migration cases.
  But this is suspicious.
  Of the last tests exactly and only those with the new qemu failed.

  impish
    ppc64el
      tests-in-lxd                   (F  5% f  0% S  0% B  0% => P 95%/) F.........F.............................
      systemd-fsckd                  (F  0% f  0% S 100% B  0% => P  0%/) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
      upstream-1                     (F 15% f  0% S  0% B  0% => P 85%/) F..FFFF....F............................
      upstream-2                     (F 12% f  0% S  0% B  0% => P 87%/) F..FFFF.................................

  For an insight in flakyness/reproducibility I've retriggered the missing qemu and the non-qemu cases a few times. If those reproduce all-bad vs all-good again this would further indicate a real issue.

  Unfortunately the ppc maas seems down right now and canonistack also
  isn't too nice this week - overall that inhibits the testing a bit :-/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1935617/+subscriptions