[Bug 1931088] Re: boot-and-services tests fails in impish on armhf (248.3)

Wed Jun 23 16:10:04 UTC 2021

Looks like I can at least reproduce the issue on an arm64 Canonistack
machine, utilizing an armhf container:

ubuntu at focal-arm64:~$ time autopkgtest systemd --test-name=boot-and-services -U -- lxd autopkgtest/ubuntu/impish/armhf
[...]
autopkgtest [15:59:21]: test boot-and-services: [-----------------------
lxc
Created symlink /etc/systemd/system/default.target → /lib/systemd/system/graphical.target.
bash: line 1:  3532 Killed                  /tmp/autopkgtest.OAYOpR/build.yrL/src/debian/tests/boot-and-services 2> >(tee -a /tmp/autopkgtest.OAYOpR/boot-and-services-stderr >&2) > >(tee -a /tmp/autopkgtest.OAYOpR/boot-and-services-stdout)
autopkgtest [15:59:24]: test process requested reboot with marker boot1
Unexpected error:
Traceback (most recent call last):
  File "/usr/share/autopkgtest/lib/VirtSubproc.py", line 738, in mainloop
    command()
  File "/usr/share/autopkgtest/lib/VirtSubproc.py", line 667, in command
    r = f(c, ce)
  File "/usr/share/autopkgtest/lib/VirtSubproc.py", line 363, in cmd_reboot
    caller.hook_wait_reboot()
  File "/usr/bin/autopkgtest-virt-lxd", line 231, in hook_wait_reboot
    wait_booted()
  File "/usr/bin/autopkgtest-virt-lxd", line 104, in wait_booted
    VirtSubproc.check_exec(['lxc', 'exec', container_name, '--', 'sh', '-ec', '[ ! -d /run/systemd/system ] || systemctl start network-online.target'], timeout=60)
  File "/usr/share/autopkgtest/lib/VirtSubproc.py", line 182, in check_exec
    (status, out, err) = execute_timeout(None, timeout, real_argv,
  File "/usr/share/autopkgtest/lib/VirtSubproc.py", line 144, in execute_timeout
    (out, err) = sp.communicate(instr)
  File "/usr/lib/python3.8/subprocess.py", line 1014, in communicate
    stderr = self.stderr.read()
  File "/usr/share/autopkgtest/lib/VirtSubproc.py", line 64, in alarm_handler
    raise Timeout()
VirtSubproc.Timeout
autopkgtest [16:01:18]: ERROR: testbed failure: unexpected eof from the testbed

real	8m29.354s
user	0m8.866s
sys	0m8.141s

Let's see what we can do about it. It looks like a dnsmasq (security-)update was pushed to Bionic++ on 2021-06-02 – about the same time that this test started failing on Bionic++ ... I wonder if this might somehow be related? ... need to further investigate.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to isc-dhcp in Ubuntu.
https://bugs.launchpad.net/bugs/1931088

Title:
  boot-and-services tests fails in impish on armhf (248.3)

Status in Auto Package Testing:
  Incomplete
Status in dnsmasq package in Ubuntu:
  Incomplete
Status in gdm3 package in Ubuntu:
  Incomplete
Status in glibc package in Ubuntu:
  Incomplete
Status in isc-dhcp package in Ubuntu:
  Incomplete
Status in systemd package in Ubuntu:
  Confirmed
Status in dnsmasq source package in Impish:
  Incomplete
Status in gdm3 source package in Impish:
  Incomplete
Status in glibc source package in Impish:
  Incomplete
Status in isc-dhcp source package in Impish:
  Incomplete
Status in systemd source package in Impish:
  Confirmed

Bug description:
  Systemd 248.3-1ubuntu1 is rather new, but had 5 successful tests on armhf
  before now slipping into a bad mode.

  Now it seems all tests failed in boot-and-services by hanging until killed by
  VirtSubproc.Timeout of autokgtest.

  The last [1] test log has a bit more, it shows a python stack overflow
  ```
     VirtSubproc.Timeout
     Fatal Python error: Cannot recover from stack overflow.
     Python runtime state: initialized

     Current thread 0x00007f108e840740 (most recent call first):
       File "/home/ubuntu/autopkgtest/lib/adtlog.py", line 36 in log
       File "/home/ubuntu/autopkgtest/lib/adtlog.py", line 86 in debug
       File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 472 in send
       File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 521 in command
  ```

  I have seen a bunch of packages including even gdm3 and glibc being blocked by
  that so I wanted to at least track down the issue until we can put it on
  someones task list to resolve.
  No one replied to my pings yet, but maybe that means someone is already
  debugging this and had enabled some debugging?

  By running the same in armhf container on arm64 VM on canonistack I've seen
  no issues. The test worked fine and had no hang/issues.

  root at systemd-test-fail:~/systemd-248.3# ./debian/tests/boot-and-services
  lxc
  1
  test_profile (__main__.AppArmorTest)
  AppArmor confined unit ... skipped 'fails on armhf testbeds, see LP: #1842352'
  test_help (__main__.CLITest)
  --help works and succeeds ... ok
  test_invalid_option (__main__.CLITest)
  Calling with invalid option fails ... ok
  test_version (__main__.CLITest)
  --version works and succeeds ... ok
  test_cpushares (__main__.CgroupsTest)
  service with CPUShares ... ok
  test_simple (__main__.CgroupsTest)
  simple service ... ok
  test_bash_crash (__main__.CoredumpTest) ... skipped 'systemd-coredump does not work in containers'
  test_log_for_service (__main__.JournalTest) ... ok
  test_no_options (__main__.JournalTest) ... ok
  test_boot (__main__.NspawnTest) ... skipped 'nspawn does not work in most containers'
  test_service (__main__.NspawnTest) ... skipped 'nspawn does not work in most containers'
  test_failing (__main__.SeccompTest) ... ok
  test_0_init (__main__.ServicesTest)
  Verify that init is systemd ... ok
  test_cron (__main__.ServicesTest) ... ok
  test_dbus (__main__.ServicesTest) ... ok
  test_gdm3 (__main__.ServicesTest) ... skipped 'gdm3 not found'
  test_logind (__main__.ServicesTest) ... ok
  test_network_manager (__main__.ServicesTest) ... ok
  test_no_failed (__main__.ServicesTest)
  No failed units ... ok
  test_rsyslog (__main__.ServicesTest) ... ok
  test_tmp_cleanup (__main__.ServicesTest) ... ok
  test_tmp_mount (__main__.ServicesTest) ... ok
  test_udev (__main__.ServicesTest) ... skipped 'udev does not work in containers'
  ----------------------------------------------------------------------
  Ran 23 tests in 5.589s
  OK (skipped=6)

  So I can not reproduce this on canonistack, but it blocks autopkgtests of
  various packages pretty reproducibly :-/

  [1]: https://autopkgtest.ubuntu.com/results/autopkgtest-
  impish/impish/armhf/s/systemd/20210604_081326_d4319@/log.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1931088/+subscriptions