[Bug 2037417] Re: mantic arm64 images are failing to deploy with failure to mount root and kernel filesystems

[Francis Ginther]

I have done some debugging, but I'm at a loss for what to do next.

Booting with 'apparmor=0' did not help.

Here's what the mounts look like:

# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=262851016k,nr_inodes=65712754,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=52585616k,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/root.tmp.img (deleted) on /media/root-ro type squashfs (ro,relatime,errors=continue,threads=single)
tmpfs-root on /media/root-rw type tmpfs (rw,relatime,inode64)
overlayroot on / type overlay (ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_,xino=off,nouserxattr)
copymods on /usr/lib/modules type tmpfs (rw,relatime,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /etc/machine-id type tmpfs (ro,relatime,size=52585616k,mode=755,inode64)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)

I can boot into single user mode and interact with the console to
perform further debugging.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2037417

Title:
  mantic arm64 images are failing to deploy with failure to mount root
  and kernel filesystems

Status in maas-images:
  New
Status in systemd package in Ubuntu:
  New

Bug description:
  Mantic arm64 deploys started failing on Sept 18th with:

  [   41.913552] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
           Starting systemd-remount-f鈥t Root and Kernel File Systems...
  [   41.940748] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
           Starting systemd-udev-trig鈥0m - Coldplug All udev Devices...
  [   41.964758] systemd[1]: Started systemd-journald.service - Journal Service.
  [  OK  ] Started systemd-journald.service - Journal Service.
  [  OK  ] Mounted dev-hugepages.mount - Huge Pages File System.
  [  OK  ] Mounted dev-mqueue.mount[鈥�- POSIX Message Queue File System.
  [  OK  ] Mounted sys-kernel-debug.m鈥t - Kernel Debug File System.
  [  OK  ] Mounted sys-kernel-tracing鈥t - Kernel Trace File System.
  [  OK  ] Finished keyboard-setup.se鈥�- Set the console keyboard layout.
  [  OK  ] Finished kmod-static-nodes鈥eate List of Static Device Nodes.
  [  OK  ] Finished lvm2-monitor.serv鈥ing dmeventd or progress polling.
  [  OK  ] Finished modprobe at configfs鈥0m - Load Kernel Module configfs.
  [  OK  ] Finished modprobe at dm_mod.s鈥 - Load Kernel Module dm_mod.
  [  OK  ] Finished modprobe at drm.service - Load Kernel Module drm.
  [  OK  ] Finished modprobe at efi_psto鈥 - Load Kernel Module efi_pstore.
  [  OK  ] Finished modprobe at fuse.service - Load Kernel Module fuse.
  [  OK  ] Finished modprobe at loop.service - Load Kernel Module loop.
  [  OK  ] Finished systemd-modules-l鈥ervice - Load Kernel Modules.
  [FAILED] Failed to start systemd-re鈥unt Root and Kernel File Systems.
  See 'systemctl status systemd-remount-fs.service' for details.

  After this many other services and cloud-init fails. See the full
  kopter-0918.log. For comparison, a log from the prior day's test is
  also attached.

To manage notifications about this bug go to:
https://bugs.launchpad.net/maas-images/+bug/2037417/+subscriptions