[Bug 2080474] Re: ubuntu installation failing for systems having SAN disk

[Olivier Gayot]

Hello,

I have investigated this problem again today. What seems to be causing
the regression is a change [1] adding support for device activation when
systemd is not present.

More specifically, this hunk [2] makes udev able to activate LVM devices
in the initramfs (systemd does not run in the initramfs in Ubuntu for
now):

TEST!="/run/systemd/system", GOTO="lvm_direct_vgchange"
[...]
LABEL="lvm_direct_vgchange"
ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(LVM_EXEC)/lvm vgchange -aay --autoactivation event $env{LVM_VG_NAME_COMPLETE}"
GOTO="lvm_end"

In the actual root filesystem, multipath and lvm2 compete for taking
ownership of the block devices and there are measures in place (more or
less robust) to give preference to multipath.

OTOH, in the initramfs (of the installer), we currently only have the
LVM stack, not the multipath stack. Therefore, LVM freely takes
ownership of /dev/sda, /dev/sdb (assuming those are the block devices
involved).

The regression was introduced between the 22.04 and 22.10 releases, when
we bumped lvm2 to 2.03.16-1.

As mentioned here:

> Found a possible fix for this bug
> ===========================
> 
> I tried adding multipath module to the installer initrd via initramfs-tools. This had the effect of
> initializing multipath during early initrd and before dm-multipath and lvm were initialized. With this
> change ( also a possible bug fix in initramfs-tools described below ) the dm-multipath disks were
> properly probed and recognized by probert.

having the multipath stack in the initramfs could be a way to address
the issue. The package multipath-tools-boot provides a possible
implementation. However, if stopping multipathd is a problem, I suppose
we could use a different implementation where we only rely on multipath,
not multipathd.

I suspect it might be easier to go back to the behavior we had in 22.04,
where LVM does not run in the initramfs. Maybe we don't entirely drop
the "new" udev rules but disable them when booting the installer, if
that is possible.

[1] https://salsa.debian.org/lvm-team/lvm2/-/commit/2340adad4b3875331be1ba7abba881cc1b6e6738#b3d5381b3f503956bf78fdc525b656292a304bf4
[2] https://paste.ubuntu.com/p/2qHK5xNJzV/

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to subiquity in Ubuntu.
https://bugs.launchpad.net/bugs/2080474

Title:
  ubuntu installation failing for systems having SAN disk

Status in subiquity:
  New
Status in The Ubuntu-power-systems project:
  Incomplete
Status in multipath-tools package in Ubuntu:
  New
Status in subiquity package in Ubuntu:
  New

Bug description:
  == Comment: - Anushree Mathur ==
  OS: 24.04 LTS (Noble Numbat)
  I started the Ubuntu installation for 24.04 LTS (Noble Numbat) on L1(HOST) having SAN disk, it failed with the following error just after I chose the disk.

  Ubuntu 24.04 LTS ubuntu-server hvc0

  
  connecting...  
  waiting for cloud-init...  
  generating crash report
  report saved to /var/crash/1724388235.797082424.ui.crash
  Traceback (most recent call last):
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/client/controllers/filesystem.py", line 273, in _guided_choice
      self.ui.set_body(FilesystemView(self.model, self))
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/ui/views/filesystem/filesystem.py", line 485, in __init__
      self.refresh_model_inputs()
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/ui/views/filesystem/filesystem.py", line 540, in refresh_model_inputs
      self.avail_list.refresh_model_inputs()
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/ui/views/filesystem/filesystem.py", line 417, in refresh_model_inputs
      for obj, cells in summarize_device(device, filter):
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/ui/views/filesystem/helpers.py", line 32, in summarize_device
      anns = labels.annotations(device) + labels.usage_labels(device)
    File "/snap/subiquity/5745/usr/lib/python3.10/functools.py", line 889, in wrapper
      return dispatch(args[0].__class__)(*args, **kw)
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/common/filesystem/labels.py", line 100, in _annotations_vg
      member = next(iter(vg.devices))
  StopIteration

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/snap/subiquity/5745/usr/bin/subiquity", line 8, in <module>
      sys.exit(main())
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/cmd/tui.py", line 158, in main
      asyncio.run(run_with_loop())
    File "/snap/subiquity/5745/usr/lib/python3.10/asyncio/runners.py", line 44, in run
      return loop.run_until_complete(main)
    File "/snap/subiquity/5745/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
      return future.result()
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/cmd/tui.py", line 156, in run_with_loop
      await subiquity_interface.run()
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquity/client/client.py", line 403, in run
      await super().run()
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquitycore/tui.py", line 351, in run
      await super().run()
    File "/snap/subiquity/5745/lib/python3.10/site-packages/subiquitycore/core.py", line 134, in run
      raise exc
  RuntimeError: coroutine raised StopIteration

  Ubuntu 24.04 LTS ubuntu-server hvc0

  
  connecting...  
  ProblemType: Bug
  Architecture: ppc64el
  CrashDB: {'impl': 'launchpad', 'project': 'subiquity'}

  I tried following 2 installation methods, it is failing in both the ways: 
  1) kexec method
  2) attaching vdvd and starting installer

  NOTE: It is happening only when the system has SAN disks otherwise this installation worked fine.
  I will be attaching the crash report for this!

  == Comment:- Hariharan T S ==
  Verified the following cases. 
  Installation on Disk from VIOS  - PASSED
  Installatoin on Disk from VIOS and system had Disks from SAN - FAILED
  Installation on Disk from SAN and system ahd Disk from VIOS - FAILED

  == Comment:- Vaibhav Jain ==
  Problem seems to happening when Subuquity enters the disk partition view. The system has a SAN disk and an existing Mullti Path DM volume on it.

  Mirroring to distro

To manage notifications about this bug go to:
https://bugs.launchpad.net/subiquity/+bug/2080474/+subscriptions