[Bug 2037281] Re: Shutdown when triggering daemon-reload eary in boot

[Valentin David]

** Description changed:

  In Ubuntu Core 20, and Ubuntu Core 22, we are encountering an issue
  where if a service, started earlier than devices are processed by udev,
  does `systemctl daemon-reload`, the system shuts down. This is due to
  devices for mounted filesystem temporarily taken dead, which pulls most
  units down.
  
  This was fixed by upstream in
  https://github.com/systemd/systemd/pull/23218.
  
  But this was not backported to the versions used by Ubuntu packages for
  focal and jammy. The needed commit from that PR is the one with message
  `core/device: ignore DEVICE_FOUND_UDEV bit on switching root`.
  
  This patch applies to 245.4-4ubuntu3.22 (focal) without rebasing needed.
  And I suppose it does also for jammy.
  
  I have manually tested the fix with Ubuntu Core 20, and this fixes our
  issue.
  
  We would like this patch to be backported to focal-updates and jammy-
  updates.
  
  Thank you in advance.
  
+ [ Impact ]
  
- [ Impact ] 
-  * An explanation of the effects of the bug on users and
-  * justification for backporting the fix to the stable release.
-  * In addition, it is helpful, but not required, to include an
-    explanation of how the upload fixes this bug.
+ If a user adds a service that calls `systemctl daemon-reload`, and if
+ this service is started before systemd-udevd. And if the initrd is
+ systemd (the case of Ubuntu Core), then most service will be stopped or
+ cancel, and the machine will mostly shutdown everything and hang.
+ 
+ The fix has been backported down to 250 upstream. It is already on
+ kinetic and later.
+ 
+ The fix only affects systems where systemd is used in initrd.
  
  [ Test Plan ]
-  * detailed instructions how to reproduce the bug
-  * these should allow someone who is not familiar with the affected
-    package to reproduce the bug and verify that the updated package fixes
-    the problem.
-  * if other testing is appropriate to perform before landing this update,
-    this should also be described here.
+ 
+ On Ubuntu Core 20 (with Core 22 kernel) or on Ubuntu Core 22. Or on any
+ system that uses systemd in initrd.
+ 
+ Add a systemd service that calls `systemctl daemon-reload`.
+ The service should have `DefaultDependencies=no` in order to start as soon as possible and be enabled.
+ 
+ Restart the machine.
+ 
+ If fix is not applied, after the service is started, most of units with
+ be shutdown, and the system will be unusable.
  
  [ Where problems could occur ]
-  * Think about what the upload changes in the software. Imagine the change is
-    wrong or breaks something else: how would this show up?
-  * It is assumed that any SRU candidate patch is well-tested before
-    upload and has a low overall risk of regression, but it's important
-    to make the effort to think about what ''could'' happen in the
-    event of a regression.
-  * This must '''never''' be "None" or "Low", or entirely an argument as to why
-    your upload is low risk.
-  * This both shows the SRU team that the risks have been considered,
-    and provides guidance to testers in regression-testing the SRU.
  
- [ Other Info ]
-  * Anything else you think is useful to include
-  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
-  * and address these questions in advance
+ This should affect systems with systemd in initrd.
+ 
+ There are risks on systems that have an udev rule in initrd not present
+ in the main system.
+ 
+ There are risks on systems that use db_persist in initrd where the
+ device can potentially get dead state. Though this does not seem to
+ happen on Ubuntu Core 22, even though we use db_persist for dev mapper
+ devices. Regression is upstream bug #23429. Commits named "core/device:
+ device_coldplug(): don't set DEVICE_DEAD" and "core/device: do not
+ downgrade device state if it is already enumerated" could be applied as
+ well.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2037281

Title:
  Shutdown when triggering daemon-reload eary in boot

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Focal:
  New
Status in systemd source package in Jammy:
  New

Bug description:
  In Ubuntu Core 20, and Ubuntu Core 22, we are encountering an issue
  where if a service, started earlier than devices are processed by
  udev, does `systemctl daemon-reload`, the system shuts down. This is
  due to devices for mounted filesystem temporarily taken dead, which
  pulls most units down.

  This was fixed by upstream in
  https://github.com/systemd/systemd/pull/23218.

  But this was not backported to the versions used by Ubuntu packages
  for focal and jammy. The needed commit from that PR is the one with
  message `core/device: ignore DEVICE_FOUND_UDEV bit on switching root`.

  This patch applies to 245.4-4ubuntu3.22 (focal) without rebasing
  needed. And I suppose it does also for jammy.

  I have manually tested the fix with Ubuntu Core 20, and this fixes our
  issue.

  We would like this patch to be backported to focal-updates and jammy-
  updates.

  Thank you in advance.

  [ Impact ]

  If a user adds a service that calls `systemctl daemon-reload`, and if
  this service is started before systemd-udevd. And if the initrd is
  systemd (the case of Ubuntu Core), then most service will be stopped
  or cancel, and the machine will mostly shutdown everything and hang.

  The fix has been backported down to 250 upstream. It is already on
  kinetic and later.

  The fix only affects systems where systemd is used in initrd.

  [ Test Plan ]

  On Ubuntu Core 20 (with Core 22 kernel) or on Ubuntu Core 22. Or on
  any system that uses systemd in initrd.

  Add a systemd service that calls `systemctl daemon-reload`.
  The service should have `DefaultDependencies=no` in order to start as soon as possible and be enabled.

  Restart the machine.

  If fix is not applied, after the service is started, most of units
  with be shutdown, and the system will be unusable.

  [ Where problems could occur ]

  This should affect systems with systemd in initrd.

  There are risks on systems that have an udev rule in initrd not
  present in the main system.

  There are risks on systems that use db_persist in initrd where the
  device can potentially get dead state. Though this does not seem to
  happen on Ubuntu Core 22, even though we use db_persist for dev mapper
  devices. Regression is upstream bug #23429. Commits named
  "core/device: device_coldplug(): don't set DEVICE_DEAD" and
  "core/device: do not downgrade device state if it is already
  enumerated" could be applied as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2037281/+subscriptions