[Bug 1837227] Re: systemd mount units fail during boot, while file system is correctly mounted

[Heitor Alves de Siqueira]

I've now successfully tested this on Bionic, Focal and Jammy. Using a
test script based on upstream's bug report, several systemd mount units
are created that are tmpfs-backed for better performance. The basic
routine of the script is as follows:

- mount the unit (or target mount point)
- sleep 1s
- verify whether the mount exists and is mounted correctly
  
This is done concurrently with 100 mountpoints, through 40 test loops by default. Additionally, the script covers all variants of this issue that were reported upstream:
1) install all systemd units at once, mount with 'systemctl start' all units
2) install each systemd unit when mounting, mount with 'systemctl start' each unit
3) mount directly with 'systemd-mount'
4) mount directly with 'mount'
5) install each systemd unit when mounting, mount with 'systemctl start',
   trigger multiple 'systemctl daemon-reload' throughout test loop
Each of these scenarios has an equivalent test "variant" in the script that can be invoked directly. Unless specified otherwise, only scenario 4 is tested.

Without the systemd and kernel fixes, at least one of the above
scenarios seems to fail consistently in Bionic and Focal. After
installing test packages from the PPAs [0] and [1], I've been able to
run the test script without any errors for multiple iterations.

[0] https://launchpad.net/~halves/+archive/ubuntu/339757-test-kernel
[1] https://launchpad.net/~halves/+archive/ubuntu/339757-test-systemd

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1837227

Title:
  systemd mount units fail during boot, while file system is correctly
  mounted

Status in systemd:
  New
Status in Ubuntu Pro:
  In Progress
Status in Ubuntu Pro 18.04 series:
  In Progress
Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Bionic:
  Won't Fix
Status in systemd source package in Focal:
  In Progress
Status in systemd source package in Jammy:
  Fix Released

Bug description:
  [Impact]
  systemd mount units fail during boot, and the system boots into emergency mode

  [Test Plan]
  This issue seems to happen randomly, and doesn't seem related to a specific mount unit.

  We've used a test script with good results during investigation to
  reproduce similar mount failures in a running system, and have seen a
  strong correlation between the script failures and the boot time mount
  failures.

  The attached 'rep-tmpfs.sh' script should be used to validate that
  mount points are working correctly under stress. One can run through
  the different variants as below:

  # ./rep-tmpfs.sh --variant-0
  # ./rep-tmpfs.sh --variant-1
  # ./rep-tmpfs.sh --variant-2
  # ./rep-tmpfs.sh --variant-3
  # ./rep-tmpfs.sh --variant-4

  All of these should run successfully without any reported errors.

  [Where problems could occur]
  The patches change the way systemd tracks and handles mount points in general, so potential regressions could affect other mount units. We should keep an eye out for any issues with mounting file systems, as well as rapid mount/unmount operations. Successful test runs with the reproducer script should increase reliability in having no new regressions.

  [Other Info]
  This has been tackled upstream with several attempts, which have resulted in the final patch from 2022:
    01400460ae16 core/mount: adjust deserialized state based on /proc/self/mountinfo

  For Bionic, systemd requires several dependency patches as below:
    6a1d4d9fa6b9 core: properly reset all ExecStatus structures when entering a new unit cycle
    7eba1463dedc mount: flush out cycle state on DEAD→MOUNTED only, not the other way round
    350804867dbc mount: rescan /proc/self/mountinfo before processing waitid() results
    1d086a6e5972 mount: mark an existing "mounting" unit from /proc/self/mountinfo as "just_mounted"

  Additionally, the kernel also requires the following patches:
    28ca0d6d39ab list: introduce list_for_each_continue()
    9f6c61f96f2d proc/mounts: add cursor

  [Original Description]
  In Ubuntu 18.04 at least, we sometimes get a random server in emergency mode with a failed mount unit (ext4 file system), while the corresponding file system is in fact correctly mounted. It happens roughly once every 1000 reboots.

  It seems to be related with this bug :
  https://github.com/systemd/systemd/issues/10872

  Is it possible to apply the fix
  (https://github.com/systemd/systemd/commit/350804867dbcc9b7ccabae1187d730d37e2d8a21)
  in Ubuntu 18.04 ?

  Thanks in advance.

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1837227/+subscriptions