[Bug 1837227] Re: Random mount units sometimes fail, while file system is correctly mounted

Thu Jun 29 19:09:53 UTC 2023

I'm marking Bionic as Won't Fix, as this is no longer under standard
support. The fixes for this bug will be made available under Ubuntu Pro
for 18.04.

** Also affects: systemd (Ubuntu Jammy)
   Importance: Undecided
       Status: New

** Also affects: systemd (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Changed in: systemd (Ubuntu Focal)
       Status: New => In Progress

** Changed in: systemd (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: systemd (Ubuntu Focal)
     Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: systemd (Ubuntu Jammy)
     Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: systemd (Ubuntu Jammy)
       Status: New => In Progress

** Tags removed: ddstreet
** Tags added: se-sponsor-halves

** Summary changed:

- Random mount units sometimes fail, while file system is correctly mounted
+ Mount units sometimes fail, while file system is correctly mounted

** Summary changed:

- Mount units sometimes fail, while file system is correctly mounted
+ systemd mount units fail during boot, while file system is correctly mounted

** Also affects: systemd (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Changed in: systemd (Ubuntu Bionic)
     Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: systemd (Ubuntu Bionic)
       Status: New => Won't Fix

** Description changed:

- In Ubuntu 18.04 at least, we sometimes get a random server in emergency
- mode with a failed mount unit (ext4 file system), while the
- corresponding file system is in fact correctly mounted. It happens
- roughly once every 1000 reboots.
+ [Impact]
+ systemd mount units fail during boot, and the system boots into emergency mode
+ 
+ [Test Plan]
+ This issue seems to happen randomly, and doesn't seem related to a specific mount unit.
+ 
+ We've used a test script with good results during investigation to
+ reproduce similar mount failures in a running system, and have seen a
+ strong correlation between the script failures and the boot time mount
+ failures.
+ 
+ [Where problems could occur]
+ The patches change the way systemd tracks and handles mount points in general, so potential regressions could affect other mount units. We should keep an eye out for any issues with mounting file systems, as well as rapid mount/unmount operations. Successful test runs with the reproducer script should increase reliability in having no new regressions.
+ 
+ [Other Info]
+ This has been tackled upstream with several attempts, which have resulted in the final patch from 2022:
+   01400460ae16 core/mount: adjust deserialized state based on /proc/self/mountinfo 
+ 
+ For Bionic, systemd requires several dependency patches as below:
+   6a1d4d9fa6b9 core: properly reset all ExecStatus structures when entering a new unit cycle
+   7eba1463dedc mount: flush out cycle state on DEAD→MOUNTED only, not the other way round
+   350804867dbc mount: rescan /proc/self/mountinfo before processing waitid() results
+   1d086a6e5972 mount: mark an existing "mounting" unit from /proc/self/mountinfo as "just_mounted"
+ 
+ Additionally, the kernel also requires the following patches:
+   28ca0d6d39ab list: introduce list_for_each_continue()
+   9f6c61f96f2d proc/mounts: add cursor
+ 
+ [Original Description]
+ In Ubuntu 18.04 at least, we sometimes get a random server in emergency mode with a failed mount unit (ext4 file system), while the corresponding file system is in fact correctly mounted. It happens roughly once every 1000 reboots.

  It seems to be related with this bug :
  https://github.com/systemd/systemd/issues/10872

  Is it possible to apply the fix
  (https://github.com/systemd/systemd/commit/350804867dbcc9b7ccabae1187d730d37e2d8a21)
  in Ubuntu 18.04 ?

  Thanks in advance.

** Also affects: ubuntu-pro
   Importance: Undecided
       Status: New

** Also affects: ubuntu-pro/18.04
   Importance: Undecided
       Status: New

** Changed in: ubuntu-pro/18.04
       Status: New => In Progress

** Changed in: ubuntu-pro/18.04
   Importance: Undecided => High

** Changed in: ubuntu-pro/18.04
     Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1837227

Title:
  systemd mount units fail during boot, while file system is correctly
  mounted

Status in systemd:
  New
Status in Ubuntu Pro:
  In Progress
Status in Ubuntu Pro 18.04 series:
  In Progress
Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Bionic:
  Won't Fix
Status in systemd source package in Focal:
  In Progress
Status in systemd source package in Jammy:
  Fix Released

Bug description:
  [Impact]
  systemd mount units fail during boot, and the system boots into emergency mode

  [Test Plan]
  This issue seems to happen randomly, and doesn't seem related to a specific mount unit.

  We've used a test script with good results during investigation to
  reproduce similar mount failures in a running system, and have seen a
  strong correlation between the script failures and the boot time mount
  failures.

  [Where problems could occur]
  The patches change the way systemd tracks and handles mount points in general, so potential regressions could affect other mount units. We should keep an eye out for any issues with mounting file systems, as well as rapid mount/unmount operations. Successful test runs with the reproducer script should increase reliability in having no new regressions.

  [Other Info]
  This has been tackled upstream with several attempts, which have resulted in the final patch from 2022:
    01400460ae16 core/mount: adjust deserialized state based on /proc/self/mountinfo 

  For Bionic, systemd requires several dependency patches as below:
    6a1d4d9fa6b9 core: properly reset all ExecStatus structures when entering a new unit cycle
    7eba1463dedc mount: flush out cycle state on DEAD→MOUNTED only, not the other way round
    350804867dbc mount: rescan /proc/self/mountinfo before processing waitid() results
    1d086a6e5972 mount: mark an existing "mounting" unit from /proc/self/mountinfo as "just_mounted"

  Additionally, the kernel also requires the following patches:
    28ca0d6d39ab list: introduce list_for_each_continue()
    9f6c61f96f2d proc/mounts: add cursor

  [Original Description]
  In Ubuntu 18.04 at least, we sometimes get a random server in emergency mode with a failed mount unit (ext4 file system), while the corresponding file system is in fact correctly mounted. It happens roughly once every 1000 reboots.

  It seems to be related with this bug :
  https://github.com/systemd/systemd/issues/10872

  Is it possible to apply the fix
  (https://github.com/systemd/systemd/commit/350804867dbcc9b7ccabae1187d730d37e2d8a21)
  in Ubuntu 18.04 ?

  Thanks in advance.

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1837227/+subscriptions