[Bug 2101888] Re: fsck.xfs can drop into a rescue shell when fsck.mode=force is set

[Andreas Hasenack]

Does anybody have a fix in mind, or are we better off with an immediate
revert?

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to xfsprogs in Ubuntu.
https://bugs.launchpad.net/bugs/2101888

Title:
  fsck.xfs can drop into a rescue shell when fsck.mode=force is set

Status in xfsprogs package in Ubuntu:
  Confirmed

Bug description:
  Description:	Ubuntu 22.04 LTS
  Release:	22.04
  Package: xfsprogs 5.13.0-1ubuntu2.1
  Kernel cmdline containing: fsck.mode=force fsck.repair=yes

  [Issue]

  LP #2081163
  (https://bugs.launchpad.net/ubuntu/+source/xfsprogs/+bug/2081163)
  backported a fix for fsck.xfs not being run during system startup:

      commit 19dde7fac0f38af2990e367ef4dd8ec512920c12
      Author: Gerald Yang <gerald.yang at canonical.com>
      Date:   Tue Aug 13 15:25:51 2024 +0800

          fsck.xfs: fix fsck.xfs run by different shells when fsck.mode=force is set
          
          When fsck.mode=force is specified in the kernel command line, fsck.xfs
          is executed during the boot process. However, when the default shell is
          not bash, $PS1 should be a different value, consider the following script:
          cat ps1.sh
          echo "$PS1"
          
          run ps1.sh with different shells:
          ash ./ps1.sh
          $
          bash ./ps1.sh
          
          dash ./ps1.sh
          $
          ksh ./ps1.sh
          
          zsh ./ps1.sh
          
          On systems like Ubuntu, where dash is the default shell during the boot
          process to improve startup speed. This results in FORCE being incorrectly
          set to false and then xfs_repair is not invoked:
          if [ -n "$PS1" -o -t 0 ]; then
                  FORCE=false
          fi
          
          Other distros may encounter this issue too if the default shell is set
          to anoother shell.
          
          Check "-t 0" is enough to determine if we are in interactive mode, and
          xfs_repair is invoked as expected regardless of the shell used.
          
          Fixes: 04a2d5dc ("fsck.xfs: allow forced repairs using xfs_repair")
          Signed-off-by: Gerald Yang <gerald.yang at canonical.com>
          Reviewed-by: Darrick J. Wong <djwong at kernel.org>

  
  That fix was originally applied upstream for 6.11.0, but it appears to implicitly depend on an earlier commit from xfsprogs 6.1.0:

      commit 79ba1e15d80eba3aff4396f44629eb8960722d36
      Author: Srikanth C S <srikanth.c.s at oracle.com>
      Date:   Tue Dec 13 22:45:43 2022 +0530

          fsck.xfs: mount/umount xfs fs to replay log before running
  xfs_repair

          After a recent data center crash, we had to recover root filesystems
          on several thousands of VMs via a boot time fsck. Since these
          machines are remotely manageable, support can inject the kernel
          command line with 'fsck.mode=force fsck.repair=yes' to kick off
          xfs_repair if the machine won't come up or if they suspect there
          might be deeper issues with latent errors in the fs metadata, which
          is what they did to try to get everyone running ASAP while
          anticipating any future problems. But, fsck.xfs does not address the
          journal replay in case of a crash.

          fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is
          possible that when the machine crashes, the fs is in inconsistent
          state with the journal log not yet replayed. This can drop the machine
          into the rescue shell because xfs_fsck.sh does not know how to clean the
          log. Since the administrator told us to force repairs, address the
          deficiency by cleaning the log and rerunning xfs_repair.

          Run xfs_repair -e when fsck.mode=force and repair=auto or yes.
          Replay the logs only if fsck.mode=force and fsck.repair=yes. For
          other option -fa and -f drop to the rescue shell if repair detects
          any corruptions.

          Signed-off-by: Srikanth C S <srikanth.c.s at oracle.com>
          Reviewed-by: Carlos Maiolino <cmaiolino at redhat.com>
          Signed-off-by: Carlos Maiolino <cem at kernel.org>

  Now that xfs_repair is actually invoked when fsck.mode=force is set,
  it's possible for xfs_repair to fail at boot time and drop into a
  rescue shell.

  This behaviour is seen with xfsprogs 5.13.0-1ubuntu2.1 in Jammy. It
  would presumably also affect 5.3.0-1ubuntu2.1 in Focal, but I haven't
  tried it personally.

  
  [Potential Solution]

  I applied 79ba1e15d80eba3aff4396f44629eb8960722d36 to xfs_fsck.sh in
  5.13.0-1ubuntu2.1 and purposely corrupted the fs.

  Booting with fsck.mode=force fsck.repair=yes, fsck.xfs correctly
  mounts, replays the log, unmounts, repairs, and mounts the filesystem
  at boot.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/xfsprogs/+bug/2101888/+subscriptions