[Bug 1824227] [NEW] console-setup failure due to race with systemd-tmpfiles

Robert C Jennings 1824227 at bugs.launchpad.net
Wed Apr 10 19:38:14 UTC 2019 

Public bug reported:

I'm seeing a console-setup.service failure quite regularly in testing
where the temp file that should have been created can't be found.
This is a regular xenial cloud image.

  19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
  ...
  19:51:13 systemd[1]: Starting Set console font and keymap...
  19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file   
  19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE
  19:51:15 systemd[1]: Failed to start Set console font and keymap.
  19:51:15 systemd[1]: console-setup.service: Unit entered failed state.
  19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'.
        ...

/bin/setupcon has a lovely tempfile function that looks like:
  if \
    TMPFILE=`mktemp /tmp/tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp /run/tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp /dev/.tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp 2>/dev/null`

Here's our edited IRC conversation on the bug:
<@vorlon> so I do think you're being hit by the tmp cleaner
<@vorlon> also this seems like bad pathological default behavior for
          the tmp cleaner, to delete files that have just been created
<@vorlon> but we should fix console-setup to not rely on /tmp
<@vorlon> and I prefer that we do that instead of trying to fiddle with
          the ordering of the systemd units on startup
<@vorlon> i.e. console-setup has an undeclared dependency
          on systemd-tmpfiles-clean; let's remove the dependency
          instead of declaring it

<@vorlon> are you failing the race more often now than in the past?
<@rcj>    vorlon: it feels like it's failing more often but I don't have
          data to answer that.

<@vorlon> are we shipping an image with a dirty rootfs?
<@vorlon> dirty in the sense that e2fsck doesn't take one look at it,
          say "yep, nothing to do here" and exit
<@vorlon> in the sense that this is what would make dev-sda1.device slow
          to complete AIUI
<@rcj>    would filesystem resize on first boot mark it dirty?  Because
          that will happen
<@vorlon> huh good question
<@vorlon> it might
<xnox>    rcj, unclean shutdown?
<@rcj>    xnox: first boot

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: console-setup 1.108ubuntu15.4
ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18
Uname: Linux 4.15.0-1029-gcp x86_64
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
Date: Wed Apr 10 19:24:12 2019
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: console-setup
UpgradeStatus: No upgrade log present (probably fresh install)

** Affects: console-setup (Ubuntu)
     Importance: High
         Status: Triaged

** Affects: console-setup (Ubuntu Bionic)
     Importance: High
         Status: Triaged

** Affects: console-setup (Ubuntu Cosmic)
     Importance: High
         Status: Triaged

** Affects: console-setup (Ubuntu Disco)
     Importance: High
         Status: Triaged


** Tags: amd64 apport-bug uec-images xenial

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to console-setup in Ubuntu.
https://bugs.launchpad.net/bugs/1824227

Title:
  console-setup failure due to race with systemd-tmpfiles

Status in console-setup package in Ubuntu:
  Triaged
Status in console-setup source package in Bionic:
  Triaged
Status in console-setup source package in Cosmic:
  Triaged
Status in console-setup source package in Disco:
  Triaged

Bug description:
  I'm seeing a console-setup.service failure quite regularly in testing
  where the temp file that should have been created can't be found.
  This is a regular xenial cloud image.

    19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
    ...
    19:51:13 systemd[1]: Starting Set console font and keymap...
    19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file   
    19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE
    19:51:15 systemd[1]: Failed to start Set console font and keymap.
    19:51:15 systemd[1]: console-setup.service: Unit entered failed state.
    19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'.
          ...

  /bin/setupcon has a lovely tempfile function that looks like:
    if \
      TMPFILE=`mktemp /tmp/tmpkbd.XXXXXX 2>/dev/null` \
        || TMPFILE=`mktemp /run/tmpkbd.XXXXXX 2>/dev/null` \
        || TMPFILE=`mktemp /dev/.tmpkbd.XXXXXX 2>/dev/null` \
        || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XXXXXX 2>/dev/null` \
        || TMPFILE=`mktemp 2>/dev/null`

  Here's our edited IRC conversation on the bug:
  <@vorlon> so I do think you're being hit by the tmp cleaner
  <@vorlon> also this seems like bad pathological default behavior for
            the tmp cleaner, to delete files that have just been created
  <@vorlon> but we should fix console-setup to not rely on /tmp
  <@vorlon> and I prefer that we do that instead of trying to fiddle with
            the ordering of the systemd units on startup
  <@vorlon> i.e. console-setup has an undeclared dependency
            on systemd-tmpfiles-clean; let's remove the dependency
            instead of declaring it

  <@vorlon> are you failing the race more often now than in the past?
  <@rcj>    vorlon: it feels like it's failing more often but I don't have
            data to answer that.

  <@vorlon> are we shipping an image with a dirty rootfs?
  <@vorlon> dirty in the sense that e2fsck doesn't take one look at it,
            say "yep, nothing to do here" and exit
  <@vorlon> in the sense that this is what would make dev-sda1.device slow
            to complete AIUI
  <@rcj>    would filesystem resize on first boot mark it dirty?  Because
            that will happen
  <@vorlon> huh good question
  <@vorlon> it might
  <xnox>    rcj, unclean shutdown?
  <@rcj>    xnox: first boot

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: console-setup 1.108ubuntu15.4
  ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18
  Uname: Linux 4.15.0-1029-gcp x86_64
  ApportVersion: 2.20.1-0ubuntu2.18
  Architecture: amd64
  Date: Wed Apr 10 19:24:12 2019
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm-256color
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: console-setup
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+subscriptions

More information about the foundations-bugs mailing list