[Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start

[Scott Moser]

I agree with reverting change in bionic, and that xenial still needs
some fix.

I recreated your failure in xenial and agree with your change there.
I used this job as it actually tries to write to the private tmp.

If I change 'ExecStart' below to be  just '/bin/true' like in your job,
it does not fail (status says 'code=exited, status=0/SUCCESS).  But the
command below seems to never get exited (it produces no output).

So it seems to me that the 'Read only filesystem' error is not even from
the service itself, but rather from systemd, and also dependent on the
content in ExecStart.

$ cat /lib/systemd/system/my-ptmp-test.service
[Unit]
Description=foo
DefaultDependencies=no
[Service]
Type=oneshot
PrivateTmp=yes
ExecStart=/bin/sh -ec 'f=/tmp/my.txt; uptime > $f; echo ==; cat $f; echo ==; cat /proc/mounts'
[Install]
WantedBy=multi-user.target


Then we can see the same failure:

$ journalctl -o short-monotonic --no-pager | grep --context=2 my-ptmp-test
[    7.715220] xenial-20180223-142555 systemd[1]: Created slice System Slice.
[    7.718923] xenial-20180223-142555 systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    7.740250] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Failed to run 'start' task: Read-only file system
[    7.749135] xenial-20180223-142555 systemd[1]: Failed to start foo.
[    7.754114] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Unit entered failed state.
[    7.760217] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Failed with result 'resources'.
[    7.764170] xenial-20180223-142555 systemd[1]: Starting Load Kernel Modules...
[    7.775558] xenial-20180223-142555 systemd[1]: Starting Journal Service...

$ systemctl status my-ptmp-test.service --no-pager --full
● my-ptmp-test.service - foo
   Loaded: loaded (/lib/systemd/system/my-ptmp-test.service; enabled; vendor preset: enabled)
   Active: failed (Result: resources)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1750780

Title:
  Race with local file systems can make open-vm-tools fail to start

Status in cloud-init:
  Invalid
Status in open-vm-tools package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in open-vm-tools source package in Xenial:
  Triaged
Status in systemd source package in Xenial:
  New
Status in open-vm-tools package in Debian:
  Incomplete

Bug description:
  Since the change in [1] open-vm-tools-service starts very (very) early.
  Not so much due to the 
  Before=cloud-init-local.service
  But much more by
  DefaultDependencies=no

  That can trigger an issue that looks like
  root at ubuntuguest:~# systemctl status -l open-vm-tools.service
  ● open-vm-tools.service - Service for virtual machines hosted on VMware
     Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled)
     Active: failed (Result: resources)

  
  As it is right now open-vm-tools can race with the other early start and then fail.
  In detail one can find a message like:
    open-vm-tools.service: Failed to run 'start' task: Read-only file system"

  This is due to privtaeTmp=yes which is also set needing a writable
  /var/tmp [2]

  To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably.
  I added
  After=local-fs.target
  which made it work for me in 3/3 tests.

  I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service
  I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data.

  To summarize:
  # cloud-init-local #
  DefaultDependencies=no
  Wants=network-pre.target
  After=systemd-remount-fs.service
  Before=NetworkManager.service
  Before=network-pre.target
  Before=shutdown.target
  Before=sysinit.target
  Conflicts=shutdown.target
  RequiresMountsFor=/var/lib/cloud

  # open-vm-tools #
  DefaultDependencies=no
  Before=cloud-init-local.service

  Proposed is to add to the latter:
  After=local-fs.target

  [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677
  [2]: https://github.com/systemd/systemd/issues/5610

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions