[Bug 1959047] Re: systemd ignores RootDirectory option in .service units

Tue Mar 29 23:51:31 UTC 2022

Bionic verification was successfully done using the steps outlined in
the bug description. The important parts are captured here:

$ lxc exec lp1959047 -- apt-get install -y lxd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  acl apparmor dns-root-data dnsmasq-base ebtables iptables libip6tc0 libiptc0 liblxc-common liblxc1 liblzo2-2 libnetfilter-conntrack3 libnfnetlink0 libuv1 lxcfs lxd-client rsync squashfs-tools uidmap xdelta3
Suggested packages:
  apparmor-profiles-extra apparmor-utils criu lxd-tools openssh-server
The following NEW packages will be installed:
  acl apparmor dns-root-data dnsmasq-base ebtables iptables libip6tc0 libiptc0 liblxc-common liblxc1 liblzo2-2 libnetfilter-conntrack3 libnfnetlink0 libuv1 lxcfs lxd lxd-client rsync squashfs-tools uidmap xdelta3
0 upgraded, 21 newly installed, 0 to remove and 9 not upgraded.
Need to get 10.9 MB of archives.
After this operation, 41.3 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libnfnetlink0 amd64 1.0.1-3 [13.3 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 liblzo2-2 amd64 2.08-1.2 [48.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apparmor amd64 2.12-4ubuntu5.1 [487 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 libip6tc0 amd64 1.6.1-2ubuntu2 [19.9 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 libiptc0 amd64 1.6.1-2ubuntu2 [9308 B]
Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libnetfilter-conntrack3 amd64 1.0.6-2 [37.8 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 iptables amd64 1.6.1-2ubuntu2 [269 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 rsync amd64 3.1.2-2.1ubuntu1.3 [335 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 acl amd64 2.2.52-3build1 [38.5 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 dns-root-data all 2018013001 [5360 B]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 dnsmasq-base amd64 2.79-1ubuntu0.5 [307 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ebtables amd64 2.0.10.4-3.5ubuntu2.18.04.3 [79.9 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 liblxc1 amd64 3.0.3-0ubuntu1~18.04.1 [264 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 liblxc-common amd64 3.0.3-0ubuntu1~18.04.1 [438 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libuv1 amd64 1.18.0-3 [64.4 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 lxcfs amd64 3.0.3-0ubuntu1~18.04.2 [39.0 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 lxd-client amd64 3.0.3-0ubuntu1~18.04.2 [3025 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 squashfs-tools amd64 1:4.3-6ubuntu0.18.04.4 [111 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 uidmap amd64 1:4.5-1ubuntu2.3 [68.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 xdelta3 amd64 3.0.11-dfsg-1ubuntu1 [68.9 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 lxd amd64 3.0.3-0ubuntu1~18.04.2 [5199 kB]
Fetched 10.9 MB in 4s (2980 kB/s)
...
Unpacking xdelta3 (3.0.11-dfsg-1ubuntu1) ...
Selecting previously unselected package lxd.
Preparing to unpack .../20-lxd_3.0.3-0ubuntu1~18.04.2_amd64.deb ...
Adding system user `lxd' (UID 105) ...
Adding new user `lxd' (UID 105) with group `nogroup' ...
Creating home directory `/var/lib/lxd/' ...
Adding group `lxd' (GID 109) ...
Done.
Unpacking lxd (3.0.3-0ubuntu1~18.04.2) ...
...
Setting up liblxc1 (3.0.3-0ubuntu1~18.04.1) ...
Setting up liblxc-common (3.0.3-0ubuntu1~18.04.1) ...
apparmor.service is not active, cannot reload.
invoke-rc.d: initscript apparmor, action "reload" failed.
Setting up lxd (3.0.3-0ubuntu1~18.04.2) ...
...

$ lxc exec lp1959047 -- lxc exec c1 -- journalctl -b0 --grep 'Failed to set up namespace'
-- No entries --

So it worked, thanks!

** Tags removed: verification-needed verification-needed-bionic
** Tags added: verification-done verification-done-bionic

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1959047

Title:
  systemd ignores RootDirectory option in .service units

Status in lxd package in Ubuntu:
  Invalid
Status in systemd package in Ubuntu:
  Confirmed
Status in lxd source package in Bionic:
  Fix Committed
Status in systemd source package in Bionic:
  Confirmed
Status in systemd source package in Focal:
  Confirmed
Status in systemd source package in Impish:
  Confirmed

Bug description:
  [Impact]

  Ubuntu carries a patch on top of systemd [a] to silence
  namespace set up failures. This is meant as a workaround
  for a bug in the LXD version shipped in Ubuntu 18.04.

  Masking namespace set up failures creates a false sense of
  security for the user/admin.

  As mentioned in comment #1, systemd upstream explains that silencing
  this kind of error is dangerous and should be avoided.

  Backporting the LXD fix [b] to Ubuntu 18.04 would allow namespaces
  to work inside containers. This is the goal of this SRU.

  Ultimately, once LXD in Ubuntu 18.04 includes the fix [b], it would
  be possible to drop the Ubuntu-specific patch for systemd [a]. This
  is however *not an immediate concern for this SRU*.

  [Test Plan]

  1) Create a 18.04 VM:
  $ lxc launch images:ubuntu/18.04 lp1959047 --vm
  $ sleep 30  # give it time to boot

  1.5) Enable bionic-proposed:
  $ echo "deb http://archive.ubuntu.com/ubuntu bionic-proposed main restricted universe multiverse" | lxc file push - lp1959047/etc/apt/sources.list.d/proposed.list

  2) Install and initialize LXD in it:
  $ lxc exec lp1959047 -- apt-get update
  $ lxc exec lp1959047 -- apt-get install -y lxd
  $ lxc exec lp1959047 -- lxd init --auto

  3) Create a Jammy container and enable systemd debugging:
  $ lxc exec lp1959047 -- lxc init images:ubuntu/22.04 c1
  $ lxc exec lp1959047 -- lxc config set c1 raw.lxc 'lxc.init.cmd = /sbin/init systemd.log_level=debug'
  $ lxc exec lp1959047 -- lxc start c1

  4) Check if namespace set up failures are logged:
  $ lxc exec lp1959047 --  lxc exec c1 -- journalctl -b0 --grep 'Failed to set up namespace'
  Mar 24 23:29:19 c1 systemd[99]: systemd-udevd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:19 c1 systemd[132]: systemd-networkd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:19 c1 systemd[131]: systemd-logind.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:20 c1 systemd[136]: systemd-resolved.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:20 c1 systemd[128]: e2scrub_reap.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:23 c1 systemd[243]: systemd-hostnamed.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied

  If LXD in Ubuntu 18.04 has the patch, the "Failed to set up namespace"
  messages wouldn't be there.

  [Where problems could occur]

  The LXD fix changes the Apparmor profile used for containers. This essentially
  loosen the mount restrictions applied to containers.

  Weakening the Apparmor profile could make it easier for a process in the container
  to do damage that would have otherwise been blocked. On the other hand, this
  allows making use of namespaces/sandboxing inside the container.

  Upstream LXD has the fix since 2019 which make it less likely to run into
  problems with the backport.

  The backported fix was also tested manually to ensure LXD still behaved normally
  and that it avoided the namespace set up failures in Jammy containers.

  [a]: https://git.launchpad.net/ubuntu/+source/systemd/tree/debian/patches/debian/UBUNTU-Revert-namespace-be-more-careful-when-handling-namespacin.patch?h=ubuntu/jammy
  [b]: https://github.com/lxc/lxd/commit/a6b780703350faff8328f3d565f6bac7b6dcf59f

  [Initial bug description]

  The version of systemd (249.5-2ubuntu4) currently packaged for the
  Ubuntu development version (22.04 Jammy Jellyfish) totally ignores the
  RootDirectory= option in systemd service files. With RootDirectory,
  systemd should start the service after calling chroot() on the
  supplied directory.

  To test/reproduce, create a test service file with the following
  contents:

  # /etc/systemd/system/lsb-release.service
  [Unit]
  Description=LSB Release Information

  [Service]
  Type=simple
  RootDirectory=/var/chroot/trusty
  ExecStartPre=/bin/pwd
  ExecStart=/usr/bin/lsb_release -a

  You should have a chroot environment in the specified RootDirectory,
  even though you can still deduce if systemd attempted to chroot or not
  from the resulting error message.

  In my example, I installed an end-of-life Ubuntu 14.04 Trusty Tahr in
  the chroot environment. On systems NOT affected by the problem, I get
  the following result when I start this test service. This is what I'd
  expect.

  Jan 25 20:40:40 dolly systemd[1]: Starting LSB Release Information...
  Jan 25 20:40:40 dolly pwd[361]: /
  Jan 25 20:40:40 dolly systemd[1]: Started LSB Release Information.
  Jan 25 20:40:40 dolly lsb_release[362]: No LSB modules are available.
  Jan 25 20:40:40 dolly lsb_release[362]: Distributor ID:        Ubuntu
  Jan 25 20:40:40 dolly lsb_release[362]: Description:        Ubuntu 14.04 LTS
  Jan 25 20:40:40 dolly lsb_release[362]: Release:        14.04
  Jan 25 20:40:40 dolly lsb_release[362]: Codename:        trusty
  Jan 25 20:40:40 dolly systemd[1]: lsb-release.service: Succeeded.

  On the problematic system, however, I get the following result.

  Jan 25 21:21:08 savelog systemd[1]: Starting LSB Release Information...
  Jan 25 21:21:08 savelog systemd[1]: Started LSB Release Information.
  Jan 25 21:21:08 savelog pwd[81114]: /
  Jan 25 21:21:08 savelog lsb_release[81115]: No LSB modules are available.
  Jan 25 21:21:08 savelog lsb_release[81115]: Distributor ID:        Ubuntu
  Jan 25 21:21:08 savelog lsb_release[81115]: Description:        Ubuntu Jammy Jellyfish (development branch)
  Jan 25 21:21:08 savelog lsb_release[81115]: Release:        22.04
  Jan 25 21:21:08 savelog lsb_release[81115]: Codename:        jammy
  Jan 25 21:21:08 savelog systemd[1]: lsb-release.service: Deactivated successfully.

  It totally run the service on the host's root filesystem, it didn't
  care even the slightest that a RootDirectory is specified.

  Tested on the following releases / systemd versions:

  Ubuntu 18.04.6 Bionic Beaver – ISSUE NOT PRESENT
  systemd 237
  +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

  Ubuntu 20.04.3 Focal Fossa – ISSUE NOT PRESENT
  systemd 245 (245.4-4ubuntu3.15)
  +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

  Ubuntu 21.10 Impish Indri – ISSUE NOT PRESENT
  systemd 248 (248.3-1ubuntu8.2)
  +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

  Ubuntu 22.04 Jammy Jellyfish (development branch) – ISSUE PRESENT
  systemd 249 (249.5-2ubuntu4)
  +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

  Note that the problem is produced under an LXC container; since
  systemd detects virtualization, it might change how it behaves.

  It's either a bug or an intentional change I don't understand yet
  (i.e. the RootDirectory option has deprecated and is about to be
  replaced with something else, or there are additional conditions to be
  met before RootDirectory is considered), but I think in the latter
  case I should at least get a warning that there is a change in
  configuration. I imagine suddenly everyone's existing service units
  utilizing RootDirectory silently stop working without any information
  regarding why.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1959047/+subscriptions