[Bug 2046486] Re: units with credentials fail in LXD containers

[Nick Rosbrook]

I found that after working around this issue (with seccomp rules) there
are yet more AppArmor denials during namespace set up.

All in all, systemd services with sandboxing settings (i.e. settings
that require the use of various namespaces) hit more and more denials in
LXD containers. So, after discussing with LXD folks, the plan is to
enable security.nesting: true by default for unprivileged containers
[1].

[1] https://github.com/canonical/lxd/issues/13631

** Summary changed:

- units with SetCredential= fail in LXD containers
+ units with credentials fail in LXD containers

** Tags added: block-proposed

** Bug watch added: github.com/canonical/lxd/issues #13631
   https://github.com/canonical/lxd/issues/13631

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2046486

Title:
  units with credentials fail in LXD containers

Status in systemd package in Ubuntu:
  Triaged

Bug description:
  To demonstrate this, in an unprivileged LXD container, create the
  following unit (taken from the systemd test suite):

  $ cat > /etc/systemd/system/exec-set-credential.service << EOF
  # SPDX-License-Identifier: LGPL-2.1-or-later
  [Unit]
  Description=Test for SetCredential=

  [Service]
  ExecStart=/bin/sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
  ExecStartPost=/bin/sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
  ExecStop=/bin/sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
  ExecStopPost=/bin/sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
  Type=oneshot
  SetCredential=test-execute.set-credential:hoge
  EOF
  $ systemctl daemon-reload
  $ systemctl start exec-set-credential.service
  Job for exec-set-credential.service failed because the control process exited with error code.
  See "systemctl status exec-set-credential.service" and "journalctl -xeu exec-set-credential.service" for details.

  With debug logs enabled, we see:

  $ journalctl -u exec-set-credential.service -b --no-pager
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Trying to enqueue job exec-set-credential.service/start/replace
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Installed new job exec-set-credential.service/start as 2740
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Enqueued job exec-set-credential.service/start as 2740
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Will spawn child (service_enter_start): /bin/sh
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Failed to set 'trusted.invocation_id' xattr on control group /system.slice/exec-set-credential.service, ignoring: Operation not permitted
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Failed to remove 'trusted.delegate' xattr flag on control group /system.slice/exec-set-credential.service, ignoring: Operation not permitted
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Failed to remove 'trusted.survive_final_kill_signal' xattr flag on control group /system.slice/exec-set-credential.service, ignoring: Operation not permitted
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Passing 0 fds to service
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: About to execute: /bin/sh -x -c "test \"1031(cat /run/credentials/exec-set-credential.service/test-execute.set-credential)\" = \"hoge\""
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Forked /bin/sh as 2183
  Dec 14 19:24:24 noble (sh)[2183]: PR_SET_MM_ARG_START failed: Operation not permitted
  Dec 14 19:24:24 noble (sh)[2183]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
  Dec 14 19:24:24 noble (sh)[2183]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Changed dead -> start
  Dec 14 19:24:24 noble systemd[1]: Starting exec-set-credential.service - Test for SetCredential=...
  Dec 14 19:24:24 noble (sh)[2183]: Successfully forked off '(sd-mkdcreds)' as PID 2184.
  Dec 14 19:24:24 noble (sd-[2184]: Changing mount propagation /dev (MS_REC|MS_SLAVE "")
  Dec 14 19:24:24 noble (sd-[2184]: Mounting ramfs (ramfs) on /dev/shm (MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_NOSYMFOLLOW "mode=0700")...
  Dec 14 19:24:24 noble (sd-[2184]: Changing mount flags /dev/shm (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_NOSYMFOLLOW|MS_BIND "")...
  Dec 14 19:24:24 noble (sd-[2184]: Failed to mount n/a (type n/a) on /dev/shm (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_NOSYMFOLLOW|MS_BIND ""): Permission denied
  Dec 14 19:24:24 noble (sh)[2183]: (sd-mkdcreds) failed with exit status 1.
  Dec 14 19:24:24 noble (sh)[2183]: exec-set-credential.service: Failed to set up credentials: Protocol error
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Child 2183 belongs to exec-set-credential.service.
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Main process exited, code=exited, status=243/CREDENTIALS
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Will spawn child (service_enter_stop_post): /bin/sh
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: About to execute: /bin/sh -x -c "test \"1031(cat /run/credentials/exec-set-credential.service/test-execute.set-credential)\" = \"hoge\""
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Forked /bin/sh as 2186
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Changed start -> stop-post
  Dec 14 19:24:24 noble (sh)[2186]: PR_SET_MM_ARG_START failed: Operation not permitted
  Dec 14 19:24:24 noble (sh)[2186]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
  Dec 14 19:24:24 noble (sh)[2186]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
  Dec 14 19:24:24 noble sh[2186]: + test 1031(cat /run/credentials/exec-set-credential.service/test-execute.set-credential) = hoge
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Child 2186 belongs to exec-set-credential.service.
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Control process exited, code=exited, status=1/FAILURE
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Got final SIGCHLD for state stop-post.
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Failed with result 'exit-code'.
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Service will not restart (restart setting)
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Changed stop-post -> failed
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Job 2740 exec-set-credential.service/start finished, result=failed
  Dec 14 19:24:24 noble systemd[1]: Failed to start exec-set-credential.service - Test for SetCredential=.
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Unit entered failed state.
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Consumed 23ms CPU time.
  Dec 14 19:24:24 noble systemd[1]: exec-set-credential.service: Releasing resources...

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2046486/+subscriptions