[Bug 2055806] Re: sshd.service ssh.socket systemd-tmpfiles-setup:Before= - Missing privilege separation directory: /run/sshd

Sun Mar 3 09:35:44 UTC 2024

Found this reference also to this folder.

```
''man sshd''

/run/sshd
               chroot(2) directory used by sshd during privilege separation in the pre‐authentication phase.  The directory should not contain any files and must be owned by root and not group or world‐writable.
```

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2055806

Title:
  sshd.service ssh.socket systemd-tmpfiles-setup:Before= - Missing
  privilege separation directory: /run/sshd

Status in openssh package in Ubuntu:
  New

Bug description:
  Symptom: ssh.service is not running and not accepting new SSH
  connections and returns socket closed error upon attempted connection.
  ssh.service shows a failed with error message "Missing privilege
  separation directory: /run/sshd".

  The following line is needed in the SystemD Unit configuration of the
  ssh.service file for the openssh-server package to fix what appears to
  be a race condition when ssh.service is sometimes started before
  systemd-tmpfiles-setup.service is finished processing the file in
  /usr/lib/tmpfiles.d/openssh.server.conf that is required to create the
  /run/sshd directory that is required by ssh.service to start property.

  There might be more depth or an alternative reason to why /run/sshd/
  still didn't exist after using ```systemctl reset-failed
  ssh.service```.  So more investigation is needed by someone with more
  background in systemd-tmpfiles-setup.service and openssh-server
  package after it when to socket activation.

  = Error Messages =

  ```
  $journalctl -u ssh.service -b-1

  Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 21:30:56 server sshd[1271]: Missing privilege separation directory: /run/sshd
  Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:56 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 1.
  Mar 02 21:30:56 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 21:30:56 server sshd[1369]: Missing privilege separation directory: /run/sshd
  Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:56 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 2.
  Mar 02 21:30:56 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 21:30:56 server sshd[1454]: Missing privilege separation directory: /run/sshd
  Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 3.
  Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:57 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 21:30:57 server sshd[1465]: Missing privilege separation directory: /run/sshd
  Mar 02 21:30:57 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 4.
  Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:57 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 21:30:57 server sshd[1475]: Missing privilege separation directory: /run/sshd
  Mar 02 21:30:57 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 5.
  Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 21:30:57 server systemd[1]: ssh.service: Start request repeated too quickly.
  Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:25 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:25 server sshd[47238]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:25 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:25 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:25 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:25 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 1.
  Mar 02 22:19:25 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:25 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:25 server sshd[47240]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:25 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:25 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:25 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 2.
  Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:26 server sshd[47241]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 3.
  Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:26 server sshd[47242]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 4.
  Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:26 server sshd[47243]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 5.
  Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:26 server systemd[1]: ssh.service: Start request repeated too quickly.
  Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:37 server sshd[47253]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:37 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 1.
  Mar 02 22:19:37 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:37 server sshd[47254]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:37 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 2.
  Mar 02 22:19:37 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:37 server sshd[47255]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 3.
  Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:38 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:38 server sshd[47256]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:38 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 4.
  Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:38 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
  Mar 02 22:19:38 server sshd[47257]: Missing privilege separation directory: /run/sshd
  Mar 02 22:19:38 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION
  Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 5.
  Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:38 server systemd[1]: ssh.service: Start request repeated too quickly.
  Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  Mar 02 22:19:45 server systemd[1]: ssh.service: Start request repeated too quickly.
  Mar 02 22:19:45 server systemd[1]: ssh.service: Failed with result 'exit-code'.
  Mar 02 22:19:45 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
  ```

  Note: Errors showed up a total of 3x5 times because the ```systemctl
  reset-failed ssh.service``` command was used and it still showed up.

  = Add to Package SystemD Unit File =

  ```
  ''/lib/systemd/system/ssh.service''

  [Unit]
  After=systemd-tmpfiles-setup.service
  ```

  = System Configuration =

  ```
  ''lsb_release -rd''

  No LSB modules are available.
  Description:    Ubuntu 23.10
  Release:        23.10
  ```

  ```
  ''cat /etc/lsb-release''

  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=23.10
  DISTRIB_CODENAME=mantic
  DISTRIB_DESCRIPTION="Ubuntu 23.10"
  ```

  ```
  ''apt-cache policy openssh-server''

  openssh-server:
    Installed: 1:9.3p1-1ubuntu3.2
    Candidate: 1:9.3p1-1ubuntu3.2
    Version table:
   *** 1:9.3p1-1ubuntu3.2 500
          500 http://us.archive.ubuntu.com/ubuntu mantic-updates/main amd64 Packages
          500 http://security.ubuntu.com/ubuntu mantic-security/main amd64 Packages
          100 /var/lib/dpkg/status
       1:9.3p1-1ubuntu3 500
          500 http://us.archive.ubuntu.com/ubuntu mantic/main amd64 Packages
  ```

  = Package Configuration Problems =

  ```
  ''systemctl --property=After show ssh.service''

  After=network.target sysinit.target basic.target auditd.service system.slice ssh.socket pollinate.service systemd-journald.socket
  ```

  ```
  ''systemctl --no-pager --property=Before show systemd-tmpfiles-setup.service''

  Before=ssh.service man-db.service logrotate.service systemd-timesyncd.service vgauth.service ModemManager.service fwupd.service e2scrub_reap.service initrd-switch-root.target systemd-update-utmp.service bluetooth.service open-vm-tools.service systemd-resolved.service polkit.service systemd-logind.service cockpit.service sysinit.target upower.service shutdown.target
  ```

  **Note: ssh.service needs to be in the Before= clause above to fix the
  race condition of the ssh.service starting before the
  /usr/lib/tmpfiles.d/openssh-server.conf is processed by systemd-
  tmpfiles-setup.service during start-up sequence otherwise /run/sshd
  directory won't exist.**

  ```
  ''cat /usr/lib/tmpfiles.d/openssh-server.conf''

  #Type   Path                            Mode    UID     GID     Age     Arguments
  D       /run/sshd                       0755    root    root    -       -
  ```

  = Temporary Solution =

  Create the directory and the Unit file with the After= clause to
  populate the Before= clause for systemd-tmpfiles-setup.service

  ```
  ''mkdir -p /etc/systemd/system/ssh.service.d''

  cat >/etc/systemd/system/ssh.service.d/after.conf <<EOF
  [Unit]
  After=systemd-tmpfiles-setup.service
  EOF

  ```

  Verify the file.

  ```
  ''cat /etc/systemd/system/ssh.service.d/after.conf''

  [Unit]
  After=systemd-tmpfiles-setup.service
  ```

  ```

  Reload SystemD config

  ```
  systemctl daemon-reload
  ```

  Verify new active config.

  ```
  ''systemctl --no-pager --property=Before show systemd-tmpfiles-setup.service''

  Before=... ssh.service ...
  ```

  ```
  ''systemctl --property=After show ssh.service''

  After=... systemd-tmpfiles-setup.service ...```

  === End of Bug Report ===

  ProblemType: Bug
  DistroRelease: Ubuntu 23.10
  Package: openssh-server 1:9.3p1-1ubuntu3.2
  ProcVersionSignature: Ubuntu 6.5.0-21.21-generic 6.5.8
  Uname: Linux 6.5.0-21-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.27.0-0ubuntu5
  Architecture: amd64
  CasperMD5CheckResult: unknown
  Date: Sun Mar  3 02:22:39 2024
  InstallationDate: Installed on 2020-12-14 (1174 days ago)
  InstallationMedia: Ubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
  ProcEnviron:
   LANG=en_US.UTF-8
   PATH=(custom, no user)
   SHELL=/bin/bash
   TERM=putty
  SourcePackage: openssh
  UpgradeStatus: Upgraded to mantic on 2024-02-26 (6 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2055806/+subscriptions