[Bug 2011458] Re: ssh fails to rebind when it is killed with -HUP
Nick Rosbrook
2011458 at bugs.launchpad.net
Fri Jun 2 14:38:39 UTC 2023
I also tested upgrades from 1:9.0p1-1ubuntu7.3 in kinetic-proposed to
1:9.0p1-1ubuntu8.2 in lunar-proposed to confirm we won't be introducing
bug 2020474:
root at kinetic:~# ssh localhost
Welcome to Ubuntu 22.10 (GNU/Linux 6.2.0-20-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
root at kinetic:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu7.3
Candidate: 1:9.0p1-1ubuntu7.3
Version table:
*** 1:9.0p1-1ubuntu7.3 500
500 http://archive.ubuntu.com/ubuntu kinetic-proposed/main amd64 Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu7.1 500
500 http://archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
1:9.0p1-1ubuntu7 500
500 http://archive.ubuntu.com/ubuntu kinetic/main amd64 Packages
root at kinetic:~# sed -i 's/kinetic/lunar/g' /etc/apt/sources.list
root at kinetic:~# apt update
Get:1 http://archive.ubuntu.com/ubuntu lunar InRelease [267 kB]
Get:2 http://security.ubuntu.com/ubuntu lunar-security InRelease [109 kB]
Get:3 http://security.ubuntu.com/ubuntu lunar-security/main amd64 Packages [69.9 kB]
Get:4 http://archive.ubuntu.com/ubuntu lunar-updates InRelease [109 kB]
Get:5 http://security.ubuntu.com/ubuntu lunar-security/main Translation-en [20.5 kB]
Get:6 http://security.ubuntu.com/ubuntu lunar-security/universe amd64 Packages [33.4 kB]
Get:7 http://security.ubuntu.com/ubuntu lunar-security/universe Translation-en [11.0 kB]
Get:8 http://archive.ubuntu.com/ubuntu lunar-proposed InRelease [255 kB]
Get:9 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages [1,396 kB]
Get:10 http://archive.ubuntu.com/ubuntu lunar/main Translation-en [513 kB]
Get:11 http://archive.ubuntu.com/ubuntu lunar/restricted amd64 Packages [143 kB]
Get:12 http://archive.ubuntu.com/ubuntu lunar/restricted Translation-en [21.9 kB]
Get:13 http://archive.ubuntu.com/ubuntu lunar/universe amd64 Packages [15.0 MB]
Get:14 http://archive.ubuntu.com/ubuntu lunar/universe Translation-en [5,906 kB]
Get:15 http://archive.ubuntu.com/ubuntu lunar/multiverse amd64 Packages [236 kB]
Get:16 http://archive.ubuntu.com/ubuntu lunar/multiverse Translation-en [112 kB]
Get:17 http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 Packages [124 kB]
Get:18 http://archive.ubuntu.com/ubuntu lunar-updates/main Translation-en [33.5 kB]
Get:19 http://archive.ubuntu.com/ubuntu lunar-updates/universe amd64 Packages [63.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu lunar-updates/universe Translation-en [20.7 kB]
Get:21 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages [64.7 kB]
Get:22 http://archive.ubuntu.com/ubuntu lunar-proposed/main Translation-en [18.3 kB]
Get:23 http://archive.ubuntu.com/ubuntu lunar-proposed/restricted amd64 Packages [82.7 kB]
Get:24 http://archive.ubuntu.com/ubuntu lunar-proposed/restricted Translation-en [15.6 kB]
Get:25 http://archive.ubuntu.com/ubuntu lunar-proposed/universe amd64 Packages [20.0 kB]
Get:26 http://archive.ubuntu.com/ubuntu lunar-proposed/universe Translation-en [9,732 B]
Get:27 http://archive.ubuntu.com/ubuntu lunar-proposed/multiverse amd64 Packages [8,192 B]
Get:28 http://archive.ubuntu.com/ubuntu lunar-proposed/multiverse Translation-en [2,572 B]
Fetched 24.7 MB in 4s (5,656 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
204 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at kinetic:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu7.3
Candidate: 1:9.0p1-1ubuntu8
Version table:
1:9.0p1-1ubuntu8.2 100
100 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages
1:9.0p1-1ubuntu8 500
500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
*** 1:9.0p1-1ubuntu7.3 100
100 /var/lib/dpkg/status
root at kinetic:~# lsof -i :22
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 42u IPv6 1991217 0t0 TCP *:ssh (LISTEN)
sshd 1011 root 3u IPv6 1991217 0t0 TCP *:ssh (LISTEN)
ssh 1015 root 3u IPv6 1988536 0t0 TCP localhost:34296->localhost:ssh (ESTABLISHED)
sshd 1016 root 4u IPv6 1992984 0t0 TCP localhost:ssh->localhost:34296 (ESTABLISHED)
root at kinetic:~# apt install openssh-server=1:9.0p1-1ubuntu8.2 openssh-client=1:9.0p1-1ubuntu8.2 openssh-sftp-server=1:9.0p1-1ubuntu8.2 -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
libssl3 openssl
Suggested packages:
keychain libpam-ssh monkeysphere ssh-askpass molly-guard ufw
The following packages will be upgraded:
libssl3 openssh-client openssh-server openssh-sftp-server openssl
5 upgraded, 0 newly installed, 0 to remove and 199 not upgraded.
Need to get 4,449 kB of archives.
After this operation, 88.1 kB disk space will be freed.
Get:1 http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 libssl3 amd64 3.0.8-1ubuntu1.2 [1,902 kB]
Get:2 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 openssh-sftp-server amd64 1:9.0p1-1ubuntu8.2 [38.3 kB]
Get:3 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 openssh-server amd64 1:9.0p1-1ubuntu8.2 [431 kB]
Get:4 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 openssh-client amd64 1:9.0p1-1ubuntu8.2 [894 kB]
Get:5 http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 openssl amd64 3.0.8-1ubuntu1.2 [1,183 kB]
Fetched 4,449 kB in 0s (10.6 MB/s)
Preconfiguring packages ...
(Reading database ... 19344 files and directories currently installed.)
Preparing to unpack .../libssl3_3.0.8-1ubuntu1.2_amd64.deb ...
Unpacking libssl3:amd64 (3.0.8-1ubuntu1.2) over (3.0.5-2ubuntu2.3) ...
Preparing to unpack .../openssh-sftp-server_1%3a9.0p1-1ubuntu8.2_amd64.deb ...
Unpacking openssh-sftp-server (1:9.0p1-1ubuntu8.2) over (1:9.0p1-1ubuntu7.3) ...
Preparing to unpack .../openssh-server_1%3a9.0p1-1ubuntu8.2_amd64.deb ...
Unpacking openssh-server (1:9.0p1-1ubuntu8.2) over (1:9.0p1-1ubuntu7.3) ...
Preparing to unpack .../openssh-client_1%3a9.0p1-1ubuntu8.2_amd64.deb ...
Unpacking openssh-client (1:9.0p1-1ubuntu8.2) over (1:9.0p1-1ubuntu7.3) ...
Preparing to unpack .../openssl_3.0.8-1ubuntu1.2_amd64.deb ...
Unpacking openssl (3.0.8-1ubuntu1.2) over (3.0.5-2ubuntu2.3) ...
Setting up libssl3:amd64 (3.0.8-1ubuntu1.2) ...
Setting up openssl (3.0.8-1ubuntu1.2) ...
Installing new version of config file /etc/ssl/openssl.cnf ...
Setting up openssh-client (1:9.0p1-1ubuntu8.2) ...
Setting up openssh-sftp-server (1:9.0p1-1ubuntu8.2) ...
Setting up openssh-server (1:9.0p1-1ubuntu8.2) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Processing triggers for libc-bin (2.36-0ubuntu4) ...
root at kinetic:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu8.2
Candidate: 1:9.0p1-1ubuntu8.2
Version table:
*** 1:9.0p1-1ubuntu8.2 100
100 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu8 500
500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
root at kinetic:~# systemctl status ssh.socket ssh.service
● ssh.socket - OpenBSD Secure Shell server socket
Loaded: loaded (/lib/systemd/system/ssh.socket; enabled; preset: enabled)
Active: active (listening) since Fri 2023-06-02 14:24:43 UTC; 19s ago
Until: Fri 2023-06-02 14:24:43 UTC; 19s ago
Triggers: ● ssh.service
Listen: [::]:22 (Stream)
CGroup: /system.slice/ssh.socket
Jun 02 14:24:43 kinetic systemd[1]: Listening on OpenBSD Secure Shell
server socket.
○ ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: inactive (dead) since Fri 2023-06-02 14:24:42 UTC; 20s ago
Duration: 2min 12.034s
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 1011 (code=exited, status=0/SUCCESS)
CPU: 104ms
Jun 02 14:22:30 kinetic sshd[1011]: Server listening on :: port 22.
Jun 02 14:22:30 kinetic systemd[1]: Started OpenBSD Secure Shell server.
Jun 02 14:22:34 kinetic sshd[1012]: Connection closed by authenticating user root ::1 port 37598 [preauth]
Jun 02 14:22:45 kinetic sshd[1016]: Accepted publickey for root from ::1 port 34296 ssh2: RSA SHA256:BpuS10kEBPBrKFF6c>
Jun 02 14:22:45 kinetic sshd[1016]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jun 02 14:22:46 kinetic sshd[1016]: pam_env(sshd:session): deprecated reading of user environment enabled
Jun 02 14:24:42 kinetic sshd[1011]: Received signal 15; terminating.
Jun 02 14:24:42 kinetic systemd[1]: Stopping OpenBSD Secure Shell server...
Jun 02 14:24:42 kinetic systemd[1]: ssh.service: Deactivated successfully.
Jun 02 14:24:42 kinetic systemd[1]: Stopped OpenBSD Secure Shell server.
** Tags removed: verification-needed verification-needed-kinetic verification-needed-lunar
** Tags added: verification-done verification-done-kinetic verification-done-lunar
** Tags removed: block-proposed-kinetic block-proposed-lunar
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2011458
Title:
ssh fails to rebind when it is killed with -HUP
Status in openssh package in Ubuntu:
Fix Released
Status in openssh source package in Kinetic:
Fix Committed
Status in openssh source package in Lunar:
Fix Committed
Bug description:
[Impact]
The sshd re-execution logic is generally broken with systemd socket activation, which means that (1) sshd fails when it is told to re-exec
via SIGHUP (e.g. systemctl reload ssh), and (2) sshd fails when started in debug mode.
[Test Case]
(1) Test systemctl reload ssh:
* On a machine with openssh-server installed, make a connection to
localhost to activate ssh.service (the connection does not need to be
complete, so you can just say "no" at the host key verification
stage):
$ ssh localhost
[...]
* Send SIGHUP to sshd by calling systemctl reload ssh:
$ systemctl reload ssh
* Check the service state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:43:27 UTC; 4s ago
Duration: 2min 44.132s
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1112 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION)
Process: 1152 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1153 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 1112 (code=exited, status=255/EXCEPTION)
CPU: 79ms
Apr 17 20:40:43 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Apr 17 20:41:06 lunar sshd[1113]: Connection closed by 127.0.0.1 port 54666 [preauth]
Apr 17 20:43:27 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure Shell server...
Apr 17 20:43:27 lunar sshd[1112]: Received SIGHUP; restarting.
Apr 17 20:43:27 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on :: failed: Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: fatal: Cannot bind any address.
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
* On an affected machine, the service will fail as shown above.
(2) Test debug mode:
* On a machine with openssh-server installed, edit /etc/default/ssh to
configure debug mode for sshd:
$ cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
* Attempt to make a connection to localhost:
$ ssh localhost
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 22
* On an affected machine, the attempt will fail as shown above, and
the service will be in a failed state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:46:34 UTC; 2min 27s ago
Duration: 5ms
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1166 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1167 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION)
Main PID: 1167 (code=exited, status=255/EXCEPTION)
CPU: 40ms
Apr 17 20:46:34 lunar sshd[1167]: Server listening on :: port 22.
Apr 17 20:46:34 lunar sshd[1167]: debug3: fd 4 is not O_NONBLOCK
Apr 17 20:46:34 lunar sshd[1167]: debug1: Server will not fork when running in debugging mode.
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: entering fd = 7 config len 3456
Apr 17 20:46:34 lunar sshd[1167]: debug3: ssh_msg_send: type 0
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: done
Apr 17 20:46:34 lunar sshd[1167]: debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
Apr 17 20:46:34 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
[Where problems could occur]
The fix expands Ubuntu's patch for systemd socket activation to try
and make sure that any fds passed from systemd are not closed across
re-executions of sshd. If we saw a problem, it would most likely be an
attempt to operate on a closed fd, or the wrong fd, as a result of an
edge case in one of the re-execution paths.
[Original Description]
In kinetic and lunar gce images we are facing an issue when ssh is being killed with -HUP
SSH is failing to rebind port 22. It is not failing in other previous systems.
It can be reproduced by running
# pkill -o -HUP sshd || true
# journalctl -n 20
Mar 13 14:58:52 mar131454-025105 sshd[1371]: Received SIGHUP; restarting.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on :: failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: fatal: Cannot bind any address.
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Failed with result 'exit-code'.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2011458/+subscriptions
More information about the foundations-bugs
mailing list