[Bug 2011458] Re: ssh fails to rebind when it is killed with -HUP
Nick Rosbrook
2011458 at bugs.launchpad.net
Mon May 15 14:25:23 UTC 2023
I verified the fix using openssh-server 1:9.0p1-1ubuntu8.1 from lunar-
proposed:
Test 1:
root at lunar:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu8.1
Candidate: 1:9.0p1-1ubuntu8.1
Version table:
*** 1:9.0p1-1ubuntu8.1 100
100 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu8 500
500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
root at lunar:~# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:AbCnblNKQGNc02dY90hZSSobjAiPKfXOOs0YFUhVe9c.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? no
Host key verification failed.
root at lunar:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Mon 2023-05-15 13:51:10 UTC; 16s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 148 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 149 (sshd)
Tasks: 1 (limit: 18854)
Memory: 1.3M
CPU: 71ms
CGroup: /system.slice/ssh.service
└─149 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
May 15 13:51:10 lunar systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
May 15 13:51:10 lunar sshd[149]: Server listening on :: port 22.
May 15 13:51:10 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
May 15 13:51:12 lunar sshd[150]: Connection closed by ::1 port 37598 [preauth]
root at lunar:~# systemctl reload ssh
root at lunar:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Mon 2023-05-15 13:51:10 UTC; 23s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 148 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 158 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 159 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 149 (sshd)
Tasks: 1 (limit: 18854)
Memory: 1.3M
CPU: 127ms
CGroup: /system.slice/ssh.service
└─149 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
May 15 13:51:10 lunar systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
May 15 13:51:10 lunar sshd[149]: Server listening on :: port 22.
May 15 13:51:10 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
May 15 13:51:12 lunar sshd[150]: Connection closed by ::1 port 37598 [preauth]
May 15 13:51:33 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure Shell server...
May 15 13:51:33 lunar sshd[149]: Received SIGHUP; restarting.
May 15 13:51:33 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
May 15 13:51:33 lunar sshd[149]: Server listening on :: port 22.
Test 2:
root at lunar:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu8.1
Candidate: 1:9.0p1-1ubuntu8.1
Version table:
*** 1:9.0p1-1ubuntu8.1 100
100 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu8 500
500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
root at lunar:~# vi /etc/default/ssh
root at lunar:~# cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
root at lunar:~# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:AbCnblNKQGNc02dY90hZSSobjAiPKfXOOs0YFUhVe9c.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Welcome to Ubuntu 23.04 (GNU/Linux 6.2.0-20-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
debug3: Copy environment: XDG_SESSION_ID=17
debug3: Copy environment: XDG_RUNTIME_DIR=/run/user/0
debug3: Copy environment: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
debug3: Copy environment: XDG_SESSION_TYPE=tty
debug3: Copy environment: XDG_SESSION_CLASS=user
debug3: Copy environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
debug3: Copy environment: LANG=en_US.UTF-8
Environment:
LANG=en_US.UTF-8
USER=root
LOGNAME=root
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL=/bin/bash
TERM=xterm-256color
XDG_SESSION_ID=17
XDG_RUNTIME_DIR=/run/user/0
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user
SSH_CLIENT=::1 38966 22
SSH_CONNECTION=::1 38966 ::1 22
SSH_TTY=/dev/pts/2
root at lunar:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Mon 2023-05-15 14:22:15 UTC; 8s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 153 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 154 (sshd)
Tasks: 0 (limit: 18854)
Memory: 1.9M
CPU: 89ms
CGroup: /system.slice/ssh.service
‣ 154 "sshd: root at pts/2"
May 15 14:22:17 lunar sshd[154]: debug1: server_input_channel_req: channel 0 request shell reply 1
May 15 14:22:17 lunar sshd[154]: debug1: session_by_channel: session 0 channel 0
May 15 14:22:17 lunar sshd[154]: debug1: session_input_channel_req: session 0 req shell
May 15 14:22:17 lunar sshd[154]: Starting session: shell on pts/2 for root from ::1 port 38966 id 0
May 15 14:22:17 lunar sshd[154]: debug2: fd 5 setting TCP_NODELAY
May 15 14:22:17 lunar sshd[154]: debug3: set_sock_tos: set socket 5 IPV6_TCLASS 0x10
May 15 14:22:17 lunar sshd[154]: debug2: channel 0: rfd 11 isatty
May 15 14:22:17 lunar sshd[154]: debug2: fd 11 setting O_NONBLOCK
May 15 14:22:17 lunar sshd[154]: debug3: fd 8 is O_NONBLOCK
May 15 14:22:17 lunar sshd[154]: debug3: send packet: type 99
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2011458
Title:
ssh fails to rebind when it is killed with -HUP
Status in openssh package in Ubuntu:
Fix Committed
Status in openssh source package in Kinetic:
Fix Committed
Status in openssh source package in Lunar:
Fix Committed
Bug description:
[Impact]
The sshd re-execution logic is generally broken with systemd socket activation, which means that (1) sshd fails when it is told to re-exec
via SIGHUP (e.g. systemctl reload ssh), and (2) sshd fails when started in debug mode.
[Test Case]
(1) Test systemctl reload ssh:
* On a machine with openssh-server installed, make a connection to
localhost to activate ssh.service (the connection does not need to be
complete, so you can just say "no" at the host key verification
stage):
$ ssh localhost
[...]
* Send SIGHUP to sshd by calling systemctl reload ssh:
$ systemctl reload ssh
* Check the service state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:43:27 UTC; 4s ago
Duration: 2min 44.132s
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1112 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION)
Process: 1152 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1153 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 1112 (code=exited, status=255/EXCEPTION)
CPU: 79ms
Apr 17 20:40:43 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Apr 17 20:41:06 lunar sshd[1113]: Connection closed by 127.0.0.1 port 54666 [preauth]
Apr 17 20:43:27 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure Shell server...
Apr 17 20:43:27 lunar sshd[1112]: Received SIGHUP; restarting.
Apr 17 20:43:27 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on :: failed: Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: fatal: Cannot bind any address.
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
* On an affected machine, the service will fail as shown above.
(2) Test debug mode:
* On a machine with openssh-server installed, edit /etc/default/ssh to
configure debug mode for sshd:
$ cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
* Attempt to make a connection to localhost:
$ ssh localhost
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 22
* On an affected machine, the attempt will fail as shown above, and
the service will be in a failed state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:46:34 UTC; 2min 27s ago
Duration: 5ms
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1166 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1167 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION)
Main PID: 1167 (code=exited, status=255/EXCEPTION)
CPU: 40ms
Apr 17 20:46:34 lunar sshd[1167]: Server listening on :: port 22.
Apr 17 20:46:34 lunar sshd[1167]: debug3: fd 4 is not O_NONBLOCK
Apr 17 20:46:34 lunar sshd[1167]: debug1: Server will not fork when running in debugging mode.
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: entering fd = 7 config len 3456
Apr 17 20:46:34 lunar sshd[1167]: debug3: ssh_msg_send: type 0
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: done
Apr 17 20:46:34 lunar sshd[1167]: debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
Apr 17 20:46:34 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
[Where problems could occur]
The fix expands Ubuntu's patch for systemd socket activation to try
and make sure that any fds passed from systemd are not closed across
re-executions of sshd. If we saw a problem, it would most likely be an
attempt to operate on a closed fd, or the wrong fd, as a result of an
edge case in one of the re-execution paths.
[Original Description]
In kinetic and lunar gce images we are facing an issue when ssh is being killed with -HUP
SSH is failing to rebind port 22. It is not failing in other previous systems.
It can be reproduced by running
# pkill -o -HUP sshd || true
# journalctl -n 20
Mar 13 14:58:52 mar131454-025105 sshd[1371]: Received SIGHUP; restarting.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on :: failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: fatal: Cannot bind any address.
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Failed with result 'exit-code'.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2011458/+subscriptions
More information about the foundations-bugs
mailing list