[Bug 2004551] Re: upgrade to lunar fails due to rescue-ssh.target or port 22 takeover
Miriam España Acebal
2004551 at bugs.launchpad.net
Fri Jul 21 08:58:29 UTC 2023
Hi,
I'm facing the same issue on the same machine (on MAAS), but I didn't
reach lunar yet: first, I did a do-dist-upgrade from jammy to Kinetic
(prompt normal).
I collected the info requested in comment #2:
ubuntu at node-horsea:~$ sudo fuser -n tcp 22
22/tcp: 1133 214495 214551
ubuntu at node-horsea:~$ sudo netstat -natpl | grep ":22"
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1133/sshd: /usr/sbi
tcp 0 52 10.245.171.244:22 10.172.195.194:55174 ESTABLISHED 214495/sshd: ubuntu
tcp6 0 0 :::22 :::* LISTEN 1133/sshd: /usr/sbi
ubuntu at node-horsea:~$ sudo ps -ef | grep 1133 | grep -v grep
root 1133 1 0 Jul13 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 214495 1133 0 Jul20 ? 00:00:00 sshd: ubuntu [priv]
ubuntu at node-horsea:~$ sudo ps -ef | grep 214551 | grep -v grep
ubuntu 214551 214495 0 Jul20 ? 00:00:00 sshd: ubuntu at pts/0
ubuntu 214552 214551 0 Jul20 pts/0 00:00:00 -bash
I experienced other messages indicating system degradation such as:
- ubuntu at node-horsea:~$ systemctl reload ssh.service
Failed to reload ssh.service: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
See system logs and 'systemctl status ssh.service' for details.
- ubuntu at node-horsea:~$ systemctl status ssh.service
Failed to get properties: Connection timed out
So I did:
ubuntu at node-horsea:~$ sudo telinit u
ubuntu at node-horsea:~$ sudo systemctl daemon-reexec
ubuntu at node-horsea:~$ sudo systemctl daemon-reload
And I get then:
ubuntu at node-horsea:~$ sudo systemctl status -l ssh.service
○ ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: inactive (dead)
TriggeredBy: × ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Jul 20 10:37:30 node-horsea sshd[225140]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jul 20 10:37:30 node-horsea sshd[225140]: error: Bind to port 22 on :: failed: Address already in use.
Jul 20 10:37:30 node-horsea sshd[225140]: fatal: Cannot bind any address.
Jul 20 10:37:30 node-horsea systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Jul 20 10:37:30 node-horsea systemd[1]: ssh.service: Failed with result 'exit-code'.
Jul 20 10:37:30 node-horsea systemd[1]: ssh.service: Unit process 1133 (sshd) remains running after unit stopped.
Jul 20 10:37:30 node-horsea systemd[1]: Failed to start OpenBSD Secure Shell server.
Jul 20 10:54:54 node-horsea sshd[278339]: Accepted publickey for ubuntu from 10.172.195.194 port 49122 ssh2: RSA SHA256:2MSrD9nviIiiRrpcOIF/MA8eD/>
Jul 20 10:54:54 node-horsea sshd[278339]: pam_unix(sshd:session): session opened for user ubuntu(uid=1000) by (uid=0)
Jul 20 10:54:55 node-horsea sshd[278339]: pam_env(sshd:session): deprecated reading of user environment enabled
The process is inactive but alive (I'm logged via ssh, and I was able to
make a second connection).
And, in a short period of time, the system behaves again the same:
ubuntu at node-horsea:~$ systemctl reload ssh.service
Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
And trying to get the package to a proper installation state still fails:
ubuntu at node-horsea:~$ sudo dpkg-reconfigure openssh-server
/usr/sbin/dpkg-reconfigure: openssh-server is broken or not fully installed
ubuntu at node-horsea:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
#
# An OpenSSL vulnerability has recently been fixed with USN-6188-1 & 6119-1:
# CVE-2023-2650: possible DoS translating ASN.1 object identifiers.
# Ensure you have updated the package to its latest version.
#
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up openssh-server (1:9.0p1-1ubuntu7.3) ...
Replacing config file /etc/ssh/sshd_config with new version
Replacing config file /etc/ssh/sshd_config with new version
Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ssh
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
installed openssh-server package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
openssh-server
Error: Timeout was reached
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
ubuntu at node-horsea:~$ sudo apt install --fix-broken
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up openssh-server (1:9.0p1-1ubuntu7.3) ...
Replacing config file /etc/ssh/sshd_config with new version
Replacing config file /etc/ssh/sshd_config with new version
Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ssh
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
installed openssh-server package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
openssh-server
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2023-2650
** Changed in: openssh (Ubuntu)
Status: Expired => New
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551
Title:
upgrade to lunar fails due to rescue-ssh.target or port 22 takeover
Status in openssh package in Ubuntu:
New
Bug description:
Hi,
I just upgraded a system from Jammy to Lunar and openssh-server refuses to upgrade well.
Setting up openssh-server (1:9.0p1-1ubuntu8) ...
Replacing config file /etc/ssh/sshd_config with new version
Replacing config file /etc/ssh/sshd_config with new version
Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ssh
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
installed openssh-server package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.11.2-1) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
Errors were encountered while processing:
openssh-server
Error: Timeout was reached
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
I'm not sure what exactly it is.
This output complains about rescue-ssh.target and indeed that can not be started even directly.
$ sudo systemctl start rescue-ssh.target
A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for details.
And in postinst is a try to start it:
$ grep rescue /var/lib/dpkg/info/openssh-server.postinst
deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null || true
But I think the underlying issue is that ssh is already on, and I'm logged in via it.
And that makes the service restart of the ssh socket which was added break.
Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result 'resources'.
Now, whichever it is, it is hard to resolve.
The only way to get the socket to own it would be rebooting so that sshd lets go and systemd can take over.
I could reboot, but that is not the point.
What if I'd want to get the service and upgrade completed before reboot.
Because as of now dpkg considers the system unhappy, and that would usually be a sign for "better not reboot before being resolved" to me.
One thing though, I have not upgraded with do-release-upgrade - would
we / do we have magic there to make the ssh socket activation
transition smoother?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions
More information about the foundations-bugs
mailing list