[Bug 2004551] Re: upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

[Christian Ehrhardt ]

Yet another hit on the same

1. Maas deployed jammy GA 22.04
  This has 1:8.9p1-3

2. Tonight unattended-upgrade ran
   upgraded a lot of things
   # This is massive, but might be the first to recreate this

3. it triggers the same issue again

Aug 03 06:44:41 node-horsea sshd[8697]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 03 06:44:41 node-horsea sshd[8697]: error: Bind to port 22 on :: failed: Address already in use.
Aug 03 06:44:41 node-horsea sshd[8697]: fatal: Cannot bind any address.


Extra info:

There are a few things on this machine which might play into this.
It has some special network adapters, not all connected well (intentionally).

ubuntu at node-horsea:~$ networkctl 
IDX LINK  TYPE     OPERATIONAL SETUP      
  1 lo    loopback carrier     unmanaged
  2 eno1  ether    routable    configured 
  3 eno2  ether    degraded    configured 
  4 eno3  ether    no-carrier  configuring
  5 eno4  ether    no-carrier  configuring
  6 eno49 ether    degraded    configured 
  7 eno50 ether    degraded    configured 
  8 ens1  ether    no-carrier  configuring

e.g. this fails:

ubuntu at node-horsea:~$ /lib/systemd/systemd-networkd-wait-online
managing: eno2
managing: eno50
managing: eno49
managing: eno1

Timeout occurred while waiting for network connectivity.


When we activate apt-daily directly it fails over that.
But if not, then it updates and causes this crash.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551

Title:
  upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

Status in openssh package in Ubuntu:
  Confirmed
Status in systemd package in Ubuntu:
  New

Bug description:
  Hi,
  I just upgraded a system from Jammy to Lunar and openssh-server refuses to upgrade well.

  Setting up openssh-server (1:9.0p1-1ubuntu8) ...
  Replacing config file /etc/ssh/sshd_config with new version
  Replacing config file /etc/ssh/sshd_config with new version
  Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
  Executing: /lib/systemd/systemd-sysv-install disable ssh
  rescue-ssh.target is a disabled or a static unit not running, not starting it.
  Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
  dpkg: error processing package openssh-server (--configure):
   installed openssh-server package post-installation script subprocess returned error exit status 1
  Processing triggers for man-db (2.11.2-1) ...
  Processing triggers for libc-bin (2.36-0ubuntu4) ...
  Errors were encountered while processing:
   openssh-server
  Error: Timeout was reached
  needrestart is being skipped since dpkg has failed
  E: Sub-process /usr/bin/dpkg returned an error code (1)

  I'm not sure what exactly it is.
  This output complains about rescue-ssh.target and indeed that can not be started even directly.

  $ sudo systemctl start rescue-ssh.target
  A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for details.

  And in postinst is a try to start it:
  $  grep rescue /var/lib/dpkg/info/openssh-server.postinst 
  		deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null || true

  
  But I think the underlying issue is that ssh is already on, and I'm logged in via it.
  And that makes the service restart of the ssh socket which was added break.

  Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result 'resources'.

  
  Now, whichever it is, it is hard to resolve.
  The only way to get the socket to own it would be rebooting so that sshd lets go and systemd can take over.
  I could reboot, but that is not the point.
  What if I'd want to get the service and upgrade completed before reboot.
  Because as of now dpkg considers the system unhappy, and that would usually be a sign for "better not reboot before being resolved" to me.

  One thing though, I have not upgraded with do-release-upgrade - would
  we / do we have magic there to make the ssh socket activation
  transition smoother?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions