[Bug 2004551] Re: upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

[Christian Ehrhardt ]

Sadly, or better gladly, Pitti killed most of my theories which were
around the transition from pre-socket to socket-activated mode. Thanks
for stopping me from doing more debug that way :-)

FYI - There recently has been bug 2020474 fixed in Mantic.
It seems to be rather similar and throws another tunable into the mix, having proposed enabled or not.
See comment #17 in that bug for more.
Sadly - the example by Andreas in comment #19 is after this fix for mantic. So it can't be all that is needed.

Anyway this made me realize I was trying to recreate on mantic (which I
now know has this fix) while it was seen on lunar before.

So I removed all I had on mantic and went back to Lunar ... trying to
look for weird state combinations to trigger this.

In the background I ran:
$ while /bin/true; do sleep 0.1s; for s in ssh.service ssh.socket; do printf '%15s: %15s %15s '  "$s" "$(systemctl is-enabled $s)" "$(systemctl is-active $s)"; done; printf "\n"; done

And in another session I ran server actions or loops of actions.

After install:
    ssh.service: disabled inactive               ssh.socket: enabled active
logging in via ssh:              
    ssh.service: disabled activating             ssh.socket: enabled active
    ssh.service: disabled active                 ssh.socket: enabled active
upgrade or re-install pkg
    ssh.service: disabled inactive               ssh.socket: enabled inactive
    ssh.service: disabled inactive               ssh.socket: enabled active

So much for states and states that are to be expected.
But so far no matter how many concurrent re-installs, downgrades (to 1:9.0p1-1ubuntu8), upgrades, enable/disable, ssh logins, ... I did - I could not yet trigger the problem in an isolated reproducible environment.
As Robie said, we are looking for someone able to fill that gap to continue ...

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551

Title:
  upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

Status in openssh package in Ubuntu:
  Confirmed
Status in systemd package in Ubuntu:
  New

Bug description:
  Hi,
  I just upgraded a system from Jammy to Lunar and openssh-server refuses to upgrade well.

  Setting up openssh-server (1:9.0p1-1ubuntu8) ...
  Replacing config file /etc/ssh/sshd_config with new version
  Replacing config file /etc/ssh/sshd_config with new version
  Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
  Executing: /lib/systemd/systemd-sysv-install disable ssh
  rescue-ssh.target is a disabled or a static unit not running, not starting it.
  Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
  dpkg: error processing package openssh-server (--configure):
   installed openssh-server package post-installation script subprocess returned error exit status 1
  Processing triggers for man-db (2.11.2-1) ...
  Processing triggers for libc-bin (2.36-0ubuntu4) ...
  Errors were encountered while processing:
   openssh-server
  Error: Timeout was reached
  needrestart is being skipped since dpkg has failed
  E: Sub-process /usr/bin/dpkg returned an error code (1)

  I'm not sure what exactly it is.
  This output complains about rescue-ssh.target and indeed that can not be started even directly.

  $ sudo systemctl start rescue-ssh.target
  A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for details.

  And in postinst is a try to start it:
  $  grep rescue /var/lib/dpkg/info/openssh-server.postinst 
  		deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null || true

  
  But I think the underlying issue is that ssh is already on, and I'm logged in via it.
  And that makes the service restart of the ssh socket which was added break.

  Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result 'resources'.

  
  Now, whichever it is, it is hard to resolve.
  The only way to get the socket to own it would be rebooting so that sshd lets go and systemd can take over.
  I could reboot, but that is not the point.
  What if I'd want to get the service and upgrade completed before reboot.
  Because as of now dpkg considers the system unhappy, and that would usually be a sign for "better not reboot before being resolved" to me.

  One thing though, I have not upgraded with do-release-upgrade - would
  we / do we have magic there to make the ssh socket activation
  transition smoother?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions