[Bug 2004551] Re: upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

Mon Jul 24 10:28:25 UTC 2023

Today I'm trying to reproduce the error in a lxc vm without success: I
logged in via ssh to start the do-release-upgrade there so the system
would initiate sshd on port 1022 as well. The upgrade process went fine
the times I tried it.

Thanks Andreas for spotting the same error in the autopkgtest log. I'm
looking for similarities between it and the horsea machine.

Checking horsea's log, I found the following

"FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ubuntu-
release-upgrader-x6gsdfa4/removal_denylist.cfg'"

in the screenlog.0 under /var/log/dist-upgrade directory:

Progress: [ 96%]
Processing triggers for man-db (2.10.2-1) ...
packages have been installed but needrestart is suspended
packages have been installed but needrestart is suspended
packages have been installed but needrestart is suspended

Reading package lists... 0%

Reading package lists... 100%

Reading package lists... Done

Building dependency tree... 0%

Building dependency tree... 50%

Building dependency tree... Done

Reading state information... 0%

Reading state information... Done

Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/kinetic", line 8, in <module>
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeMain.py", line 241, in main
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 2042, in run
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 1986, in fullUpgrade
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 207, in openCache
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 231, in _openCache
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeCache.py", line 142, in __init__
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeConfigParser.py", line 89, in getListFromFile
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ubuntu-release-upgrader-x6gsdfa4/removal_denylist.cfg'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 76, in apport_excepthook
    binary = os.path.realpath(os.path.join(os.getcwd(), sys.argv[0]))
FileNotFoundError: [Errno 2] No such file or directory

Original exception was:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/kinetic", line 8, in <module>
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeMain.py", line 241, in main
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 2042, in run
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 1986, in fullUpgrade
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 207, in openCache
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 231, in _openCache
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeCache.py", line 142, in __init__
  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/DistUpgrade/DistUpgradeConfigParser.py", line 89, in getListFromFile
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ubuntu-release-upgrader-x6gsdfa4/removal_denylist.cfg'

=== Command terminated with exit status 1 (Thu Jul 20 10:29:30 2023) ===-- 0:kinetic -- time-stamp -- Jul/20/23 10:31:29 --

It also appears in /var/log/dist-upgrade/main.log :

2023-07-20 10:27:36,606 DEBUG inhibit screensaver
2023-07-20 10:28:21,258 DEBUG quirks: running StartUpgrade
2023-07-20 10:28:21,258 DEBUG running Quirks.StartUpgrade
2023-07-20 10:28:21,259 DEBUG skipping 'README' (no '.')
2023-07-20 10:28:21,260 DEBUG killing update-notifier
2023-07-20 10:28:21,266 DEBUG killing kblueplugd kbluetooth4
2023-07-20 10:28:21,272 DEBUG apt btrfs snapshots supported: False
2023-07-20 10:28:21,272 INFO cache.commit()
2023-07-20 10:28:22,889 DEBUG cache.commit() returned None
2023-07-20 10:28:23,146 DEBUG quirks: running StartUpgrade
2023-07-20 10:28:23,146 DEBUG running Quirks.StartUpgrade
2023-07-20 10:28:23,147 DEBUG skipping 'README' (no '.')
2023-07-20 10:28:23,147 DEBUG killing update-notifier
2023-07-20 10:28:23,152 DEBUG killing kblueplugd kbluetooth4
2023-07-20 10:28:23,157 DEBUG apt btrfs snapshots supported: False
2023-07-20 10:28:23,157 INFO cache.commit()
2023-07-20 10:29:30,470 DEBUG cache.commit() returned None
2023-07-20 10:29:30,471 DEBUG openCache()
2023-07-20 10:29:30,847 ERROR getListFromFile: no '/tmp/ubuntu-release-upgrader-x6gsdfa4/removal_denylist.cfg' found
2023-07-20 10:29:30,848 ERROR not handled exception:
Traceback (most recent call last):

  File "/tmp/ubuntu-release-upgrader-x6gsdfa4/kinetic", line 8, in
<module>

  File "/tmp/ubuntu-release-
upgrader-x6gsdfa4/DistUpgrade/DistUpgradeMain.py", line 241, in main

  File "/tmp/ubuntu-release-
upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 2042, in
run

  File "/tmp/ubuntu-release-
upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 1986, in
fullUpgrade

  File "/tmp/ubuntu-release-
upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 207, in
openCache

  File "/tmp/ubuntu-release-
upgrader-x6gsdfa4/DistUpgrade/DistUpgradeController.py", line 231, in
_openCache

  File "/tmp/ubuntu-release-
upgrader-x6gsdfa4/DistUpgrade/DistUpgradeCache.py", line 142, in
__init__

  File "/tmp/ubuntu-release-
upgrader-x6gsdfa4/DistUpgrade/DistUpgradeConfigParser.py", line 89, in
getListFromFile

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ubuntu-
release-upgrader-x6gsdfa4/removal_denylist.cfg'

2023-07-20 10:29:30,848 DEBUG running apport_crash()

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551

Title:
  upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

Status in openssh package in Ubuntu:
  New

Bug description:
  Hi,
  I just upgraded a system from Jammy to Lunar and openssh-server refuses to upgrade well.

  Setting up openssh-server (1:9.0p1-1ubuntu8) ...
  Replacing config file /etc/ssh/sshd_config with new version
  Replacing config file /etc/ssh/sshd_config with new version
  Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
  Executing: /lib/systemd/systemd-sysv-install disable ssh
  rescue-ssh.target is a disabled or a static unit not running, not starting it.
  Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
  dpkg: error processing package openssh-server (--configure):
   installed openssh-server package post-installation script subprocess returned error exit status 1
  Processing triggers for man-db (2.11.2-1) ...
  Processing triggers for libc-bin (2.36-0ubuntu4) ...
  Errors were encountered while processing:
   openssh-server
  Error: Timeout was reached
  needrestart is being skipped since dpkg has failed
  E: Sub-process /usr/bin/dpkg returned an error code (1)

  I'm not sure what exactly it is.
  This output complains about rescue-ssh.target and indeed that can not be started even directly.

  $ sudo systemctl start rescue-ssh.target
  A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for details.

  And in postinst is a try to start it:
  $  grep rescue /var/lib/dpkg/info/openssh-server.postinst 
  		deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null || true

  But I think the underlying issue is that ssh is already on, and I'm logged in via it.
  And that makes the service restart of the ssh socket which was added break.

  Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result 'resources'.

  Now, whichever it is, it is hard to resolve.
  The only way to get the socket to own it would be rebooting so that sshd lets go and systemd can take over.
  I could reboot, but that is not the point.
  What if I'd want to get the service and upgrade completed before reboot.
  Because as of now dpkg considers the system unhappy, and that would usually be a sign for "better not reboot before being resolved" to me.

  One thing though, I have not upgraded with do-release-upgrade - would
  we / do we have magic there to make the ssh socket activation
  transition smoother?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions