[Bug 2028282] Re: [SRU] SSH pubkey authetication fails when GSSAPI enabled

Mon Mar 10 14:22:59 UTC 2025

I tested openssh (1:9.6p1-3ubuntu13.9) from noble-proposed, according to
"Test Plan 2". Looking good!

[ Test Plan 2 ]

###Set up a Noble LXD container & install openssh-server from proposed:
root at nnsru:~# apt list *openssh-server*
Listing... Done
openssh-server/noble-proposed,now 1:9.6p1-3ubuntu13.9 amd64 [installed]
root at nnsru:~# adduser test
info: Adding user `test' ...
info: Selecting UID/GID from range 1000 to 59999 ...
info: Adding new group `test' (1001) ...
info: Adding new user `test' (1001) with group `test (1001)' ...
info: Creating home directory `/home/test' ...
info: Copying files from `/etc/skel' ...
New password:  [test]
Retype new password: [test]
passwd: password updated successfully
Changing the user information for test
Enter the new value, or press ENTER for the default
	Full Name []: 
	Room Number []: 
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] 
info: Adding new user `test' to supplemental / extra groups `users' ...
info: Adding user `test' to group `users' ...

### Adopt SSH config & restart service
root at nnsru:~# vim /etc/ssh/sshd_config.d/60-cloudimg-settings.conf
root at nnsru:~# grep -R PasswordAuthentication /etc/ssh/
/etc/ssh/ssh_config:#   PasswordAuthentication yes
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication yes
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication.  Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
root at nnsru:~# systemctl restart ssh.service
root at nnsru:~# ip a show eth0
199: eth0 at if200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:04:41:7d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.238.94.186/24 metric 100 brd 10.238.94.255 scope global dynamic eth0
       valid_lft 3319sec preferred_lft 3319sec
    inet6 fd42:7213:f20e:bd74:216:3eff:fe04:417d/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe04:417d/64 scope link 
       valid_lft forever preferred_lft forever

### From the host (password login OK):
$ ssh test at 10.238.94.186
The authenticity of host '10.238.94.186 (10.238.94.186)' can't be established.
ED25519 key fingerprint is SHA256:t2qgSMbZOw1Pm6PXVOL5lXYCtD/JDBUoJZjwygRshNs.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.238.94.186' (ED25519) to the list of known hosts.
test at 10.238.94.186's password: [test]

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

test at nnsru:~$ ssh-import-id-lp slyon
2025-03-10 14:17:08,923 INFO Authorized key ['4096', 'SHA256:sciOAYEEOgZuev6e/fxLpojXxsiZsJPzn1Jk8LaYvVg', 'lukas.maerdian at canonical.com', '(RSA)']
2025-03-10 14:17:08,923 INFO [1] SSH keys [Authorized]

### From the host (pubkey login OK):
$ ssh -i ~/.ssh/canonical_id_rsa test at 10.238.94.186
Enter passphrase for key '/home/lukas/.ssh/canonical_id_rsa': 
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-54-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

 System information as of Mon Mar 10 14:18:23 UTC 2025

  System load:           1.57
  Usage of /:            69.8% of 294.23GB
  Memory usage:          0%
  Swap usage:            0%
  Temperature:           49.0 C
  Processes:             29
  Users logged in:       1
  IPv4 address for eth0: 10.238.94.186
  IPv6 address for eth0: fd42:7213:f20e:bd74:216:3eff:fe04:417d

Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status

Last login: Mon Mar 10 14:16:57 2025 from 10.238.94.1
test at nnsru:~$

=> All working as expected!

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2028282

Title:
  [SRU] SSH pubkey authetication fails when GSSAPI enabled

Status in openssh package in Ubuntu:
  Fix Released
Status in openssh source package in Jammy:
  Triaged
Status in openssh source package in Noble:
  Fix Committed
Status in openssh source package in Oracular:
  Fix Released
Status in openssh source package in Plucky:
  Fix Released
Status in openssh package in Debian:
  Fix Released

Bug description:
  [ Impact ]

   * Login with publickey fails when openssh server is configured to use
  GSSAPI authentication, too. Error: "sign_and_send_pubkey: internal
  error: initial hostkey not recorded"

   * To trigger it, one needs to (a) perform a successful GSSAPI key
  exchange, (b) attempt public key authentication.

   * In addition, the client and the server must both have the hostbound
  authentication protocol extension enabled for the problem to manifest
  itself (On by default).

   * This is not a very common combination, but it can happen if one has
  Kerberos credentials for the correct realm but the wrong user, and a
  private key for the right user.

   * This SRU fixes this by adding an additional
  "ssh->kex->initial_hostkey != NULL" check in
  sshconnect2.c:sign_and_send_pubkey(), as suggested by upstream in
  https://bugzilla.mindrot.org/show_bug.cgi?id=3406 (comment 2).

  [ Test Plan ]

   The reproducer was codified in autopkgtests, thanks to Colin Watson!

   * Make sure to have the latest debian/tests/ssh-gssapi test case
  (included as of 1:9.9p1-2, and shipped as part of this SRU),
  especially the delta described in
  https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2028282/+attachment/5845545/+files/dep8-verifier.diff

   * Execute the "ssh-gssapi" dep8 test:
  $ autopkgtest -U openssh --apt-pocket=proposed=src:openssh --test-name=ssh-gssapi -- lxd autopkgtest/ubuntu/oracular/amd64

   * Confirm the log contains 3 login attempts, with the final one using the "publickey" authentication method ("Accepted publickey for testuser..."):
  """
  ## Checking ssh logs to confirm publickey auth was used
  Dec 14 22:44:16 sshd-gssapi.example.fake sshd-session[2213]: Accepted publickey for testuser2020-2 from 127.0.0.1 port 43364 ssh2: ED25519 SHA256:7vF3468XCZOawompwDThLsGsnPoUaP5Ki/3KaQLq/2M
  ## PASS test_gssapi_keyex_pubkey_fallback
  """

  [ Test Plan 2 ]

   * In addition to the codified test for this specific issue, we want
  to confirm normal password and publickey login are still working as
  expected.

   * Enable "PasswordAuthentication yes" in /etc/ssh/sshd_config &
  restart ssh.service

   * Login using password, confirm success

   * Copy public key over to system-under-test

   * Enable "PubkeyAuthentication yes" in /etc/ssh/sshd_config & restart

   * Login using private key, confirm success

  [ Where problems could occur ]

   * This SRU tweaks the authentication logic of OpenSSH, therefore it's
  a high-impact change. If something goes wrong, it could lock people
  out of their remote machines.

   * The change has been deployed to Debian testing and Ubuntu Plucky
  since October 2024, without major issues raised.

   * I've added "[ Test Plan 2 ]" to confirm normal publickey & password
  login is still working as expected

  [ Other Info ]

   * Fixed as of 1:9.9p1-2 (e.g. in Plucky)

   * Rejected upstream, due to being a bug in the Debian delta:
     https://bugzilla.mindrot.org/show_bug.cgi?id=3406

   * Fixed in Debian by Colin Watson:
     https://salsa.debian.org/ssh-team/openssh/-/commit/7d291bb

  === original bug report ===
  Since the upgrade from Ubuntu 20.04 to 22.04 the SSH login via a SSH pubkey to our servers fails, while password and kerberos are still working.

  $ssh user at server
  sign_and_send_pubkey: internal error: initial hostkey not recorded

  This seem related to the bugreport at openssh:
  https://bugzilla.mindrot.org/show_bug.cgi?id=3406

  ProblemType: Bug
  DistroRelease: Ubuntu 22.04
  Package: openssh-server 1:8.9p1-3ubuntu0.1
  ProcVersionSignature: Ubuntu 5.15.0-76.83-generic 5.15.99
  Uname: Linux 5.15.0-76-generic x86_64
  ApportVersion: 2.20.11-0ubuntu82.5
  Architecture: amd64
  CasperMD5CheckResult: pass
  CloudArchitecture: x86_64
  CloudID: none
  CloudName: none
  CloudPlatform: none
  CloudSubPlatform: config
  Date: Thu Jul 20 17:25:01 2023
  InstallationDate: Installed on 2020-08-24 (1060 days ago)
  InstallationMedia: Ubuntu-Server 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
  SourcePackage: openssh
  UpgradeStatus: Upgraded to jammy on 2023-07-20 (0 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2028282/+subscriptions