[Bug 1915345] Re: [SRU] eic_harvest_hostkeys fails in local zones

Fri Apr 9 12:22:19 UTC 2021

I have tested in AWS local zone us-west-2-lax-1a

xenial:
 image: ami-008b09448b998a562
 build serial: 20201014
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3~16.04.2

bionic: 
 image: ami-02701bcdc5509e57b
 build serial: 20210224
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3~18.04.2

focal:
 image: ami-0ca5c3bd5a268e7db
 build serial: 20210223
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3.20.04.1

groovy:
 image: ami-0c1204e0c5e73ef4c
 build serial: 20210325
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3.20.10.1

Are you sure this is right?  Yes, the systemd unit no longer fails
because the patch ignored the script failure, but does it *work*?  The
package update has no changes to /usr/share/ec2-instance-
connect/eic_harvest_hostkeys to match the string format for a local zone
still.  So while the feature is available in local and wavelength zones
the package in -proposed fails to address the underlying failure and so
ec2-instance-connect still broken in those zones (just silently now).

Again, here is the failure:

$ sudo sh -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys 2>&1 | tail -9
+ /usr/bin/curl -s -f -m 1 -H X-aws-ec2-metadata-token: AQAEAF6AxckVUQFPqe3ivPjLa0b7dlvf4To2TaAReHD-lMpqgvuXBQ== http://169.254.169.254/latest/meta-data/placement/availability-zone/
+ zone=us-west-2-lax-1b
+ zone_exit=0
+ [ 0 -ne 0 ]
+ /bin/echo us-west-2-lax-1b
+ /usr/bin/head -n 1
+ /bin/grep -Eq ^([a-z]+-){2,3}[0-9][a-z]$
+ exit 255
+ rm -rf /dev/shm/eic-hostkey-WZBt1Vck

Please look at the grep on line 101 of the script:
 # Validate the zone
 /bin/echo "${zone}" | /usr/bin/head -n 1 | /bin/grep -Eq "^([a-z]+-){2,3}[0-9][a-z]$" || exit 255

The script needs to handle matches to the existing regex, but also local
zones like 'us-west-2-lax-1b' and wavelength zones like 'us-west-2-wl1
-den-wlz-1'

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ec2-instance-connect in Ubuntu.
https://bugs.launchpad.net/bugs/1915345

Title:
  [SRU] eic_harvest_hostkeys fails in local zones

Status in Ec2 Instance Connect:
  New
Status in ec2-instance-connect package in Ubuntu:
  Fix Released
Status in ec2-instance-connect source package in Xenial:
  Fix Committed
Status in ec2-instance-connect source package in Bionic:
  Fix Committed
Status in ec2-instance-connect source package in Focal:
  Fix Committed
Status in ec2-instance-connect source package in Groovy:
  Fix Committed

Bug description:
  [Impact]

  * ec2-instance-connect breaks during host key harvesting for instances
  launched in local zones [1] making the system boot to degraded mode
  only.

  [Test Plan]

  * Start a system with the the fixed ec2-instance-connect package in a
  local zone [1] or break the the /usr/share/ec2-instance-
  connect/eic_harvest_hostkeys script to exit with failure.

  [Where problems could occur]

  * The fix is ignoring the eic_harvest_hostkeys script's exit code
  which may hide actual problems in the script or in the infrastructure
  preventing connecting to the instance using Instance Connect. This is
  a decision by upstream. There are no other expected issues.

  [Original Bug Text]

  ec2-instance-connect breaks during host key harvesting for instances
  launched in local zones[1].  Here are is the relevant debug data:

  $ systemctl is-system-running
  degraded

  $ systemctl list-units --failed
    UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
  ● ec2-instance-connect.service loaded failed failed EC2 Instance Connect Host Key Harvesting

  $ journalctl --unit ec2-instance-connect
  -- Logs begin at Wed 2021-02-10 22:47:47 UTC, end at Wed 2021-02-10 22:55:46 UTC. --
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Starting EC2 Instance Connect Host Key Harvesting...
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Main process exited, code=exited, status=255/EXCEPTION
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Failed with result 'exit-code'.
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Failed to start EC2 Instance Connect Host Key Harvesting.

  $ dpkg-query -l ec2-instance-connect
  ii  ec2-instance-connect 1.1.13-0ubuntu1 all          Configures ssh daemon to accept EC2 Instance Connect ssh keys

  $ lsb_release -c
  Codename:	hirsute

  $ cat /etc/cloud/build.info
  build_name: server
  serial: 20210208

  $ ec2metadata --availability-zone --ami-id
  us-west-2-lax-1a
  ami-098f71a7a25a0f1f2

  $ bash -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys
  ...
  ++ /usr/bin/curl -s -f -m 1 -H 'X-aws-ec2-metadata-token: AQAEAEvStI0Ugwz1C3GQh7oubFTah7bXQllCmFU6BtMI6b6l5zMkVQ==' http://169.254.169.254/latest/meta-data/placement/availability-zone/
  + zone=us-west-2-lax-1a
  + zone_exit=0
  + '[' 0 -ne 0 ']'
  + /bin/echo us-west-2-lax-1a
  + /bin/grep -Eq '^([a-z]+-){2,3}[0-9][a-z]$'
  + /usr/bin/head -n 1
  + exit 255

  [1] https://aws.amazon.com/about-aws/global-infrastructure/localzones/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ec2-instance-connect/+bug/1915345/+subscriptions