[Bug 1915345] Re: [SRU] eic_harvest_hostkeys fails in local zones

[Balint Reczey]

** Description changed:

+ [Impact]
+ 
+ * ec2-instance-connect breaks during host key harvesting for instances
+ launched in local zones [1] making the system boot to degraded mode
+ only.
+ 
+ [Test Plan]
+ 
+ * Start a system with the the fixed ec2-instance-connect package in a
+ local zone [1] or break the the /usr/share/ec2-instance-
+ connect/eic_harvest_hostkeys script to exit with failure.
+ 
+ [Where problems could occur]
+ 
+ * The fix is ignoring the eic_harvest_hostkeys script's exit code which
+ may hide actual problems in the script or in the infrastructure
+ preventing connecting to the instance using Instance Connect. This is a
+ decision by upstream. There are no other expected issues.
+ 
+ [Original Bug Text]
+ 
  ec2-instance-connect breaks during host key harvesting for instances
  launched in local zones[1].  Here are is the relevant debug data:
  
- $ systemctl is-system-running 
+ $ systemctl is-system-running
  degraded
  
  $ systemctl list-units --failed
-   UNIT                         LOAD   ACTIVE SUB    DESCRIPTION                             
+   UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
  ● ec2-instance-connect.service loaded failed failed EC2 Instance Connect Host Key Harvesting
- 
  
  $ journalctl --unit ec2-instance-connect
  -- Logs begin at Wed 2021-02-10 22:47:47 UTC, end at Wed 2021-02-10 22:55:46 UTC. --
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Starting EC2 Instance Connect Host Key Harvesting...
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Main process exited, code=exited, status=255/EXCEPTION
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Failed with result 'exit-code'.
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Failed to start EC2 Instance Connect Host Key Harvesting.
  
  $ dpkg-query -l ec2-instance-connect
  ii  ec2-instance-connect 1.1.13-0ubuntu1 all          Configures ssh daemon to accept EC2 Instance Connect ssh keys
  
- 
  $ lsb_release -c
  Codename:	hirsute
  
- $ cat /etc/cloud/build.info 
+ $ cat /etc/cloud/build.info
  build_name: server
  serial: 20210208
  
  $ ec2metadata --availability-zone --ami-id
  us-west-2-lax-1a
  ami-098f71a7a25a0f1f2
- 
  
  $ bash -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys
  ...
  ++ /usr/bin/curl -s -f -m 1 -H 'X-aws-ec2-metadata-token: AQAEAEvStI0Ugwz1C3GQh7oubFTah7bXQllCmFU6BtMI6b6l5zMkVQ==' http://169.254.169.254/latest/meta-data/placement/availability-zone/
  + zone=us-west-2-lax-1a
  + zone_exit=0
  + '[' 0 -ne 0 ']'
  + /bin/echo us-west-2-lax-1a
  + /bin/grep -Eq '^([a-z]+-){2,3}[0-9][a-z]$'
  + /usr/bin/head -n 1
  + exit 255
  
  [1] https://aws.amazon.com/about-aws/global-infrastructure/localzones/

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ec2-instance-connect in Ubuntu.
https://bugs.launchpad.net/bugs/1915345

Title:
  [SRU] eic_harvest_hostkeys fails in local zones

Status in Ec2 Instance Connect:
  New
Status in ec2-instance-connect package in Ubuntu:
  Fix Released

Bug description:
  [Impact]

  * ec2-instance-connect breaks during host key harvesting for instances
  launched in local zones [1] making the system boot to degraded mode
  only.

  [Test Plan]

  * Start a system with the the fixed ec2-instance-connect package in a
  local zone [1] or break the the /usr/share/ec2-instance-
  connect/eic_harvest_hostkeys script to exit with failure.

  [Where problems could occur]

  * The fix is ignoring the eic_harvest_hostkeys script's exit code
  which may hide actual problems in the script or in the infrastructure
  preventing connecting to the instance using Instance Connect. This is
  a decision by upstream. There are no other expected issues.

  [Original Bug Text]

  ec2-instance-connect breaks during host key harvesting for instances
  launched in local zones[1].  Here are is the relevant debug data:

  $ systemctl is-system-running
  degraded

  $ systemctl list-units --failed
    UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
  ● ec2-instance-connect.service loaded failed failed EC2 Instance Connect Host Key Harvesting

  $ journalctl --unit ec2-instance-connect
  -- Logs begin at Wed 2021-02-10 22:47:47 UTC, end at Wed 2021-02-10 22:55:46 UTC. --
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Starting EC2 Instance Connect Host Key Harvesting...
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Main process exited, code=exited, status=255/EXCEPTION
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Failed with result 'exit-code'.
  Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Failed to start EC2 Instance Connect Host Key Harvesting.

  $ dpkg-query -l ec2-instance-connect
  ii  ec2-instance-connect 1.1.13-0ubuntu1 all          Configures ssh daemon to accept EC2 Instance Connect ssh keys

  $ lsb_release -c
  Codename:	hirsute

  $ cat /etc/cloud/build.info
  build_name: server
  serial: 20210208

  $ ec2metadata --availability-zone --ami-id
  us-west-2-lax-1a
  ami-098f71a7a25a0f1f2

  $ bash -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys
  ...
  ++ /usr/bin/curl -s -f -m 1 -H 'X-aws-ec2-metadata-token: AQAEAEvStI0Ugwz1C3GQh7oubFTah7bXQllCmFU6BtMI6b6l5zMkVQ==' http://169.254.169.254/latest/meta-data/placement/availability-zone/
  + zone=us-west-2-lax-1a
  + zone_exit=0
  + '[' 0 -ne 0 ']'
  + /bin/echo us-west-2-lax-1a
  + /bin/grep -Eq '^([a-z]+-){2,3}[0-9][a-z]$'
  + /usr/bin/head -n 1
  + exit 255

  [1] https://aws.amazon.com/about-aws/global-infrastructure/localzones/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ec2-instance-connect/+bug/1915345/+subscriptions