[Bug 1724623] Re: Update ubuntu cloud info

Chad Smith 1724623 at bugs.launchpad.net
Thu Jun 29 19:42:17 UTC 2023 

SRU Verification script

** Description changed:

  [ Impact ]
  Apport reported bug add invalid bug tags such as `uec-images` which no longer has meaning or `ec2-images` on openstack. Since cloud-init is installed in all these images and detects the correct datasource, leverage cloud-init's instance-data.json or cloud-id
  
  [ Test Plan ]
  
  1. Launch LXD container for each series
  2. upgrade to cloud-init from -proposed.
  3. Execute apport-bug cloud-init and view report to assess that correct tags are present for LXD datasource.
  4. replace /run/cloud-init/instance-data.json cloud-id content with examples from openstack, ec2, configdrive and assert appropriate tags match the platform details.
- 5. Assert no stray uec-images or invalid ec2-images tags on openstack.
  
- scriptlet TBD:
+ verification scriptlet (attached):
+ 
+ #!/bin/bash
+ set -ex
+ 
+ cat > setup_proposed.sh <<EOF
+ #/bin/bash
+ mirror=http://archive.ubuntu.com/ubuntu
+ echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
+ apt-get update -q
+ apt-get install -qy cloud-init
+ EOF
+ 
  
  cat > openstack.json <<EOF
  {
-   "v1": {
-         "cloud_id": "openstack",
-         "cloud_name": "openstack",
-         "machine": "CloudArchitecture",
-         "platform": "openstack",
-         "region": null,
-         "subplatform": "metadata (http://169.254.169.254)",
-   }
+   "ds" : {},
+   "v1": {
+         "cloud_id": "openstack",
+         "cloud_name": "openstack",
+         "machine": "CloudArchitecture",
+         "platform": "openstack",
+         "region": null,
+         "subplatform": "metadata (http://169.254.169.254)"
+   }
  }
  EOF
  
  cat > ec2.json << EOF
  {
-   "v1": {
-     "cloud_name": "aws",
-     "distro": "ubuntu",
-     "distro_release": "jammy",
-     "distro_version": "22.04",
-     "instance_id": "i-06b5687b4d7b8595d",
-     "machine": "x86_64",
-     "platform": "ec2",
-     "python_version": "3.10.4",
-     "region": "us-east-2",
-     "variant": "ubuntu"
-   }
+   "ds": {
+     "imageId": "ami-123",
+     "instanceType": "m1.tiny",
+     "region": "us-east-1"
+   },
+   "v1": {
+     "cloud_name": "aws",
+     "distro": "ubuntu",
+     "distro_release": "jammy",
+     "distro_version": "22.04",
+     "instance_id": "i-06b5687b4d7b8595d",
+     "machine": "x86_64",
+     "platform": "ec2",
+     "python_version": "3.10.4",
+     "region": "us-east-2",
+     "variant": "ubuntu"
+   }
  }
  EOF
  
  for release in focal jammy kinetic lunar; do
-  lxc launch ubuntu-daily:$release sru-$release
-  lxc exec sru-f -- cloud-init status --wait --long
-  lxc exec sru-f -- apport-bug cloud-init
-  # step through prompts and select view/save report
-  # inspect tags to assert cloud platform is proper tags where value is non-empty/null
-         CloudID, CloudName, CloudArchitecture CloudPlatform, CloudRegion, CloudSubPlatform
+  VM=sru-$release
+  lxc launch ubuntu-daily:$release $VM
+  while ! lxc exec $VM -- cloud-init status --wait --long; do
+     sleep 5
+  done
+  echo --- 1. Generate current apport report, selecting Openstack as cloud.
+  echo --- step through prompts and select 'K' to keep report
+  lxc exec $VM -- apport-bug cloud-init
+  APPORT_FILE=$(lxc exec $VM ls /tmp | grep apport)
+  lxc file pull $VM/tmp/$APPORT_FILE apport-$VM.orig
+  lxc exec $VM rm /tmp/$APPORT_FILE
  
-  echo ### validate openstack apport-bug
-  lxc file push openstack.json dev-$release/run/cloud-init/instance-data.json
-  lxc exec sru-f -- apport-bug cloud-init
-  # step through prompts and select view/save report
-  # inspect tags to assert cloud platform is proper tags where value is non-empty/null
-         CloudID, CloudName, CloudArchitecture CloudPlatform, CloudRegion,
+  lxc file push setup_proposed.sh $VM/
+  lxc exec $VM -- bash /setup_proposed.sh | grep cloud-init
  
-  echo ### validate ec2 apport-bug
-  lxc file push ec2.json dev-$release/run/cloud-init/instance-data.json
-  lxc exec sru-f -- apport-bug cloud-init
-  # step through prompts and select view/save report
-  # inspect tags to assert cloud platform is proper tags where value is non-empty/null
-         CloudID, CloudName, CloudArchitecture CloudPlatform, CloudRegion,
+  lxc file push openstack.json $VM/run/cloud-init/instance-data.json
+  echo --- 2. Generate -proposed apport report which sources openstack instance-data.json, selecting Openstack as cloud.
+  lxc exec $VM -- apport-bug cloud-init
+  APPORT_FILE=$(lxc exec $VM ls /tmp | grep apport)
+  lxc file pull $VM/tmp/$APPORT_FILE apport-$VM.openstack-proposed
+  lxc exec $VM rm /tmp/$APPORT_FILE
+ 
+  echo --- 3. Generate -proposed apport report which sources ec2 instance-data, selecting Ec2 as cloud.
+  lxc file push ec2.json $VM/run/cloud-init/instance-data.json
+  lxc exec $VM -- apport-bug cloud-init
+  APPORT_FILE=$(lxc exec $VM ls /tmp | grep apport)
+  lxc file pull $VM/tmp/$APPORT_FILE apport-$VM.ec2-proposed
+  lxc exec $VM rm /tmp/$APPORT_FILE
+ 
+  # redact logs lines for easy diffs
+  for file in `ls apport*`; do
+      sed -i '1,/logs.tgz/!d' $file
+  done
+  echo --- 4. Inspect diff tags of orig to openstack-proposed report
+  diff -urN apport-$VM.orig apport-$VM.openstack-proposed || true
+  echo --- 5. Inspect diff tags of openstack-proposed to ec2-proposed report
+  diff -urN apport-$VM.openstack-proposed apport-$VM.ec2-proposed || true
  
  done
  
+ 
  [ Where problems could occur ]
- apport-bug could traceback on poor logic preventing simple bug filing at CLI. It could omit tags if unable to process cloud-init/instance-data.json. nothing critical to daily performance, uptime or security
+ apport-bug could traceback on poor logic preventing simple bug filing at apport CLI. It could omit tags if unable to process cloud-init/instance-data.json. nothing critical to daily performance, uptime or security
  
  [ Other Info ]
+ 
  
  [Original description]
  
  Issues:
   - Using the presence of cloud-init to flag an image as a cloud image is incorrect now that ubuntu-server includes cloud-init (and ubuntu-core images)
   - Using the presence of EC2 metadata source is incorrect as many non-EC2 clouds provide EC2 metadata.  Thus we have bugs like bug #1722946 that are tagged as an 'ec2-images' bug which are clearly on openstack
   - Marking all bugs that have cloud-init but no EC2 metadata source as an 'uec-images' bug uses a name that no longer has meaning.
  
  Solution:
   - If cloud-init is present, check for /etc/cloud/build.info to indicate an Ubuntu cloud images, tag as 'cloud-images'.  Pull the build_name and serial from that file into the bug comments.
   - If cloud-init is present, check for the presence of /run/cloud-init/cloud.cfg.  Parse this yaml to determine the datasource type.  Add the datasource used to the bug comment.
  
  I have filed bug #1724626 to ask cloud-init development to surface more
  information from ds-identify to help ID the cloud so that we can better
  tag/annotate the bug.  There may also be some info we can get to
  indicate the image ID on more clouds than just AWS.  At a minimum I
  would like to see dsidentify make the EC2 platform it found available
  for consumers in cloud.cfg.  This would allow us to identify AWS EC2
  from look-alike datasources so that we can tag a bug as ec2-images for
  bug really on AWS and add EC2 specific fields to the
  description/attachments.

** Attachment added: "sru-2025376-verification.sh"
   https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1724623/+attachment/5682929/+files/sru-2025376-verification.sh

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apport in Ubuntu.
https://bugs.launchpad.net/bugs/1724623

Title:
  Update ubuntu cloud info

Status in Apport:
  Fix Released
Status in apport package in Ubuntu:
  Fix Released
Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Focal:
  Fix Committed
Status in cloud-init source package in Jammy:
  Fix Committed
Status in cloud-init source package in Kinetic:
  Fix Committed

Bug description:
  [ Impact ]
  Apport reported bug add invalid bug tags such as `uec-images` which no longer has meaning or `ec2-images` on openstack. Since cloud-init is installed in all these images and detects the correct datasource, leverage cloud-init's instance-data.json or cloud-id

  [ Test Plan ]

  1. Launch LXD container for each series
  2. upgrade to cloud-init from -proposed.
  3. Execute apport-bug cloud-init and view report to assess that correct tags are present for LXD datasource.
  4. replace /run/cloud-init/instance-data.json cloud-id content with examples from openstack, ec2, configdrive and assert appropriate tags match the platform details.

  verification scriptlet (attached):

  #!/bin/bash
  set -ex

  cat > setup_proposed.sh <<EOF
  #/bin/bash
  mirror=http://archive.ubuntu.com/ubuntu
  echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
  apt-get update -q
  apt-get install -qy cloud-init
  EOF

  
  cat > openstack.json <<EOF
  {
    "ds" : {},
    "v1": {
          "cloud_id": "openstack",
          "cloud_name": "openstack",
          "machine": "CloudArchitecture",
          "platform": "openstack",
          "region": null,
          "subplatform": "metadata (http://169.254.169.254)"
    }
  }
  EOF

  cat > ec2.json << EOF
  {
    "ds": {
      "imageId": "ami-123",
      "instanceType": "m1.tiny",
      "region": "us-east-1"
    },
    "v1": {
      "cloud_name": "aws",
      "distro": "ubuntu",
      "distro_release": "jammy",
      "distro_version": "22.04",
      "instance_id": "i-06b5687b4d7b8595d",
      "machine": "x86_64",
      "platform": "ec2",
      "python_version": "3.10.4",
      "region": "us-east-2",
      "variant": "ubuntu"
    }
  }
  EOF

  for release in focal jammy kinetic lunar; do
   VM=sru-$release
   lxc launch ubuntu-daily:$release $VM
   while ! lxc exec $VM -- cloud-init status --wait --long; do
      sleep 5
   done
   echo --- 1. Generate current apport report, selecting Openstack as cloud.
   echo --- step through prompts and select 'K' to keep report
   lxc exec $VM -- apport-bug cloud-init
   APPORT_FILE=$(lxc exec $VM ls /tmp | grep apport)
   lxc file pull $VM/tmp/$APPORT_FILE apport-$VM.orig
   lxc exec $VM rm /tmp/$APPORT_FILE

   lxc file push setup_proposed.sh $VM/
   lxc exec $VM -- bash /setup_proposed.sh | grep cloud-init

   lxc file push openstack.json $VM/run/cloud-init/instance-data.json
   echo --- 2. Generate -proposed apport report which sources openstack instance-data.json, selecting Openstack as cloud.
   lxc exec $VM -- apport-bug cloud-init
   APPORT_FILE=$(lxc exec $VM ls /tmp | grep apport)
   lxc file pull $VM/tmp/$APPORT_FILE apport-$VM.openstack-proposed
   lxc exec $VM rm /tmp/$APPORT_FILE

   echo --- 3. Generate -proposed apport report which sources ec2 instance-data, selecting Ec2 as cloud.
   lxc file push ec2.json $VM/run/cloud-init/instance-data.json
   lxc exec $VM -- apport-bug cloud-init
   APPORT_FILE=$(lxc exec $VM ls /tmp | grep apport)
   lxc file pull $VM/tmp/$APPORT_FILE apport-$VM.ec2-proposed
   lxc exec $VM rm /tmp/$APPORT_FILE

   # redact logs lines for easy diffs
   for file in `ls apport*`; do
       sed -i '1,/logs.tgz/!d' $file
   done
   echo --- 4. Inspect diff tags of orig to openstack-proposed report
   diff -urN apport-$VM.orig apport-$VM.openstack-proposed || true
   echo --- 5. Inspect diff tags of openstack-proposed to ec2-proposed report
   diff -urN apport-$VM.openstack-proposed apport-$VM.ec2-proposed || true

  done

  
  [ Where problems could occur ]
  apport-bug could traceback on poor logic preventing simple bug filing at apport CLI. It could omit tags if unable to process cloud-init/instance-data.json. nothing critical to daily performance, uptime or security

  [ Other Info ]

  
  [Original description]

  Issues:
   - Using the presence of cloud-init to flag an image as a cloud image is incorrect now that ubuntu-server includes cloud-init (and ubuntu-core images)
   - Using the presence of EC2 metadata source is incorrect as many non-EC2 clouds provide EC2 metadata.  Thus we have bugs like bug #1722946 that are tagged as an 'ec2-images' bug which are clearly on openstack
   - Marking all bugs that have cloud-init but no EC2 metadata source as an 'uec-images' bug uses a name that no longer has meaning.

  Solution:
   - If cloud-init is present, check for /etc/cloud/build.info to indicate an Ubuntu cloud images, tag as 'cloud-images'.  Pull the build_name and serial from that file into the bug comments.
   - If cloud-init is present, check for the presence of /run/cloud-init/cloud.cfg.  Parse this yaml to determine the datasource type.  Add the datasource used to the bug comment.

  I have filed bug #1724626 to ask cloud-init development to surface
  more information from ds-identify to help ID the cloud so that we can
  better tag/annotate the bug.  There may also be some info we can get
  to indicate the image ID on more clouds than just AWS.  At a minimum I
  would like to see dsidentify make the EC2 platform it found available
  for consumers in cloud.cfg.  This would allow us to identify AWS EC2
  from look-alike datasources so that we can tag a bug as ec2-images for
  bug really on AWS and add EC2 specific fields to the
  description/attachments.

To manage notifications about this bug go to:
https://bugs.launchpad.net/apport/+bug/1724623/+subscriptions

More information about the foundations-bugs mailing list