[Bug 1879350] Re: need uc20 grade=dangerous channel=beta images built

[Ian Johnson]

> So testing images built with grade=dangerous is not a reliable proxy
for assuring the correctness of the grade=signed images.

This is partially correct, insofar as yes there are some non-trivial
behavior changes between the grade==dangerous and grade==signed images,
however much of the behavior difference is not noticeable in the kinds
of tests that we want the cert team to be running automatically. For
instance here is as complete a list as I could find of the current list
of behavior changes:

* You cannot specify unasserted snaps in a grade > dangerous image when building the image
* You cannot specify extra snaps not in the model assertion when building the image
* Snapd will not automatically import cloud-init configuration files placed in the ubuntu-seed partition at runtime (which is how cert team is currently using cloud-init with grade==dangerous channel==edge image testing)

Over time, this list will grow (and indeed we have plans for it to
change between UC20 beta release and 1.0 release) and so you are correct
in that we should not in general be substituting test results for
grade==dangerous channel==beta for grade==signed channel==beta test
results, but considering that the difference right now is so minimal I
don't think there is a significant difference in behavior to justify not
testing this considering the alternatives detailed below.

> How are these going to be tested

Well right now AFAIK they are not being automatically tested at all,
which is the whole problem. We, snapd team, are working on getting
cloud-init support for grade > dangerous images but it is not here yet
as the problem is non-trivial. With this support in place the cert team
can automatically test grade==signed channel==beta images, but until
then they either test nothing automatically or they only test
grade==dangerous channel==beta images.

> in what way is automated testing of grade=dangerous images valuable
given the behavior is deliberately different from the production images?

Currently since the cert team is unable to test the grade==dangerous
images due to issues with snapd not supporting cloud-init yet in that
environment, they are testing grade==dangerous channel==edge images, and
this is unfortunate because there is a non-trivial diff between the set
of functionality on edge versus what's on beta, and the edge images
change daily and so it's been really unclear to them if they need to
keep re-running their tests every single day or which images are really
considered "release candidates" which doesn't make sense for images
produced from the edge channel. We really want them to be testing what's
on beta, not on edge, as they have never been tasked with testing
anything on edge up to this point, and honestly edge channels for all
snaps is likely too unstable for test results to be considered reliable
anyways.

Thanks,
Ian

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to livecd-rootfs in Ubuntu.
https://bugs.launchpad.net/bugs/1879350

Title:
  need uc20 grade=dangerous channel=beta images built

Status in Ubuntu CD Images:
  New
Status in livecd-rootfs package in Ubuntu:
  New

Bug description:
  We need the following things:

  1. canonical signed model assertions for a uc20 image with grade==dangerous, channel==beta for the various supported platforms, i.e. right now generic pc-kernel amd64 and armhf pi-kernel and arm64 pi-kernel models
  2. livcd-rootfs + cdimage builds of images for the above model assertions
  3. somehow these builds get published onto http://cdimage.ubuntu.com/ubuntu-core/20/dangerous-beta/ (or some other new directory there that does not conflict with the current set of dangerous, edge and beta)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-cdimage/+bug/1879350/+subscriptions