Automate testing from -proposed

Wed Aug 4 15:01:33 UTC 2010

On lun, 2010-08-02 at 07:27 -0700, Brad Figg wrote:
> On 07/30/2010 09:46 AM, Leann Ogasawara wrote:
> > Hi All,
> >
> > A discussion regarding testing packages in the -proposed pocket has
> > recently come up.  Obviously our own personal interest lies with
testing
> > of the kernels which land in -proposed before they are promoted to
> > -updates.  Below is an initial summary of requirements that need to
be
> > resolved as well as some follow up questions which have been raised.
> >
> > Initial list of requirements from QA Team (ie Mathieu
Trudel-Lapierre):
> >
> >   * an infrastructure to support adding these special tasks to the
> > installation preseed, in other words what will add the repository
and
> > install/upgrade packages (mostly done)
> >   * a tracker to kick-start installations once new packages are
found in
> > -proposed
> >   * a way to display the fact, on the certification website, that
testing
> > was done on a release image (or point-release image), with
additional
> > packages from proposed, and *which* additional packages
> >
> > So far, the tracker and display changes have not been started yet.
This
> > is all development on the hardware certification side of things.
> >
> > Our comments/concerns from a kernel point of view are as follows:
> >   * Are the above requirements blocking current testing of packages
in
> > -proposed?  It seems QA could/should be testing every kernel
uploaded to
> > -proposed already, even if it needs to be done manually until the
> > automation is in place.

It's certainly making the whole process very complex and labor
intensive. Since test results currently end up in the certification
website, we'll at least need to get those out and build a report. That's
only for the reporting, as we still need to start installs of systems
and the upgrade on them to what's in -proposed.

On that subject, is testing -proposed once a day a reasonable way of
taking care of the kernel uploads to -proposed and any other packages?
That means there is much less development to be done for automation,
especially if we just test "everything in proposed" once a day, and
provides, I think, reasonable coverage.

I already started running tests daily (since yesterday) on some systems
in the Montreal lab (only laptops). It's not ideal but we also need to
manually test some of the systems with 10.04.1 images, and run automatic
tests for Alpha 3.

> >   * How long is it going to take for the infrastructure to be
completed
> > to an extent where you/we can start using it?

As above, I can already use the scripts necessary to start installations
and the
upgrade of these installations to the full contents of -proposed.

> >   * When will QA start on the tracker / display changes and what is
an
> > estimate for having them completed?

I've started to look at how we could check the proposed component for
new
packages, but as above, if we just test "everything in proposed", it's
not necessary. Using a cron job instead takes 5 minutes to implement.

As for displaying results, once I get them in
certification.canonical.com and displaying differently than a standard
install of 10.04 or 10.04.1, then I can start to gather these results in
a report to be shared to everyone. I'm expecting this to maybe take a
week, so I can be sure the results can really be differentiated on
certification.c.c, so that I can mine them from the database to build
the report that would go on qa.u.c. 

[...]
> >   * Where can we expect to view test results and what permissions
will we
> > need to access the results?
> >

My take on this is the results will be available in an html report on
qa.ubuntu.com, as discussed at the Platform sprint.

[...]
> 
> I think we (the kernel team) would like to get a specific date from QA
> when this testing will start. The results could be email posted to
this
> list or something else. We just want this to get going as soon as
> possible.

I think we can start testing as soon as the requirements are agreed on,
and once we know exactly what we will be testing.

Please keep in mind that testing of -proposed includes kernel uploads
but also any other packages, and needs to play nicely along with the
"normal" testing we run daily, so testing of the daily ISOs, in both
desktop and alternate form for mostly all the Ubuntu flavours; as well
as testing of the daily point release ISOs.

Does this all make sense/ seem acceptable?

-- 
Mathieu Trudel-Lapierre - mathieu.trudel-lapierre at canonical.com
Freenode: cyphermox, Jabber: mathieu.tl at gmail.com
4096R/EE018C93 1967 8F7D 03A1 8F38 732E  FF82 C126 33E1 EE01 8C93
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20100804/91b8a28c/attachment.sig>