Subject: Ideas for improving the release process

From: Cemil Azizoglu <[email-masked]>
Date: Fri, 12 Feb 2016 11:08:23 -0600

Hi,

We talked about the release process and how it could be improved. Here are
some ideas. Please add if you have others.

(Jenkaas could be leveraged for some)

   1. Minimizing the manual steps (like creation of the next target branch
   on lp, etc) using scripts/launchpad API.
   2. 'make release' target that 'll check for ABI breakage, perhaps even
   prepopulate the changelog with some static info, etc.
   3. Downstreams' build/sanity testing could be done as part of MP
   autolanding to identify breaks.
   4. Downstreams' release testing. How useful are AP tests for U8
   and Browser? General opinion is 'not very'. Should we look into doing away
   with them? Or at least identify the subset of them that is relevant to Mir,
   and run them during every release as part of autolanding as a 'nonblocking'
   test.

As always I'll be creating a trello card for this.

Thanks
Cemil Azizoglu
Mir Display Server - Team Lead
Canonical USA

---

From: Alan Griffiths <[email-masked]>
Date: Fri, 12 Feb 2016 17:18:31 +0000

On 12/02/16 17:08, Cemil Azizoglu wrote:
> Hi,
>
> We talked about the release process and how it could be improved. Here
> are some ideas. Please add if you have others.
>
>  1. Minimizing the manual steps (like creation of the next target
>     branch on lp, etc) using scripts/launchpad API.
>  2. 'make release' target that 'll check for ABI breakage, perhaps
>     even prepopulate the changelog with some static info, etc.
>     and Browser? General opinion is 'not very'. Should we look into
>     doing away with them? Or at least identify the subset of them that
>     is relevant to Mir, and run them during every release as part of
>     autolanding as a 'nonblocking' test.
>

Essentially, we have to cater for two types of release:

1. release changes from (development) trunk to a brand new series; and,
2. cherry-picking from (development) trunk to an existing series (or,
very occasionally fixing on the series directly)

In theory, there should be a third one: releasing changes from
development to the current series - but our server ABI isn't
sufficiently stable to do that.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 16 Feb 2016 09:19:39 +0800

Someone should add(!) to the list of manual steps:

Check for open critical bugs. If a bug is critical it should block any 
release:
https://bugs.launchpad.net/mir/+bugs?search=Search&field.importance=Critical

---

From: Kevin Gunn <[email-masked]>
Date: Tue, 16 Feb 2016 14:09:15 -0600

inline comments

> https://bugs.launchpad.net/mir/+bugs?search=Search&field.importance=Critical
I disagree with this, but I'll guess that means we might disagree on how
Critical is used.
Critical means Fix now or as soon as possible.
which to me means "this is one of the most important thing"...you could
have 2 criticals, you fix one, you'd certainly want to release that even if
you were stumped on the 2nd.
Also, you might be assuming the bug only exists on devel.

That's not to say you'd never have a bug that would "block"...it's more
case by case.
So I would agree with wording like
"Check for open critical bugs. Determine if a bug is a blocker (e.g. it's
_only_ on devel and would be catastrophic to release, or is already in
release and needs fixing ASAP):"

>>     autolanding to identify breaks.
where exactly - autolanding on devel?

>>     autolanding as a 'nonblocking' test.
I am not opposed to pruning down the list because it is a time killer.
however, one reason we are running these AP tests is due to the fact we've
broken them before with Mir releases. I would suggest if we want to prune
the tests, to engage with unity8 & web browser team to work up a list of
the individual AP tests we should run.

> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel
>

---

From: Cemil Azizoglu <[email-masked]>
Date: Tue, 16 Feb 2016 16:23:10 -0600

> where exactly - autolanding on devel?

​
Yes
​

> the individual AP tests we should run.

It's good to learn that they did at some point fail as part of the release.
​
That means there are some tests that are relevant.
Yes we will go through them and identify those and still use them.

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 17 Feb 2016 09:08:12 +0800

If a critical bug isn't blocking a release I guess it should be demoted 
to High.

We need some simple threshold that doesn't require the reader to 
understand the details of each bug. Just that "if importance >= critical 
then don't release". And there's no other level we can use for that 
other than critical (or 'high' later as the project matures).

On 17/02/16 04:09, Kevin Gunn wrote:
> inline comments
>
> On Mon, Feb 15, 2016 at 7:19 PM, Daniel van Vugt
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>
>     Check for open critical bugs. If a bug is critical it should block
>     any release:
> have 2 criticals, you fix one, you'd certainly want to release that even
> if you were stumped on the 2nd.
> "Check for open critical bugs. Determine if a bug is a blocker (e.g.
> it's _only_ on devel and would be catastrophic to release, or is already
> in release and needs fixing ASAP):"
>
>
> however, one reason we are running these AP tests is due to the fact
> we've broken them before with Mir releases. I would suggest if we want
> to prune the tests, to engage with unity8 & web browser team to work up
> a list of the individual AP tests we should run.
>
>     [email-masked] <mailto:[email-masked]>

---

From: "Stephen M. Webb" <[email-masked]>
Date: Tue, 16 Feb 2016 21:59:43 -0500

On 16-02-16 08:08 PM, Daniel van Vugt wrote:
> If a critical bug isn't blocking a release I guess it should be demoted to High.
> 
> We need some simple threshold that doesn't require the reader to understand the details of each bug. Just that "if
> importance >= critical then don't release". And there's no other level we can use for that other than critical (or
> 'high' later as the project matures).

I have to agree with this.  If the bug is not critical enough to block a release, it shouldn't be classed as critical.

-- 
Stephen M. Webb  <[email-masked]>

---

From: Kevin Gunn <[email-masked]>
Date: Wed, 17 Feb 2016 07:48:45 -0600

So are you both tied to the idea of using Critical/High for this?
as the lp definition/intimations for Critical/High don't really match the
"blocker" use, would you be ok with using a tag "blocker" instead?
can't see how a tag is any more costly and gets the job done all the same.

ultimately the call is Stephen's.
If you choose the Critical/High....just needs to be a) outlined somewhere &
b) added to the release instructions like duflu originally asked.

br,kg

On Tue, Feb 16, 2016 at 8:59 PM, Stephen M. Webb <[email-masked]

> understand the details of each bug. Just that "if
> > importance >= critical then don't release". And there's no other level
> we can use for that other than critical (or
> I have to agree with this.  If the bug is not critical enough to block a
> release, it shouldn't be classed as critical.
>
> --

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 18 Feb 2016 09:23:15 +0800

It's less of an issue if we also remember to review release branches 
before top approving them. In that case, hopefully code reviewers will 
also check for open critical bugs and mention them.

But yeah I feel we're not utilizing the importance field if we're 
allowing critical bugs to remain open without anyone even checking on 
them during the release process.

BTW, I demoted the remaining ones to High yesterday so all criticals are 
now resolved :)

On 17/02/16 21:48, Kevin Gunn wrote:
> as the lp definition/intimations for Critical/High don't really match
> the "blocker" use, would you be ok with using a tag "blocker" instead?
> If you choose the Critical/High....just needs to be a) outlined
> somewhere & b) added to the release instructions like duflu originally
> asked.
>
> br,kg
>
>
> On Tue, Feb 16, 2016 at 8:59 PM, Stephen M. Webb
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     I have to agree with this.  If the bug is not critical enough to
>     block a release, it shouldn't be classed as critical.
>
>     --
>     Stephen M. Webb  <[email-masked]
>     <mailto:[email-masked]>>
>
>     --
>     Mir-devel mailing list

---

From: Kevin Gunn <[email-masked]>
Date: Wed, 17 Feb 2016 19:47:00 -0600

the interesting thing is when you create a new vernacular within a team -
the rest of the world thinks you're saying Schedule to fix soon.
instead of
Fix now or as soon as possible.
and the rest of the world has no way to tell you Fix now or as soon as
possible.

On Wed, Feb 17, 2016 at 7:23 PM, Daniel van Vugt <
[email-masked]> wrote:

> before top approving them. In that case, hopefully code reviewers will also
> check for open critical bugs and mention them.
>
> But yeah I feel we're not utilizing the importance field if we're allowing
> critical bugs to remain open without anyone even checking on them during
> the release process.
>
>>     > importance >= critical then don't release". And there's no other
>> level we can use for that other than critical (or

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 18 Feb 2016 09:56:17 +0800

We don't need a new definition. Just that if there are open critical 
bugs, it would be a good idea to familiarise ourselves with them before 
proceeding with the release. Because someone has marked it as critical 
for some reason.

Tangentially I'm reminded of the wider Ubuntu problem with "Fix 
Released" not being what new users would expect. We mark bugs as "Fix 
Released" in Ubuntu even if the Ubuntu series the fix is "released" in 
is 6 months away from being actually released. But that's a limitation 
of Launchpad right now.

On 18/02/16 09:47, Kevin Gunn wrote:
> the interesting thing is when you create a new vernacular within a team
> - the rest of the world thinks you're saying Schedule to fix soon.
> instead of
> On Wed, Feb 17, 2016 at 7:23 PM, Daniel van Vugt
>     before top approving them. In that case, hopefully code reviewers
>     will also check for open critical bugs and mention them.
>
>     allowing critical bugs to remain open without anyone even checking
>     on them during the release process.
>
>     BTW, I demoted the remaining ones to High yesterday so all criticals
>     are now resolved :)
>
>
>         <[email-masked] <mailto:[email-masked]>
>         <mailto:[email-masked]
>         <mailto:[email-masked]>>> wrote:
>
>              > We need some simple threshold that doesn't require the
>         reader to understand the details of each bug. Just that "if
>              > importance >= critical then don't release". And there's
>         no other level we can use for that other than critical (or
>         <mailto:[email-masked]>
>              <mailto:[email-masked]
>         <mailto:[email-masked]>>>
>
>              --
>              Mir-devel mailing list
>         <mailto:[email-masked]
>         <mailto:[email-masked]>>