micro release exception for LibreOffice

[Bjoern Michaelsen]

Hi Martin, all,

On Tue, Jun 05, 2012 at 05:38:41PM +0200, Martin Pitt wrote:
> Bjoern Michaelsen [2012-06-01 12:07 +0200]:
> > I would like to apply for a micro release exception for LibreOffice.
> 
> In general I think it does make sense to use the new upstream
> microreleases, to benefit from their QA process and from the testing
> that happens in PPAs. However, I still have some concerns/questions:
> 
> Is there a written policy what kind of changes are permitted in
> stable upstream branches?

http://wiki.documentfoundation.org/Development/Branches

> How does that compare to our SRU policy?

Mostly comparable, if not stricter for MRU after x.y.0. Also note that most
developers keep themselves to master-only from around x.y.3-.4 onward anyway.
After that it is mostly distro packagers doing critical security/stability
backports.

> We did have more than one attempted LibO (and OO.o back then) SRU
> which never made it to -updates because the new microrelease
> introduced a regression (e. g. #919659, #623267).

https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/919659

As you can see from the bug description the bug only hits in a very specific
cornercase. I doubt there will ever be a testcase for such scenarios. Also most
of the time debugging the issue was spend on generating a reproduction
scenario. No test suite in the world will prevent you from that. Testing in a
ppa however might. That should have happened in this case, but we
(Ubuntu/Debian) were still on the old go-oo buildsystem and with split repos,
making changes slow and a pain.

Also note that on 3.4 there was merging back and forth done between master and
the release branch -- despite my protests of this being not a sane move. The
experience in the end made my opponents in that argument a bit wiser and
humbler: By now we commit only to master and then cherry-pick back to the
release branch (possibly requireing sign-offs) -- which would have been the
sane thing to do in the first place.

Finally note, that we didnt even evade that regression: We released Precise
with the regression in 3.5.3 and people are now waiting for a 3.5.4 SRU/MRE to
get the upstream fix. There is some irony hidden in there.

https://bugs.launchpad.net/ubuntu/+source/openoffice.org/+bug/623267

Thats 3.2 on go-oo -- which is a whole different world wrt. testing and release
planning. For example: the go-oo patch system mixed vendor patches and upstream
development making test result of upstream and other distros -- even debian --
irrelevant for judging what Ubuntu was about to release.
I would not assume that to be comparable at all.

> Can we do, or have
> there been, policy changes or test improvements to prevent these?

Yes, however there is no published nice writeup about these as it has been
interative development. Just one example are the release candidates -- rc1 has
been already two weeks in the wild when upstream releases:
http://wiki.documentfoundation.org/ReleasePlan/3.6
and rc2 has usually very few (<10) fixes after x.y.3. We will have an
additional week of rc2 in the ppa at upstream release, keep the same package
(modulo version name) for another week in -proposed. So by the time it comes
out of proposed:
- has been on both the master and release branch that developers work with
  everyday for more than 2-3 weeks
- upstream rc1 has been in the wild  for 3 weeks
- upstream rc2 has been in the wild  for 2 weeks
- packaged rc2 has been in the ppa   for 2 weeks
- packaged rc2 has been in -proposed for 1 week

(rc2=final usually)

> How we make sure that the changes also work on ARM platforms?

Most ARM trouble is compile-time, not runtime. And we are running the in-build
unittest during the build of course. LibreOffice is abstracted to run on OSX,
Windows and Linux. Regressions that only happen on one platform are rare, those
on only on arch of one platform even rarer. There is never a riskfree change,
but even comparing for example the #919659 regression with the fixes that 3.4.5
provided:
http://wiki.documentfoundation.org/Releases/3.4.5_info_about_fixes
(including at least six crasher fixes, with each meaning possible data loss)
one has to assess both sides of the risk. While to much effort for 3.4, by now
we can revert one specific commit when it causes a regression and MRU the rest
hopefully.

> What is the main motivation to ask for a MRE?
> Are there usually/often too many changes, or fixes which do not have
> corresponding LP bugs to verify them individually?
Yes. 3.5.4 has 73 bugfixes (only 6 of those have LP bugs, 2 where backported to
Ubuntus 3.5.3 by me). Some of those might have a LP bugs that I have not mapped
to the upstream bug -- do so isnt always a clearcut case.

> Or do the stable branches get new features, UI changes, or invasive changes
> which carry a high risk of regressions?
No.

The motivation is mostly: I would like to use the testing that upstream
provides. Manually cherry-picking patches:
- given the volume opens a reasonable chance that something important is
  overlooked
- given the size and complexity of this >10Mio. LOC beast, IMHO we run higher
  risks introducing regressions by cherry-picking, which essentially means
  running our own (untested) branch and moves us again dangerously close to the
  go-oo nightmare explained above.

Best,

Bjoern