More failing tests...

[William Reade]

On Thu, 2013-02-28 at 21:11 +1000, Ian Booth wrote:
> I do normally but may have not have done so in this case. This is the first
> project I have worked on in years without a bot of some sort guarding trunk
> commits so sometimes my muscle memory reverts to old habits in relying on the
> bot to catch issues after local test runs. Despite best intentions, we will make
> mistakes, plus it is crackful to expect our individual setups to be anyway a
> reliable way to ensure passing tests = quality. The final tests before trunk
> commit need to be done on a controlled environment. We gotta get tarmac set up.

Points well made and well taken; any process that depends on human
infallibility is unhelpfully optimistic.

So, I am almost entirely +1 on getting tarmac set up, for the following
additional reason: it will allow us to pull all the dependencies into
the main source tree (and hence allow us to know what we're actually
building) and still develop somewhat sanely (running only the subset of
tests that are likely to fail, rather than every test for every dep)
because we'll have an implacable and unfoolable backstop in place.

The only issue is that... intermittent failures still happen, and *they*
will mess with tarmac. Right? Would someone with experience chime here?

> > If you're lucky enough to be able to reproduce an intermittent failure
> > at a decent rate, please step up and try to actually fix it: we all have
> > subtle differences in hardware/OS, and if you're suffering excessively
> > from one particular failure you're probably the best person to deal with
> > it. If the issue is unclear, ofc, please seek guidance from the most
> > likely responsible individual.
> 
> Sadly they have been intermittent rather than regular. Where test failures do
> become more regular (like the Goose test double server id issue), we do fix them.

Yeah, I assigned an intermittent uniter.Filter test failure to myself
this morning -- it is surely my fault -- and it's been running perfectly
for several hours now. I don't know whether I'm just (un)lucky, or
whether it's actually a spurious failure that's only caused, rarely,
when... I dunno, when someone has a different version of mgo to me,
maybe? But I can't know this for sure; so deps-in-source, plus tarmac,
feels all the more necessary to me.

That said, I think that we do still need to be individually proactive re
intermittent failures -- every one is a real problem, either with the
tests or with the code, even if only one person ever sees it; and every
one of us should be trying to fix the ones we do see. If we don't
consciously apply pressure against the tide it will overwhelm us.

If I'm making sense here, could some kind reader be convinced to
volunteer to do the tarmac/deps work?

Cheers
William