RFC: blackbox tests are not integration tests

[Robert Collins]

Martin and I were chatting about the test duration thread, and we
realised while we were disagreeing, actually we weren't disagreeing :).
Rather the issue is that we have a big bag of tests 'blackbox' which are
sometimes UI tests, sometimes unit-tests-at-a-distance, sometimes
acceptance tests [where for some exceptional case we care about the
entire behaviour of the system] and sometimes integration tests [where
we are testing that A is glued to B correctly].

Being clearer about that would make it easier for folk that want to
cleanup a test they are touching to do that cleanup.

Thus, I propose 3 new test types (and following our current conventions,
that means 3 new subdirs):
bzrlib.tests.acceptance
bzrlib.tests.integration
bzrlib.tests.cli

The goal would be to end up with no blackbox tests, all tests being
tuned to be more aligned with their actual intent and placed into one of
those 3 containers, *or* moved to be a much more precise test.

The acceptance test suites should be the smallest of our suites: they
are intrinsically resistant to optimisations. 

The integration suite should be relatively cheap, though I suspect that
many such tests would be written in the style of acceptance tests - so
we don't want them added willynilly.

The cli tests *today* have no better way of being written; but if we
decide that a given test is testing the cli, it allows someone to make
them radically faster [e.g. by introducing test fakes or a more concrete
UI layer] without concern about dropping some intended aspect of the
test.

Thoughts?

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20090917/e5ba34f0/attachment-0002.pgp