RFC: blackbox tests are not integration tests

[Robert Collins]

On Fri, 2009-09-18 at 08:55 +1000, Martin Pool wrote:
> 
> 
> But I think it would help if you can more clearly explain the
> difference in practice between acceptance and integration tests.  It
> looks like at the moment they'll look very similar.  That's not
> necessarily a problem, but do you have a science fiction of how they
> should look? 

A good example of acceptance tests are the smart server verb counting
tests: They are measuring something that is only valid when the end to
end environment is representative of reality: we could use a
MemoryTransport without invalidating them, but not a FakeBranch. So I
think acceptance tests will look broadly like those tests. e.g.
test_branch_from_trivial_branch_streaming_acceptance

For integration tests, I see them as being focused on the coupling: the
gaps between layers. So I'd really like to see them exercise two layers,
with everything below the second layer able to be test doubles. Thinking
about it more it may be that what I'd like to see is integration tests
distributed amongst other tests: cli integration tests would test
coupling of the cli to WorkingTree, Branch etc. WorkingTree integration
tests would test WorkingTree <-> tree transform, branch.

So perhaps 'integration' isn't needed, rather we could say that the only
tests that need to be fully integrated are acceptance tests; and measure
the value and benefits of a fully integrated test by whether it really
measures something a user would pass/fail us on.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20090918/cb76246b/attachment-0002.pgp