Unit Tests & Integration Tests

[Andrew Wilkins]

On Thu, Sep 11, 2014 at 11:59 PM, roger peppe <rogpeppe at gmail.com> wrote:

> On 11 September 2014 16:29, Matthew Williams
> <matthew.williams at canonical.com> wrote:
> > Hi Folks,
> >
> > There seems to be a general push in the direction of having more mocking
> in
> > unit tests. Obviously this is generally a good thing but there is still
> > value in having integration tests that test a number of packages
> together.
> > That's the subject of this mail - I'd like to start discussing how we
> want
> > to do this. Some ideas to get the ball rolling:
>
> Personally, I don't believe this is "obviously" a good thing.
> The less mocking, the better, in my view, because it gives
> better assurance that the code will actually work in practice.
>
> Mocking also implies that you know exactly what the
> code is doing internally - this means that tests written
> using mocking are less useful as regression tests, as
> they will often need to be changed when the implementation
> changes.
>

I basically agree with everything below, but strongly disagree that mocking
implies you know exactly what the code is doing internally. A good
interface should define acceptable input, and expected output; unit tests
should test that. Mocks allow you to inject input or inspect output where
it is otherwise awkward or infeasible using "real code" (as you suggest
below). Where we go wrong a lot of the time is patching/mocking out
internal behaviour, rather than at the edges.


> Two good reasons to mock stuff in tests:
>
> 1) because it's awkward or impossible to use the real code in a test
> environment.
> 2) because the real code is slow or unreliable.
>
> When neither of these two criteria apply, I believe we should
> avoid mocking.
>
> As for integration tests, I think we should run them by default,
> but provide a flag for skipping slow tests. The go test command has a
> "-short"
> flag that can be used to skip long-running tests. Perhaps
> we could consider doing something similar (perhaps triggered
> with an environment variable).
>
>   cheers,
>     rog.
>
> > Having integration tests spread about the package and having environment
> > variables that switch on/ off them being run
> >
> > $ JUJU_INTEGRATION go test ./...
> >
> > We could make use of build tags:
> >
> > $ go test -tags integration ./...
> >
> > We could put all the integration tests in a single package:
> >
> > $ go test github.com/juju/juju/integrationtests/...
> >
> >
> > Thoughts?
> >
> > Matty
> >
> > --
> > Juju-dev mailing list
> > Juju-dev at lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140912/5a5cef94/attachment.html>