Unit Tests & Integration Tests

[Ian Booth]

On 12/09/14 01:59, roger peppe wrote:
> On 11 September 2014 16:29, Matthew Williams
> <matthew.williams at canonical.com> wrote:
>> Hi Folks,
>>
>> There seems to be a general push in the direction of having more mocking in
>> unit tests. Obviously this is generally a good thing but there is still
>> value in having integration tests that test a number of packages together.
>> That's the subject of this mail - I'd like to start discussing how we want
>> to do this. Some ideas to get the ball rolling:
> 
> Personally, I don't believe this is "obviously" a good thing.
> The less mocking, the better, in my view, because it gives
> better assurance that the code will actually work in practice.
> 
> Mocking also implies that you know exactly what the
> code is doing internally - this means that tests written
> using mocking are less useful as regression tests, as
> they will often need to be changed when the implementation
> changes.
>

Let's assume that the term stub was meant to be used instead of mocking. Well
written unit tests do not involve dependencies outside of the code being tested,
and to achieve this, stubs are typically used. As others have stated already in
this thread, unit tests are meant to be fast. Our Juju "unit" tests are in many
cases not unit tests at all - they involve bringing up the whole stack,
including mongo in replicaset mode for goodness sake, all to test a single
component. This approach is flawed and goes against what would be considered as
best practice by most software engineers. I hope we can all agree on that point.

To bring up but one of many concrete examples - we have a set of Juju CLI
commands which use a Juju client API layer to talk to an API service running on
the state server. We "unit" test Juju commands by starting a full state server
and ensuring the whole system behaves as expected, end to end. This is
expensive, slow, and unnecessary. What we should be doing here is stubbing out
the client API layer and validating that:
1. the command passes the correct parameters to the correct API call
2. the command responds the correct way when results are returned

Anything more than that is unnecessary and wasteful. Yes, we do need end-end
integration tests as well, but these are in addition to, not in place of, unit
tests. And integration tests tend to be fewer in number, and run less frequently
than, unit tests; the unit tests have already covered all the detailed
functionality and edge cases; the integration tests conform the moving pieces
mesh together as expected.

As per other recent threads to juju-dev, we have already started to introduce
infrastructure to allow us to start unit testing various Juju components the
correct way, starting with the commands, the API client layer, and the API
server layer. Hopefully we will also get to the point where we can unit test
core business logic like adding and placing machines, deploying units etc,
without having to have a state server and mongo. But that's a way off given we
first need to unpick the persistence logic from our business logic and address
cross pollination between our architectural layers.