Black-box and white-box testing

[roger peppe]

On 18 March 2013 01:11, Tim Penhey <tim.penhey at canonical.com> wrote:
> Hi all,
>
> We have two types of tests in the juju-core codebase at the moment, and
> I'd like to get some general agreement on how we lay out tests, and what
> types of tests are found where.
>
> http://en.wikipedia.org/wiki/White-box_testing
>  - tests behaviour of internal methods
>
> http://en.wikipedia.org/wiki/Black-box_testing
>  - tests provided functionality without peering into the internal state
>
> Now, from what I've understood from the standard Go practice, is that
> these are normally white-box tests, found in the same directory as the
> source with <name>_test.go as tests for <name>.go, and to have the tests
> inside the same package.
>
> These files are ignored for go build and go install calls, but compiled
> and executed using go test.
>
> Now, Juju does different (most of the time).
>
> Tests found in the same package are often in a different package called
> <name>_test.  This means that the tests are black box tests enforced by
> the language, as only public interfaces can be tested.

That's not quite true. There's a convention of making private interfaces
available to the external tests by exporting them in an export_test.go
file. It only goes a certain way, but it does mean the boundary isn't strict.

I don't think that the fact that we are mostly testing functionality through
public APIs necessarily means that we are doing black box testing.
As the wikipedia pages says: "In white-box testing an internal perspective of
the system, as well as programming skills, are used to design test cases."
Just because we're only using the public API doesn't mean the test
cases aren't designed with knowledge of the internals of the system
(I certainly try to write external tests to exercise code paths
rather than treating the code as a black box).

> This then leads to a name clash when wanting to test both internal
> functions, and the public interface using real imports for the same file.

I don't see the problem there.

> Proposal
> ========
>
> 1) Test files that are in the same directory become white-box tests,
> living in the same package as the code.

I suspect this may lead to circular imports in many cases,
although there is a standard workaround for that.

For myself, I like the fact that when writing tests against the publicly
visible API, we use qualified identifiers just like any external package.
It gives me a feel for how the API will feel to use in practice, and
it also means that
the testing code is trivial to factor to another package if necessary.

> 2) Each directory containing source, gains a "tests" subdirectory that
> contains the black-box tests for that package.  The package for the
> files in that directory is called "tests".

I'm not keen on this proposal. I think "go test somepkg" should
test somepkg, as far as is feasible.

> This way we have clear delineation between the white-box tests and the
> black-box tests.

I have another suggestion. Rather than changing every single test
we currently have, we could establish a convention that white box
tests live in files named *_intern_test.go.

As we add more such tests, the number of such files will go up;
eventually we might consider reversing the convention
to make external testing files the longer name, but
for now it will avoid huge code churn.

Thoughts?