test isolation: default ignores & repository acquisition

[Martin Pool]

2009/9/16 Robert Collins <robertc at robertcollins.net>:
>> It seems like this is not so much something you want to hook, as
>> something that should be outside of the scope of what's being tested.
>> So I wonder if we should handle that by hoisting it to say
>> initialize_per_user_data, run from run_bzr but not lower levels.
>
> We could look at doing that. At the moment its called via working tree
> methods, and config methods.

I think that's probably a bug in the workingtree or config code if
they implicitly write stuff into the home directory.

>> I (still) think that changing the test base class is a poor way to
>> describe this, and that we have tended to get it wrong in the past,
>> either by people inheriting from something too broad and expensive or
>> something too narrow and insufficiently isolated.  Also, the different
>> things you might want here (a home dir vs a memory transport) are
>> somewhat orthogonal and that maps poorly onto inheritance.
>
> I agree that the base class isn't a great way to describe the need for
> different features. However, for safety and isolation I want our topmost
> base class to prevent/inhibit/cause-errors actions outside the test
> environment. We could add a dependency on testresources, if you like,
> which has all the machinery to bring up and tear down infrastructure
> that tests need. It would be a very small amount of work to write
> always-dirty resources that provide HomeDir, Safety nets and so on.
>
> Compare that with my proposal though, which adds a single class to the
> classes we already have: its probably much less overhead to rearrange
> what we have a little and focus on changing the style later.
>
>> What I would prefer is just methods on the bzr base TestCase that say
>> "I need a home dir" or "give me a transport" and then they construct
>> the thing as it's needed.  They can of course do that from setUp of a
>> particular set of tests. Since we have addCleanup I don't see any
>> reason people need it done per class.
>
> There is no strong implication that several base TestCase classes would
> be anything other than custom setUp methods pre-canned for users. We
> already know that many tests only need X, or Y, and providing that X or
> Y are currently done via inheritance.
>
> That said, I think that a single big TestCase is pretty ugly, and its an
> unfortunate consequence of UnitTest's current design. Having multiple
> 'core' classes lets us avoid having a truely mega TestCase.
>
> There are a couple of driving factors towards having all the methods on
> the topmost class:
>  - assertions & object state
>  - helper functions & object state
>
> For the former, something like hamcrest looks really nice to me. For the
> latter, well thats what I wrote testresources for :).
>
> Using one TestCase base class and then putting different permutations of
> setup stuff into setUp of child classes isn't really any different to
> having several different TestCase base classes which children can choose
> from: in fact, the very first time that two different children in the
> former style have the same setUp, you'll end up morphing into the latter
> style (many base classes), unless you deliberately preserve the
> duplication.

To be more precise I think the problems come in when:

 * Things you can do when inheriting from class X, but that are only
safe or sensible to do when inheriting from class Y.  (Example: file
io when not in a temp dir.)

 * Features that are available or changes that can only be made by
inheriting from a particular class, even though they make sense to use
when not inside that class.  (Example: running a subprocess, or
assertIsDirectory on TestCaseWithTransport.)  These are sometimes
conflating "a test that thinks about transports at all" with "a test
that runs in an isolated directory."

 * The inheritance hierarchy conflates things: a TestCaseInTempDir
is-a TestCaseWithMemoryTransport but you could sensibly (and more
cheaply) have a temporary  directory without a memory transport, and
so on.

The interactions are bad because the unittest style encourages you to
put multiple tests in a single class, often a whole file in a single
class.

(The examples may be out of date.)

I can see now your changes are probably not making this any worse, and
may be making them better.

-- 
Martin <http://launchpad.net/~mbp/>