[RFC] Changes to 'selftest --starting-with'

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Russ Brown wrote:
> John Arbash Meinel wrote:

...

> 
> Hopefully I'm not barking up the wrong tree here, but can't you just do
> something like this instead:
> 
> bzr selftest test_command
> 
> which will run all tests that contain the string 'test_command'.
> 
> When I was writing tests for the diffstat plugin I just had to run bzr
> selftest diffstat and it Just Worked.
> 
>> John
>> =:->

That works, but it has to load the whole test suite, and then filter using
your regex. And when you have 11+k tests like bzr does, the time to figure out
the full test suite is actually rather long.

If you use -s/--starting-with it *only* loads the tests that start with the
string. This can make a very large difference in the time it takes to run the
subset you want.

For example "time bzr selftest --no-plugins --list"
takes 4.9 seconds. So it is 5s just to load all possible tests. It is much
worse on win32 and my old Mac laptop. 30+s was not unusual.

Considering that:
time bzr selftest --no-plugins -s bzrlib.tests.test_status

Finishes in 0.8s means that I spend about 5x the length of time just waiting
for the test suite to *start* than it actually spends running tests.

And if you are trying to do Test *Driven* Development, you want to run the
test suite all the time. Waiting 5s is far to long (when it gets close to 10s
my attention wanders, which really hurts productivity.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIrGp6JdeBCYSNAAMRAkTUAKDWqdUUzP7Cssymczs8DmnwHV6wLwCgsQpG
rW1RfSuHmta3xRwhHpo28dk=
=W2s5
-----END PGP SIGNATURE-----