[MERGE] Adding --starting-with <test_id> option to selftest make it run 5x faster

[Vincent Ladeuil]

>>>>> "martin" == Martin Pool <mbp at canonical.com> writes:

    martin> On Fri, Apr 25, 2008 at 1:39 AM, Vincent Ladeuil <v.ladeuil+lp at free.fr> wrote:
    >> Build on top of
    >> 
    >> http://bundlebuggy.aaronbentley.com/request/%3Cm263ugwhsf.fsf@free.fr%3E
    >> 
    >> pending approval, here is something to wet reviewers appetite :-)
    >> 
    >> This patch introduce a new option to selftest. By specifying
    >> --starting-with <test_id>, only the tests whose ID begins with
    >> the provided value will get loaded* and run.

    martin> That speedup certainly sounds nice.

Yeah, this is pretty trivial stuff build upon the previous work,
I always had that target in mind since the beginning and thanks
to all the feedback from the team, here we are.

    martin> In fact I suspect I would use this almost all the
    martin> time.

    martin> I haven't read the code in detail because John has but:

    martin>  * --starting-with is too long for something we'd use all the time.
    martin> Would be be too extreme to call it --in/-i?

So far it conflicts only with --strict, so --sta is mandatory or
we can just assign -s to --starting-with. I thought about shorter
names but I wanted to make it clear that it was not a regexp or
some other trick.

    martin>  * similarly, I don't want to retype 'bzrlib.tests'
    martin> every time, could that be presumed, if the name is
    martin> not already matched.

Hmmm.

1) nothing is known when the option is handled (and part of the
   performance win is due to the fact that some modules will not
   even be loaded at all).

2) some tests IDs begin with bzrlib.doc (just to be picky), and
   John also mentioned bzrlib.plugins (which is clearly a target
   for --start-with) but all of them start with 'bzrlib.' so I
   guess 'bzrlib.' could be added if not present. If one day we
   add tests outside of bzrlib... well, we'll see then.

3) what about an alias ? So that everybody can define as many as
   they want :)

    martin> * this should be in NEWS

Added after the review.

    martin> * probably also in the "how to run tests" section of
    martin> the developer guide

Did we have that ? ;-)

    martin> It would be kind of cool, 
    martin> though certainly not needed now, to have this also
    martin> integrate with -f, which I often use, so that the
    martin> tests you think are most important get started fast,
    martin> then everything else runs.


'run_suite' runs *after* test loading and handles
matching_tests_first and exclude_pattern (among others).

So both will be obeyed whatever options are used for the loading.

selftest *is* cool :)

    >> It addresses the main limitations of the --load-list option (you
    >> *need* a list before using it and it tends to bitrot quickly)

    martin> yes, that's great.

    >> while still being compatible with it (one use case being a list
    >> with all failing tests).

    martin> that too would be worth mentioning as a tip in the developer guide.
    martin> i'm happy to help with the text if you draft it.

I'll have a look at it and resubmit then,

   Vincent