[merge] benchmark caching

[Martin Pool]

On  9 Aug 2006, John Arbash Meinel <john at arbash-meinel.com> wrote:

> Well, there is one other benefit that I haven't mentioned. And that is
> you can construct a cache dir with a custom kernel sized tree. (For
> example, you can use the real kernel).
> 
> Without any form of caching, running any sort of post-one-commit kernel
> test really is too painful.
> I am working on making commit fast. But in the meantime, I really need
> something that makes it so I can have fast turnaround while I try
> different algorithms.
> 
> I'm not trying to be a weany, but waiting 10 minutes every time I want
> to check if my algorithm is faster/slower is really crappy versus
> waiting ~1 min to get the same information. (It takes approx 100s to
> build a 1-commit kernel tree from scratch, about 2min. If you want to
> run a couple tests that is 2min * N tests just for setup, rather than
> 20s*N tests for setup)

right.

> If you want, we can have the pqm run with caching disabled. I can
> certainly change it to add a --no-cache flag. And I would be happy to
> run the benchmarks that way from time to time.

I might be misreading it, but it looked like the cache had no effect
unless you specifically ran the selftests with --cache-dir.  Isn't that
true?  In that case we can run "official" benchmarks with no cache, but
developers can use a cache to be more productive.  It sounds like a fair
tradeoff to me.  

The worst that can happen is people may be wasting time changing
something which is not representative.

By the way we've set up an old machine to be benchmarks.bazaar-vcs.org.
I'll try to set up a cron job soon to run the benchmarks and display the
results as html.  That should make them nicely reproducible.

> But I think we really need a cache, if we want to make it easy to run
> the benchmarks.

I agree- the simple fact is we do need to be able to test what happens
on data sets that will take a long time to construct.

-- 
Martin