[rfc] benchmark reliability

[Martin Pool]

On 17 Aug 2006, John Arbash Meinel <john at arbash-meinel.com> wrote:

> Any suggestions?
> 
> Running and tweaking under --lsprof is an option, but it penalizes
> python function calls and object creation too much. I can have a huge
> lsprof improvement, and only get a minor real improvement.

Like Robert, I would guess this is caused by Python keeping some memory
allocated in a pool, so that the second time you use it it's faster.  I
am surprised it's so substantial though.  You could ask Andrew, or
Holger for an opinion as they're familiar with Python's innards.

I heard something about a new API that gives you a count of Python vm
ticks executed.  (Can't remember the name at the moment.)  It might be
interesting to call that, if only to verify that the difference really
is in how long vm operations take, not in how many are being executed.

(This is so obvious but I'll still ask) - your machine isn't using cpu
throttling or something like that, where processes speed up as they run?
Both my Ubuntu laptop and OS X desktop will do that, and it takes a fair
fraction of a second for them to ramp up.

-- 
Martin