[RFC] Benchmark reproducibility

[Matthew D. Fuller]

On Wed, May 09, 2007 at 10:18:57AM -0500 I heard the voice of
John Arbash Meinel, and lo! it spake thus:
>
> test_read_10k_index_c          OK      184ms/    1454ms
> test_read_10k_index_c_again    OK      150ms/    1372ms
> test_read_10k_index_py         OK      222ms/    1518ms
> test_read_10k_index_py_again   OK      179ms/    1406ms
> 
> These are exactly the same test, I'm just asking it to run 2 times
> instead of just once.

That's really odd.  What do you get with another handful of
invocations of it?  It'd be interesting to see just how tightly they
cluster.

Maybe lazy importing is costing you the first time through?  The
difference between the py's is slightly bigger (43, 19.4%) than the c's
(34, 18.5%), though 2 data points cant't really say much...


Somewhat peripheral, but ministat[0] is handy for quickly slapping up an
analysis of stats like this.  I invented some data points around the
above for an example:

% cat c ; echo --- ; cat py
184
150
161
157
143
152
---
222
179
180
175
168
188

% ./ministat -sw70 c py
x c
+ py
+----------------------------------------------------------------------+
|x     x x   x   x     +     +  ++   x  +                             +|
| |________M__A___________|                                            |
|                    |___________M____A________________|               |
+----------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   6           143           184         154.5     157.83333     14.218533
+   6           168           222         179.5     185.33333     19.117182
Difference at 95.0% confidence
        27.5 +/- 21.6707
        17.4234% +/- 13.7301%
        (Student's t, pooled s = 16.8469)



[0] <http://www.freebsd.org/cgi/cvsweb.cgi/src/tools/tools/ministat/ministat.c?rev=1.10;content-type=text%2Fplain>
    Needs libm, a quick "cc -o ministat ministat.c -lm" should do the
    trick.  The attached patch was needed to get it built for me on a
    Fedora 6 box; presumably something similar will do the job on
    Ubuntu.  Yeah, I know I've beat this drum before.



-- 
Matthew Fuller     (MF4839)   |  fullermd at over-yonder.net
Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/
           On the Internet, nobody can hear you scream.