Subject: [RFC] Performance test framework

From: Alexandros Frantzis <[email-masked]>
Date: Fri, 19 Jun 2015 13:22:05 +0300

Hello all,

there have recently been a few occasions where we wanted to experiment with
performance improvements, but lacked a way to easily measure their effect.
There have also been a few occasions where different implementations of the
(supposedly) same performance test came up with different results. In light of
these issues, it would be helpful to have a common performance test framework,
to make it easy to create and share performance tests scenarios.

As our codebase is C++ it would be natural to provide such a framework
in C++, providing the maximum flexibility and customizability.

On the other hand, sharing C++ code is not convenient, especially if want more
casual users to be able to try out the scenarios. As an alternative, we could
provide this test framework as a python library, possibly backed by a glue C++
library as needed (note that for some scenarios a glue layer is not needed,
since we could use our demo servers and clients, or write new ones).

Here are some pros and cons of the solutions (pros of one are cons of the other
in this case):

C++:
 + Easy integration with code base
 + Customizability of server/client behavior

python (possibly backed by C++ glue layer):
 + Easy to write
 + Convenience in sharing test scenarios
 + Plethora of existing libraries we need (e.g. python libs for babeltrace,
   uinput, statistics)

This how I imagine a performance test script could look like in Python (the C++
version wouldn't be very different in terms of abstractions):

    import time
    from mir import PerformanceTest, Server, Client, Input
    
    host = Server(reports=["input","compositor"])
    nested = Server(host=host)
    client = Client(server=nested)
    
    test = PerformanceTest([host,nested,client])
    test.start()
    
    # Send an input event every 50ms
    input = Input()
    for i in range(100):
        input.inject(...)
        time.sleep(0.050)
    
    test.stop()
    
    trace = test.babeltrace()
    ... process trace and print results ...

[Note: This is example is for illustration only. The specifics of the
performance framework API are not the main point of this post. We will have a
chance to discuss more about them in the future, after the high level decisions
have been made.]

The main drawback with the script based approach (vs a pure C++ approach) is
that it's not clear how to provide custom behavior for servers and, more
importantly, clients.

If we find that our performance tests need only a small collection of behaviors
we could make them available as configuration options:

    Client(behavior="swap_on_input")

Alternatively, we could provide more fine grained customization points that run
python code (which will of course increase the complexity of the glue layer):

    def on_input_received(...): do stuff
    Client(on_input_received=on_input_received)

So, what I would like to hear from you is:

1. Your general preference for python vs C++ for the test scripts
2. Any particular performance tests that you would like to see implemented,
   so we can get a first idea of what kinds of custom behaviors we may need
3. Any other comments, of course!

Thank you,
Alexandros

---

From: Alan Griffiths <[email-masked]>
Date: Fri, 19 Jun 2015 11:52:50 +0100

> to make it easy to create and share performance tests scenarios.

+1

>  + Plethora of existing libraries we need (e.g. python libs for babeltrace,

Actually, as I expect we'd be dynamically loading the Mir server (e.g.
ctypes should make it easy to set up tests scenarios using
libmir_demo_server_loadable.so) I don't think it would be hard to enable
python to run an alternative build. (The server lib to load just becomes
a config option - making before/after comparisons easier.)

I don't think we can avoid writing any C++ (as we may want to add
specific hooks into the code) but I'd like that rolled into the
supported libraries not as an ad-hoc add on.

I'd say try the python approach and see if we hit limits.

---

From: Alan Griffiths <[email-masked]>
Date: Fri, 19 Jun 2015 11:55:54 +0100

>> to make it easy to create and share performance tests scenarios.
...
> I'd say try the python approach and see if we hit limits.

Is there any need to capture requirements? Or do we have a few "current"
test scenarios to try automating?

---

From: Andreas Pokorny <[email-masked]>
Date: Fri, 19 Jun 2015 13:50:34 +0200

Hi,

> 3. Any other comments, of course!

So far I mostly used C++ to evaluate lttng results. I didnt felt slowed
down especially boost
accumulators made it very easy to produce statistical information. For
actual test execution
I used shell - but provided the necessary user input manually.

...So yes we need something to better share test scenarios.

To better understand system behaviour it would be good to visualise the
activities of
the processes. And especially not only libmir* itself. So I was looking for
a common way
to produce time spans of activities, or in other words pairs of lttng
traces points that can be
easily matched. But the bigger problem is getting a usable for
visualisation. The now
renamed Trace Compass tools are just far too slow to even cope with a few
seconds of data.
..

regards
Andreas

---

From: Kevin DuBois <[email-masked]>
Date: Fri, 19 Jun 2015 08:15:00 -0400

I'm okay with using python to execute the performance tests and
gather/parse lttng traces via babeltrace (it seems a decent api for
processing api traces). I agree that its better if we can visualize the
data nicely, but we have to deal with the problem that we're not collecting
this data at all right now first :)

On Fri, Jun 19, 2015 at 7:50 AM, Andreas Pokorny <
[email-masked]> wrote:

> the processes. And especially not only libmir* itself. So I was looking
> for a common way
> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel
>
>

---

From: Alexandros Frantzis <[email-masked]>
Date: Fri, 19 Jun 2015 15:36:19 +0300

> test scenarios to try automating?

I have a basic input-to-frame latency test in mind, and I would like to
hear about other scenarios that people had to actually measure in the
past, or think they will need to measure in the near future.

---

From: Gerry Boland <[email-masked]>
Date: Fri, 19 Jun 2015 16:04:06 +0100

> 3. Any other comments, of course!

Hey Alexandros,
this is something we'd very much like for unity8. I'd prefer Python.
Things we'd need:

1. ability to instrument any Mir-based server (obvious example QtMir/unity8)
2. easy means for QtMir to add events to the log - if using LTTng then
that's no problem.
3. ability to instrument any mir client - so Qt or GTK
4. similarly easy means for clients to add events to the log - again
LTTng makes this a non-issue.

We like an infrastructure to gather performance data for several topics:

a. input latency: I want to track the propagation of an event from mir
through nested servers to the client, to identify bottlenecks
b. buffer lifetimes: how long are servers holding onto buffers, would
help identify if buffers being held too long. How long must clients wait
for a new buffer after swap
c. gather motion to photon statistics.

Looking forward to it!
-G

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 22 Jun 2015 10:04:30 +0800

+1
Go nuts.

In the past the only point of contention was that people (myself 
included) made mistakes in measurement. So we need to be aware that 
non-trivial tests can have just as many bugs as the system being tested, 
and keep an eye out for them. It would be nice to build some automation 
that everyone trusts...

We also need to back up theoretical and numerical improvements with 
visual observation. Last year some optimisations were proposed that 
reported "lower latency" in their test automation but actually provided 
higher visual latency if you watched closely with human eyes.

One related example of a non-obvious side-effect you need to be aware of 
is input resampling vs no input resampling. If you turn resampling off 
then latency at that input stage is "eliminated", but as a result then 
you flood the buffer queues if it's faster than the display refresh rate 
and so get 32ms (and it increases to more over time) of extra visual 
latency. So turning off input resampling is a net increase in visual 
latency, for example...