Subject: Artificial performance limitation

From: Daniel van Vugt <[email-masked]>
Date: Mon, 03 Nov 2014 08:36:54 +0800

This bug and its sister bug are driving me crazy:
https://bugs.launchpad.net/mir/+bug/1388490
https://bugs.launchpad.net/mir/+bug/1377872

I can see in both cases that the frame rate ceiling is arbitrary and 
temporary. It has nothing at all to do with the power of the host. And 
the workaround that fixes the problem in both cases is a bit crazy.

I've so far investigated theories around:
   * buffer lifetimes
   * Mesa/intel GL batching bugs
   * lock contention (locks held too long)
   * power management modes
   * Mir socket/protocol bottlenecks

All the theories have some merit but I am not making progress. If you 
are (un)lucky enough to be able to reproduce either bug then please join 
in. More data points are required.

- Daniel

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 17:45:34 +0800

Found!

Instrumenting AsioMainLoop [replace run() with a loop around run_once()] 
you see with one client you get one task executing per frame, which 
blocks the whole event loop (idlely not using CPU) for 16.6ms every frame:

Run 0.016630977 sec
Run 0.016647716 sec
Run 0.016633778 sec
...

Additional clients only block the event loop for a fraction of a 
millisecond more. e.g. 3 clients:

Run 0.016463334 sec
Run 0.000024074 sec
Run 0.000048409 sec
Run 0.016543939 sec
Run 0.000022160 sec
Run 0.000048854 sec
Run 0.016558223 sec
Run 0.000021511 sec
Run 0.000047816 sec
Run 0.016546486 sec
Run 0.000022321 sec
Run 0.000048090 sec

However at 9-10 clients (on a Haswell i7), the time spent blocking in 
some AsioMainLoop iterations becomes two frames (hence visible frame 
skipping):

Run 0.033355181 sec
Run 0.000021174 sec
Run 0.000007288 sec
Run 0.000024818 sec
Run 0.000021997 sec
Run 0.000122351 sec
Run 0.000001801 sec
Run 0.000001012 sec
Run 0.000001152 sec
Run 0.000003406 sec
Run 0.032885940 sec
Run 0.000010293 sec
Run 0.000010438 sec
Run 0.000010319 sec
Run 0.000023508 sec
Run 0.000042345 sec
Run 0.000016997 sec
Run 0.000022163 sec
Run 0.000015114 sec
Run 0.000031571 sec

I think any task running in AsioMainLoop that blocks at all (>1ms) is a 
bug... So any experts on AsioMainLoop know where we're submitting 
blocking tasks to the poor overworked AsioMainLoop?

- Daniel

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 18:31:46 +0800

Bugger. Come to think of it, that only proves the bug exists. Not what 
the cause is. Because blocking for a full frame is correct if the event 
loop is idle. And blocking for two frames only demonstrates the bug 
exists and we're not meeting the frame deadline...

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 13 Nov 2014 11:58:37 +0800

Pretty pictures of a stuttering server attached.

Notice the CompositingFunctor where ~vector destruction is consuming 30% 
of the time, due to ~TemporaryCompositorBuffer taking 27% of the time.

I think we need to move the final buffer release/response logic out of 
the compositor thread one way or another, because blocking the 
compositor thread is a very visible problem.

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 13 Nov 2014 12:19:16 +0800

I think the cleanest solution would be a return to using the "IPC thread 
pool" (like in the old days). Presently we use it for receiving 
requests, but not for sending responses any more (unless frame dropping).

The challenge is only to ensure we don't reintroduce the problems we 
used to have with it being fixed in size. Some intelligent scaling is 
required.

Although longer term, a thread pool might not be required at all if we 
can come up with a reliably fast asynchronous select() implementation to 
replace the asio sockets/io_service.

---

From: Alan Griffiths <[email-masked]>
Date: Thu, 13 Nov 2014 14:48:56 +0000

I've not looked through your evidence, but you seem to be overlooking an
option:

Push the response logic to the BasicConnector::io_service - that uses
epoll() to farm work out to the IPC thread pool (which, despite its
default size of 1, still exists).

Yes, there's a bit of wiring things together needed to get appropriate
access in SessionMediator::exchange_buffer(), but not much - its all
frontend.

On 13/11/14 04:19, Daniel van Vugt wrote:
> I think the cleanest solution would be a return to using the "IPC
> thread pool" (like in the old days). Presently we use it for receiving
> can come up with a reliably fast asynchronous select() implementation
> to replace the asio sockets/io_service.
>
>

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 14 Nov 2014 09:51:47 +0800

Yes agreed; that's actually what I was saying in the previous email and 
I prototyped it yesterday.

No visible benefit yet so I'm trying to find out why still...

P.S. Surprisingly io_service is over-simplified and does not lend itself 
to dynamic thread pooling. Because there's no nice single entry point to 
hook for job submission, which you need to guarantee enough threads are 
allocated. Best case is you observe jobs are contending after the fact 
and react in a delayed fashion :(

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 14 Nov 2014 16:06:20 +0800

No visible benefit because I'm testing on my Intel desktop, and even 
after offloading the responses to the IPC pool the compositor thread is 
still crippled by the Intel deferred batching problem. I have a fix for 
that too; just trying to make it less ugly.

So I know how to fix the desktop performance now for sure. As for the 
netbook, I think that requires both fixes at least. Maybe more...

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 12 Nov 2014 19:12:21 +0800

Detailed profile graph (.SVG) attached, thanks to Google's profiler.

Notice the CompositingFunctor where ~vector is consuming 30% of the 
time, due to ~TemporaryCompositorBuffer taking 27% of the time.

I think we need to move the final buffer release/response logic out of 
the compositor thread (even not using callbacks?). And occluded ones 
too, need to be released in a more controlled environment (not in an 
alarm's mainloop thread where they will all queue up and contend).

With some luck, someone might have done so before I wake up...