Subject: New Buffer Semantics Planning

From: Kevin DuBois <[email-masked]>
Date: Thu, 25 Jun 2015 14:51:59 -0400

I've started spiking a bit on how to transition the system to what we've
been calling the new buffer semantics¹, and come up with a plan. We've
already landed the ipc plumbing, now we have to make use of it to its
potential.

Obviously, the number one thing to avoid is regressions in performance or
functionality while transitioning. So, we'll get the new semantics up to
par, and then switch the default from rpc exchange_buffer() to rpc
submit_buffer().

With the ability to send buffers without a client request, we're really
turning the system around and eliminating the need for
mc::BuffeQueue::client_acquire(). As this is one of the 4 important entry
points to BufferQueue, it seems difficult to have BufferQueue service both
the exchange/next_buffers swapping, and the new buffer semantics;
especially as we have a semi-nice mc::BufferStream interface that we could
write a new implementation for.

BufferQueue's unit test is tied to mc::BufferQueue, and has some threading
cruft from back when we had to wait more often. BufferQueue actually has a
simple locking strategy these days... just lock at the top of the member
function. BufferQueue's unit test is also the guard we have against
regressions, so we shouldn't move the test in order to have less
regressions.

So, I've started writing an integration-level test that currently tests
BufferQueue in terms of production/consumption² Instead of relying on
threading to tease out the different patterns, the patterns are just
specified to be ones of interest (eg, overproduction, starving the
consumer). Its not complete (meaning, covers all the cases that
BufferQueueTest does) yet.

Once that test is roughly on-par with the BufferQueue test, we start
writing the client-side and server-side production code for the new buffer
system. Once that's done and tested, we'll probably have to do a bit of
careful study using some of the latency tools Alexandros has been working
on, and some of the benchmarks before flipping the new-buffer-semantics
switch to "on".

Sharing the plan just to see if everyone's on the same page. Not blocked on
anything right now, but somewhere down the road, we might want to deprecate
at least the rpc call to next_buffer() or exchange_buffer(), as well as
come up with the client api that nested or multimedia can use to manage the
client buffers.

Thanks,
Kevin

¹ Currently, we just let clients hold own one buffer. A few compelling
cases (multimedia decoding, AV synchronization, nested latency
optimizations, SET_BUFFER_COUNT from the mali driver, and lp: #1369763)
suggest that we should let advanced clients manage their own buffers.

²
https://code.launchpad.net/~kdub/mir/bstream-integration-test/+merge/263014

---

From: Cemil Azizoglu <[email-masked]>
Date: Thu, 25 Jun 2015 15:20:27 -0500

Sounds good.

+1 on the new-buffer-semantics switch as I'm a bit worried about regressing
on some of the lag & latency reduction work that went in.

Thanks

On Thu, Jun 25, 2015 at 1:51 PM, Kevin DuBois <[email-masked]>
wrote:

> Sharing the plan just to see if everyone's on the same page. Not blocked
> on anything right now, but somewhere down the road, we might want to
> deprecate at least the rpc call to next_buffer() or exchange_buffer(), as
> well as come up with the client api that nested or multimedia can use to
> manage the client buffers.
>
> Thanks,
> Kevin
>
> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel
>
>

-- 
Cemil Azizoglu
Mir Display Server - Team Lead
Canonical USA

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 26 Jun 2015 10:39:52 +0800

I'm curious (but not yet concerned) about how the new plan will deal 
with the transitions we have between 2-3-4 buffers which is neatly 
self-contained in the single BufferQueue class right now. Although as 
some responsibilities clearly live on one side and not the other, maybe 
things could become conceptually simpler if we manage them carefully:

   framedropping: Always implemented in the client process as a 
non-blocking acquire. The server just receives new buffers quicker than 
usual and needs the smarts to deal with (skip) a high rate of incoming 
buffers [1].

   bypass/overlays: Always implemented in the server process, invisible 
to the client. The server just can't enable those code paths until at 
least two buffers have been received for a surface.

   client wake-up: Regardless of the model/mode in place the client 
would get woken up at the physical display rate by the server if it's 
had a buffer consumed (but not woken otherwise). More frequent wake-ups 
for framedropping are the responsibility of libmirclient itself and need 
not involve the server to do anything different.

[1] Idea: If the server skipped/dropped _all_ but the newest buffer it 
has for each surface on every composite() then that would eliminate 
buffer lag and solve the problem of how to replace dynamic double 
buffering. Client processes would still only be woken up at the display 
rate so vsync-locked animations would not speed up unnecessarily. 
Everyone wins -- minimal lag and maximal smoothness.

On 26/06/15 02:51, Kevin DuBois wrote:
> Obviously, the number one thing to avoid is regressions in performance
> or functionality while transitioning. So, we'll get the new semantics up
> to par, and then switch the default from rpc exchange_buffer() to rpc
> submit_buffer().
>
> mc::BuffeQueue::client_acquire(). As this is one of the 4 important
> entry points to BufferQueue, it seems difficult to have BufferQueue
> service both the exchange/next_buffers swapping, and the new buffer
> semantics; especially as we have a semi-nice mc::BufferStream interface
> that we could write a new implementation for.
>
> BufferQueue's unit test is tied to mc::BufferQueue, and has some
> threading cruft from back when we had to wait more often. BufferQueue
> actually has a simple locking strategy these days... just lock at the
> top of the member function. BufferQueue's unit test is also the guard we
> have against regressions, so we shouldn't move the test in order to have
> less regressions.
>
> writing the client-side and server-side production code for the new
> buffer system. Once that's done and tested, we'll probably have to do a
> bit of careful study using some of the latency tools Alexandros has been
> working on, and some of the benchmarks before flipping the
> new-buffer-semantics switch to "on".
>
> deprecate at least the rpc call to next_buffer() or exchange_buffer(),
> as well as come up with the client api that nested or multimedia can use
> to manage the client buffers.
>
> Thanks,
> Kevin
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Fri, 26 Jun 2015 13:39:41 +1000

On Fri, Jun 26, 2015 at 12:39 PM, Daniel van Vugt 
<[email-masked]> wrote:
> some responsibilities clearly live on one side and not the other, 
> maybe things could become conceptually simpler if we manage them 
> carefully:
> 
> non-blocking acquire. The server just receives new buffers quicker 
> than usual and needs the smarts to deal with (skip) a high rate of 
> incoming buffers [1].

Clients will need to tell the server at submit_buffer time whether or 
not this buffer should replace the other buffers in the queue. 
Different clients will need different behaviour here - the obvious case 
being a video player that wants to dump a whole bunch of time-stamped 
buffers on the compositor at once and then go to sleep for a while.

But in general, yes. The client acquires a bunch of buffers and cycles 
through them.

>   bypass/overlays: Always implemented in the server process, 
> invisible to the client. The server just can't enable those code 
> paths until at least two buffers have been received for a surface.

I don't think that's the case? Why does the server need two buffers in 
order to overlay? Even with a single buffer the server always has a 
buffer available¹.

It won't be entirely invisible to the client; we'll probably need to 
ask the client to reallocate buffers when overlay state changes, at 
least sometimes.

> had a buffer consumed (but not woken otherwise). More frequent 
> wake-ups for framedropping are the responsibility of libmirclient 
> itself and need not involve the server to do anything different.

By and large, clients will be woken up by EGL when the relevant fence 
is triggered.

I don't think libmirclient will have any role in waking the client. 
Unless maybe we want to mess around with

> [1] Idea: If the server skipped/dropped _all_ but the newest buffer 
> it has for each surface on every composite() then that would 
> eliminate buffer lag and solve the problem of how to replace dynamic 
> double buffering. Client processes would still only be woken up at 
> the display rate so vsync-locked animations would not speed up 
> unnecessarily. Everyone wins -- minimal lag and maximal smoothness.

¹: The assumption here is that a buffer can be simultaneously scanned 
out from and textured from. I *think* that's a reasonable assumption, 
and in the cases where I know it doesn't apply having multiple buffers 
doesn't help, because it's the buffer *format* that can only be scanned 
out from, not textured from.

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 26 Jun 2015 12:01:25 +0800

bypass/overlays: If you look at the current logic you will see that the 
DisplayBuffer holds the previous bypass/overlay buffer until _after_ the 
client has provided the next one. And it must, to avoid scan-out 
artefacts. So the server holds two of them very briefly. But only one is 
held most of the time. Without "predictive bypass" as I'm working on 
right now, that buffer is held for almost two frames. With "predictive 
bypass" it's closer to (but greater than still) one frame held. On 
startup, absolutely you're right that only one buffer is required to get 
bypass/overlays going. So my wording was wrong.

client wake-up: I may have worded that poorly too. The point is in the 
new world (tm) frame dropping mostly happens in the client (as opposed 
to all in the server like it is today). But some of it still needs to 
happen in the server because you don't want a compositor that tries to 
keep up with a 1000 FPS client by scheduling all of those frames on a 
60Hz display. It has to drop some.

On 26/06/15 11:39, Christopher James Halse Rogers wrote:
> not this buffer should replace the other buffers in the queue. Different
> clients will need different behaviour here - the obvious case being a
> video player that wants to dump a whole bunch of time-stamped buffers on
> the compositor at once and then go to sleep for a while.
>
> It won't be entirely invisible to the client; we'll probably need to ask
> the client to reallocate buffers when overlay state changes, at least
> sometimes.
>
> By and large, clients will be woken up by EGL when the relevant fence is
> triggered.
>
>> buffering. Client processes would still only be woken up at the
>> display rate so vsync-locked animations would not speed up

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 26 Jun 2015 12:04:08 +0800

Hmm, maybe not. If we assume the server is only communicating with the 
client at 60Hz then the client could just do all the dropping itself and 
send one frame (the newest completed one) every 16.6ms when the server asks.

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Fri, 26 Jun 2015 14:16:15 +1000

> send one frame (the newest completed one) every 16.6ms when the 

I don't think the server is ever going to ask for a frame.

All the client sees when the server is done with a buffer is that one 
of their previously submitted buffers changes state from read-only to 
exclusive-write access. It could possibly use a “has my last 
submitted buffer become writeable yet” heuristic to guess when the 
server will actually use a new buffer, but we don't guarantee that (nor 
can we, as you note with bypass).

> > bypass/overlays going. So my wording was wrong.

Right, but that's fine. If the client has submitted one buffer, and is 
a candidate for overlay, then it's clear that the old scanout buffer 
*wasn't* from the client. We hold onto the old scanout buffer and start 
scanning out of the (single) buffer the client has submitted.

When the client submits a second buffer, the first isn't released until 
we know it's no longer being scanned out of, but we don't need to have 
the client's second buffer before scanning out of the first.

We don't need to have two buffers around all the time for overlay; we 
need to have two buffers around to *switch* overlay buffers. But the 
fact that we're switching buffers already means that we've got at least 
two buffers.

This is sort of client-visible behaviour because the client can *see* 
that the server is holding more than one buffer, but it's the same 
logic for the client - “Do I have write ownership of a buffer? Yes: 
render to it. No: wait¹ for one of my buffers to become writeable, or 
allocate a new one”.

¹: Potentially “wait” by adding the fence to the GL command stream 
and submit rendering commands anyway.

> Different
> invisible

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 26 Jun 2015 12:24:27 +0800

That's why it's a hard problem to replace BufferQueue -- you have to 
figure out whether 2, 3 or 4 buffers for any given surface is correct at 
the time. Never over-allocate and never under-allocate (which causes 
freezing/deadlocks).

What I was suggesting is that a frame callback (for non-idle surfaces 
anyway) would allow us to solve most of the hard problems in a 
distributed manner between the client and server processes fairly 
elegantly. And we don't need heuristics; this is all based on certainty 
and logical reasoning.

On 26/06/15 12:16, Christopher James Halse Rogers wrote:
> All the client sees when the server is done with a buffer is that one of
> their previously submitted buffers changes state from read-only to
> exclusive-write access. It could possibly use a “has my last submitted
> buffer become writeable yet” heuristic to guess when the server will
> actually use a new buffer, but we don't guarantee that (nor can we, as
> you note with bypass).
>
>>
> Right, but that's fine. If the client has submitted one buffer, and is a
> candidate for overlay, then it's clear that the old scanout buffer
> that the server is holding more than one buffer, but it's the same logic
> for the client - “Do I have write ownership of a buffer? Yes: render to
> it. No: wait¹ for one of my buffers to become writeable, or allocate a
> new one”.
>
> ¹: Potentially “wait” by adding the fence to the GL command stream and
> submit rendering commands anyway.
>
>>
>> >

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Fri, 26 Jun 2015 14:29:17 +1000

On Fri, Jun 26, 2015 at 2:24 PM, Daniel van Vugt 
<[email-masked]> wrote:
> figure out whether 2, 3 or 4 buffers for any given surface is correct 
> at the time. Never over-allocate and never under-allocate (which 
> causes freezing/deadlocks).
> 
> elegantly. And we don't need heuristics; this is all based on 
> certainty and logical reasoning.

Ah, yes. A frame callback is indeed a fine idea that neatly bypasses 
all the problems.

>>> send one frame (the newest completed one) every 16.6ms when the 
>>> server
>>> asks.
>> 
>> submitted
>>> > DisplayBuffer holds the previous bypass/overlay buffer until 
>>> _after_
>>> the
>>> > startup, absolutely you're right that only one buffer is required 
>>> to
>>> get
>> logic

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 26 Jun 2015 15:22:26 +0800

TL;DR email is messy

See this instead: 
https://docs.google.com/spreadsheets/d/1qf3IssN_sujygMPK2sTX2AUhGiDUBF3TMYYcSRSaSy0/pubhtml

---

From: Kevin DuBois <[email-masked]>
Date: Mon, 29 Jun 2015 10:34:33 -0400

So, as far as planning goes....

I think that the spreadsheet (with a few nuances, perhaps) plus the
BufferQueue.* tests give us a good capture of the requirements.

So, seems sensible to link the translation of the BufferQueue tests and the
capture of the requirements into the new integration-level test somehow.
I'll see if I can track the translations in a spreadsheet somehow, so
people can see more easily that the new test suite maps to the old one and
the listed requirements.

---

From: Kevin DuBois <[email-masked]>
Date: Mon, 20 Jul 2015 15:28:27 -0400

Some further planning notes (also a repeat of what I chatted about in the
us/euro standup about https://trello.com/c/I8s7W2f9)

I proposed these 3 branches today:
https://code.launchpad.net/~kdub/mir/multimonitor-arbiter/+merge/265301
https://code.launchpad.net/~kdub/mir/dropping-schedule/+merge/265303
https://code.launchpad.net/~kdub/mir/queueing-schedule/+merge/265304

Hopefully its not too much of a stretch to see how these 3 classes can be
integrated. The code isn't used yet, apart from the tests. Not all the
features in the spreadsheet have been implemented yet; but it seemed better
for the purposes of ongoing feedback on the code to keep the reviews small.
Once all the serverside and client side features have been covered, then it
can be switched to default (which https://trello.com/c/w8qfQ59p is a
placeholder for)

Thanks,
Kevin

On Mon, Jun 29, 2015 at 10:34 AM, Kevin DuBois <[email-masked]>
wrote:

> So, seems sensible to link the translation of the BufferQueue tests and
> the capture of the requirements into the new integration-level test
> somehow. I'll see if I can track the translations in a spreadsheet somehow,
> so people can see more easily that the new test suite maps to the old one
> and the listed requirements.
>
>
>