Subject: HWC Multimonitor work

From: Kevin DuBois <[email-masked]>
Date: Wed, 10 Dec 2014 15:59:55 -0500

Hello All,
Have already embarked a bit on some android multimonitor work, and wanted
to share the plan as it stands for team comment/input.

There are two bigger chunks to the work:

Display Configuration and Control:
This is currently buried too deep behind mga::DisplayDevice. I'm going to
hoist it out of mga::DisplayDevice and mga::DisplayBuffer and put it in
mga::Display. Here it will be in a good position to control and gather the
display configuration of all the connected monitors. I'll be able to get
rid of ConfigurableDisplayDevice, make mga::Framebuffers lighter, and
remove  HwcCommonDevice. [1]

Display Posting:
MultiThreadedCompositor currently has one thread per display buffer, but
this isn't very friendly to android multimonitor. [2]. My plan is to rework
the MultiThreadedCompositor to figure out via mg::Display and
mg::DisplayBuffer how to setup and control its renderloop.

Cheers,
Kevin

[1]
This also has the added benefit of helping prepare for HWC 1.4, and giving
some of the classes a renaming or a removal.
[2]
Details of why:

HWC's prepare() and set() functions post to all the displays in one call.
We can't submit to the primary and external displays independently, because
we wind up with situations where the releaseFence of DisplayA feeds to the
acquireFences of DisplayB, and we ping-pong or deadlock with each display
waiting for the other to post.

So, given that we need to synchronize, and submit from one thread, I
considered if we could synchronize internally in the android code to make
the existing MultiThreadedCompositor work. The following reasons led me to
wanting to rework MultiThreadedCompositor to sometimes prefer
single-threaded operation:

1) We would have an superfluous thread for no reason, that incurs more
context switches.
2) The android code becomes more ropey because we're trying to synchronize
in the wrong place.
3) We've had a longstanding problem (even with single monitor) that some
badly written / buggy android drivers break when we deviate from the exact
threating model that SurfaceFlinger uses. This would simplify our threading
model and support junky drivers better.

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 11 Dec 2014 09:34:55 +0800

Regarding Display Posting [2] ...

MultiThreadedCompositor can be easily tricked into creating only one 
thread, one DisplayBufferCompositor and sharing it between an arbitrary 
number of outputs using a single render loop. Alexandros made sure of 
this, but I feel we have not utilised its full potential yet.

All you have to do is convince OverlappingOutputGroup that your two 
Android displays overlap (even when they don't) and you'll get one large 
shared DisplayBuffer to cover them all. Then all you need to do is 
convince the Android display code to scan-out a sub-rectangle of that 
DisplayBuffer.

Our DRM code already does all this, although we have not yet made it an 
explicitly configurable option to choose when to have one large shared 
DisplayBuffer. It does work (try clone mode with monitors of different 
resolution).

On 11/12/14 04:59, Kevin DuBois wrote:
> Hello All,
> Have already embarked a bit on some android multimonitor work, and
> wanted to share the plan as it stands for team comment/input.
>
> This is currently buried too deep behind mga::DisplayDevice. I'm going
> to hoist it out of mga::DisplayDevice and mga::DisplayBuffer and put it
> in mga::Display. Here it will be in a good position to control and
> gather the display configuration of all the connected monitors. I'll be
> able to get rid of ConfigurableDisplayDevice, make mga::Framebuffers
> lighter, and remove  HwcCommonDevice. [1]
>
> Display Posting:
> this isn't very friendly to android multimonitor. [2]. My plan is to
> rework the MultiThreadedCompositor to figure out via mg::Display and
> This also has the added benefit of helping prepare for HWC 1.4, and
> giving some of the classes a renaming or a removal.
> [2]
> Details of why:
>
> HWC's prepare() and set() functions post to all the displays in one
> call. We can't submit to the primary and external displays
> independently, because we wind up with situations where the releaseFence
> of DisplayA feeds to the acquireFences of DisplayB, and we ping-pong or
> deadlock with each display waiting for the other to post.
>
> considered if we could synchronize internally in the android code to
> make the existing MultiThreadedCompositor work. The following reasons
> led me to wanting to rework MultiThreadedCompositor to sometimes prefer
> 2) The android code becomes more ropey because we're trying to
> synchronize in the wrong place.
> badly written / buggy android drivers break when we deviate from the
> exact threating model that SurfaceFlinger uses. This would simplify our
> threading model and support junky drivers better.
>
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 11 Dec 2014 13:31:36 +1100

On Thu, Dec 11, 2014 at 7:59 AM, Kevin DuBois 
<[email-masked]> wrote:
…
> call.

Incidentally, how does this work with displays of different refresh 
rate? Does HWC drop pending flips when a new flip is scheduled?

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 11 Dec 2014 10:50:48 +0800

Even displays of the same refresh rate are a problem (see in X/Compiz 
where all but one of your displays will tear).

In Mir (the DRM platform) we've solved this with parallel rendering 
while waiting for the previous page flips. Because if you have multiple 
monitors at the same refresh rate, the upper limit on time to wait for 
vsync on all of them is (asymptotically) double that of a single monitor.

It's possible HWC could support such an approach, so long as we have the 
required triple-buffering in the compositor.

---

From: Kevin DuBois <[email-masked]>
Date: Thu, 11 Dec 2014 14:03:19 -0500

@OverlappingOutputGroup
The first goal is to have a clone mode, so the OverlappingOutputGroup could
be a good trick. (We can't designate scanout of a sub-rectangle of a larger
framebuffer surface in HWC though, so might be difficult)

If we try to have different-content monitors that really don't overlap
though, I'd want to present the DisplayBuffers as separate, so at some
point in the future we'd have to give some hint or requirement from the
platform to the server about what's the best way to drive the composition
loop.

Expounding on point 3) a bit more, a bit more input from the platform about
how to run the loop would help robustify against drivers with weird fencing
problems (which we've hit in bringup), as it aligns the
compositor-drive-model that the drivers are best-tested with. We can also
properly implement the HWC invalidate() callback (which, isnt used very
often, but we currently just drop the request)

@different refresh rate,
If there are 2 sets inbetween the pageflips, the last one sticks for that
display. IMO Mesa has thought about multimonitor better in its api, and
'don't break mesa's goodness" is a goal in accommodating HWC.

On Wed, Dec 10, 2014 at 9:50 PM, Daniel van Vugt <
[email-masked]> wrote:

> In Mir (the DRM platform) we've solved this with parallel rendering while
> waiting for the previous page flips. Because if you have multiple monitors
> at the same refresh rate, the upper limit on time to wait for vsync on all
> of them is (asymptotically) double that of a single monitor.
>
> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/mir-devel
>

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 12 Dec 2014 09:16:23 +0800

So if you need separate display buffers but a single thread then what if 
you just replaced MultiThreadedCompositor with a 
SingleThreadedCompositor which does:

while (running)
{
     wait for frame(s)
     for each display buffer db
         db->composite()
}

That will work providing you employ parallelism like in Mesa such that 
you're never waiting on the next page flip inside composite(); only 
waiting on the page flip of the previous frame. Although I think you're 
pointing out already that Android isn't that flexible?...

Yet another option would be the compromise we have in X/Compiz right 
now. Just sync to vblank on one output only (and accept tearing on all 
the others).

On 12/12/14 03:03, Kevin DuBois wrote:
> @OverlappingOutputGroup
> The first goal is to have a clone mode, so the OverlappingOutputGroup
> could be a good trick. (We can't designate scanout of a sub-rectangle of
> a larger framebuffer surface in HWC though, so might be difficult)
>
> platform to the server about what's the best way to drive the
> composition loop.
>
> Expounding on point 3) a bit more, a bit more input from the platform
> about how to run the loop would help robustify against drivers with
> weird fencing problems (which we've hit in bringup), as it aligns the
> compositor-drive-model that the drivers are best-tested with. We can
> also properly implement the HWC invalidate() callback (which, isnt used
> very often, but we currently just drop the request)
>
> @different refresh rate,
> If there are 2 sets inbetween the pageflips, the last one sticks for
> that display. IMO Mesa has thought about multimonitor better in its api,
> and 'don't break mesa's goodness" is a goal in accommodating HWC.
>
> On Wed, Dec 10, 2014 at 9:50 PM, Daniel van Vugt
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>
>     Even displays of the same refresh rate are a problem (see in
>     X/Compiz where all but one of your displays will tear).
>
>     while waiting for the previous page flips. Because if you have
>     multiple monitors at the same refresh rate, the upper limit on time
>     to wait for vsync on all of them is (asymptotically) double that of
>     a single monitor.
>
>     It's possible HWC could support such an approach, so long as we have
>     the required triple-buffering in the compositor.
>
>
>
>         <[email-masked] <mailto:[email-masked]>>
>         wrote:
>         …
>
>
>             Details of why:
>
>     [email-masked] <mailto:[email-masked]>
>     Modify settings or unsubscribe at:
>     https://lists.ubuntu.com/__mailman/listinfo/mir-devel
>     <https://lists.ubuntu.com/mailman/listinfo/mir-devel>
>
>

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 12 Dec 2014 14:46:25 +0800

Oh, a SingleThreadedCompositor might be infeasible due to:
   https://bugs.launchpad.net/mir/+bug/1395421

And I have a couple of performance fixes that are blocked by that bug 
too. Any takers?