Subject: Subsurface support, or delegated compositing

From: Christopher James Halse Rogers <[email-masked]>
Date: Mon, 25 Nov 2013 17:51:47 +1100

One of the architectural things that I want to get done at the sprint
next week is a solid idea of how we want to do nested
compositing/out-of-process plugins/subsurfaces - which all seem to me to
be aspects of the same problem.

In order to prime the discussion - and to invite outside contributions -
I thought I'd lay out the usecases as I see them before we get to
London.

There are two conceptual use-cases here -
1) “I want to delegate some of my UI to a third party”, and
2) “I need to do some compositing, and want to do this efficiently”

A Unity8 session running under unity-system-compositor falls under (2).

A video player playing a YUV stream that might also want to throw some
RGB-rendered UI over it is also (2); a video player that has some chrome
around a video widget is (1) and (2).

The “embed bits of other applications in our window” requested on
https://bugs.launchpad.net/mir/+bug/1230091 is firmly in (2).

As I see it, there are also two classes of problem:
a) How is the rendering loop coordinated between parent and child - does
the parent need to swap each time the child has new rendering, or can
the child swap independently, or are both modes available? What happens
if a child also wishes to embed children of its own?

This is the only concern for the type (2) use-case.
 
b) How is input handled? Does the parent need to proxy input events and
forward them on? How does enter/leave notification work for the parent
and child? Can the child return events to the parent? Etc.

This is necessary for the type (1) use-cases, and also seems to be the
hairy bit.

Weston (but not yet Wayland) currently partially solves (2) with the
subsurfaces protocol, which has chosen the “no child rendering appears
until the parent swaps” approach, and doesn't handle out of process
renderers at all.

For full out-of-process rendering, and for type-1 usecases, the my
understanding of the current state of the art is that the parent should
become a Wayland compositor itself. This seems a bit of a co-out to me,
and doesn't really solve case 2; however, this area is gnarly, so it
might prove to be the best solution.

For X, the relevant prior-art is XEMBED¹, in all of its
map-an-invisible-1x1-window-to-work-around-X11's-focus-model glory.

¹: http://standards.freedesktop.org/xembed-spec/xembed-spec-latest.html

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 25 Nov 2013 15:00:26 +0800

(a) What's the use-case for needing to synchronize parent/child 
rendering? I'm thinking most use-cases don't need synchronization 
between the two clients. The server already ensures there's always being 
a buffer to render (without blocking) and without tearing.

(b) Why wouldn't we just deliver input to whatever is on top and 
handling input? Any events not handled by the child/subsurface can work 
down the stack (to the parent and beyond). Although I don't think we 
have the client API ability yet to designate an "input area" or to tell 
the server to replay events to the lower level.

---

From: Thomas Voß <[email-masked]>
Date: Mon, 25 Nov 2013 08:12:57 +0100

> be aspects of the same problem.

There is a security relevant issue here, too. Trusted helpers as well
as input methods should have a way to
"compose" UIs across process boundaries, too. See
https://wiki.ubuntu.com/Security/TrustStoreAndSessions for an
introduction to the problem.

>From my pov, for out-of-process rendering approaches, the rendering
and input handling should be opaque
to top/higher-level compositors, with a compositor only requiring
knowledge of its direct children. However, I would think that we
should avoid forcing a compositor model on applications like chrome.
They already have app-local compositors running, together with event
handling. What we should offer (and commit to) is a way to stream
pixel data across process boundaries. Candidate solutions could be:

(1.) http://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external.txt:
We use this extension today on Ubuntu Touch for streaming data from
decoded video and from the builtin cameras.
(2.) EGLStream with a texture sink (somewhat in the future).

For convenience, we could see if we could offer applications a way to
become a Mir compositor (ideally by leveraging Gerry's work on
scenegraph integration). But as a bare minimum, I would think that a
well-defined of streaming textures solves the immediate use-cases for
applications that already have an app-local compositor.

> map-an-invisible-1x1-window-to-work-around-X11's-focus-model glory.

Ouch, that hurts :) And this is one of the reason why the compositor
hierarchy should be opaque from my POV.

Cheers,

  Thomas

> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/mir-devel
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 26 Nov 2013 10:29:31 +1100

> (2.) EGLStream with a texture sink (somewhat in the future).

Right - getting pixel data across process boundaries is the easy bit, as
long as you're ok with requiring the receiver of the pixel data to do
all the work. Which is acceptable (but not great) for Chrome, as they've
already that.

The interesting part is when you want to optimise.

Say you've got a video player called Totem that wants to display
subtitles on the video; this is fullscreen, running in Unity8, under
unity-system-compositor.

You're running this on your tablet, which as luck would have it has a
YUV plane and an ARGB plane.

In the just-getting-pixel-data-across world, Totem wakes up whenever the
decoder provides it with a frame, composites the YUV frame and its RGB
pixel data onto Totem's surface; calls SwapBuffers, and Unity8 passes
that on to u-s-c, which swaps it to the display (as everything should be
eligable for bypass).

In an *ideal* world, Totem sets up its video stage, specifying the
external image stream for the lower level, and making an overlay surface
for its subtitles. Totem wakes up each time it needs to change the
subtitle surface, calling SwapBuffers on that. U8 passes that straight
to u-s-c, which puts it on the ARGB plane. The decoded image stream gets
passed straight to U8, which passes it straight to u-s-c, which puts it
on the YUV plane.

Basically, the later you're able to do the compositing step the more
optimisation opportunities there are. This is reasonably important for
us, because our default setup is going to involve nesting U8 under
u-s-c.

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 26 Nov 2013 10:40:07 +1100

> a buffer to render (without blocking) and without tearing.

Most usecases I can think of, yes.

Off the top of my head, having a frame-accurate count overlay on top of
video would be one use-case.

Another, probably more common, one would be animations involving the
parent/child interface - think expanding the size of the embedded area.

> the server to replay events to the lower level.

Right. We wouldn't do this now because we don't have any ability to do
this now :).

And this obviously only works if the model *is* a stack, which I don't
believe it will be in general; it'll be a tree. An obvious example is a
browser window with two different pieces of flash embedded.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 26 Nov 2013 09:48:50 +0800

If you want to generalize Scene beyond just a stack (and I think we all 
do) then even a tree is too specific. I think the complete 
generalization is simply a queryable Scene interface which then lets the 
implementation work out the real mechanics:

Surface surface_on_top_at(Point p)
Surface next_surface_below(Surface above, Point p)

Thus you're not restricted to saying there's only one surface on top. Or 
that there's only one Surface immediately below it. The answer also 
depends on the location p. There's a world of possibility that opens up 
when you throw away the notion of a linear stack.

Although subsurfaces would have a single parent, of course.

On 26/11/13 07:40, Christopher James Halse Rogers wrote:
> And this obviously only works if the model*is*  a stack, which I don't