Subject: The case for a "SceneGraph"

From: Alan Griffiths <[email-masked]>
Date: Thu, 18 Jul 2013 11:27:00 +0100

Hi All,

at a recent hangout I took the action to produce some notes on what a
"SceneGraph" means for Mir.

We've had frequent discussions along the lines of "this information
belongs in the SceneGraph, but as we don't have one we'll stick it
<somewhere>". This, so far mythical, SceneGraph fulfills the "model"
role in our core MVC pattern (see design/Architecture.dia in the source
tree) and incorporates functionality currently present in a number of
parts of the system.

The present SurfaceStack implementation is a primitive SceneGraph that
was "the simplest thing that might work" when it was added, but we've
worked around its limitations for long enough to know that it is
simplistic and fails to support the way we want the system to grow. The
SurfaceController was initially intended as the Controller, but I think
it is now clear that the Controller should be a part of the shell.

So, what do we want from a SceneGraph?

Looking at the architecture diagram and the code (which are in
remarkable accord) there are a number of clients to the SceneGraph:

*The compositor:*

This needs access to the buffers to be composited and corresponding meta
information about the way they are to be rendered. It doesn't happen now
(the compositor has logic to determine which buffers to render), but the
SceneGraph should be supplying exactly those buffers and meta
information that the compositor needs.

Requirement: the SceneGraph need to be able to send the buffers (and
compositing metadata) that are visible on a specified output to a
receiver within the compositor.

An implication of this is that the SceneGraph needs to be able to model
the output devices and know which surfaces are associated with each output.

The current implementation has some simple logic that informs the
compositor that changes have happened. We could make this more useful
with a cleverer SceneGraph - e.g. identifying what has changed for which
output.

Side note: We currently implement visibility as a flag on each surface,
but it might equally be a association within the SceneGraph - what is
clear is that it should be part of the way that the SceneGraph passes
buffers for rendering, not part of the metadata.

*Input:*

Input needs to determine where to route events based upon the input
aware areas presented by some surfaces. The code here is undergoing a
migration from an android based input stack to something more integrated
into Mir.

There are two scenarios: keyboard events (which go to the current input
focus) and events that specify a point for which a corresponding input
target has to be found.

Currently the shell conspire with input to keep track of the input
target. (The shell, in its role as Controller, should control the focus,
but the focus information should be in the model - i.e. the SceneGraph.)

I'm not sure of the exact state of the mechanism for position based
input events currently but, to avoid race conditions when the SceneGraph
changes, it should be the SceneGraph that identifies candidate input
targets based on position.

*The Shell:*

As the Controller the shell should be mediating all updates to the
SceneGraph - it gets to decide the policy for surface placement,
ordering, focus changes, etc. This is best done by taking the raw
request and posting the corresponding updates to the SceneGraph.

>From the point of view of the SceneGraph we don't need to distinguish
between the Mir "shell" code and the Unity shell code (the requirements
on the SceneGraph should be the same). In practical terms a lot of
interactions should stay on the C++ side and these probably best fit in
the Mir codebase.

Some examples:

Surface creation: the shell needs to know when sessions create surfaces,
and have control over where they are placed. Currently the Mir
ApplicationSession receives the create_surface from frontend and has
control of the request passed to the SurfaceController: we have two
objects where we need one.

Session creation: this is currently modelled in shell.

Surface association: the shell (and possibly other clients) needs to be
able to navigate between a session and its associated surfaces. The
current SceneGraph has no concept of "session" - this relationship is
currently held within ApplicationSession: this gives ApplicationSession
a mixture of Controller and Model responsibilities.

The SceneGraph should have a concept of "session" as there are
operations it should perform on all the surfaces associated with the
session.

DepthID, z-order, etc. The stuff I've seen implemented around these seem
doesn't leave me with a clear perception of the problem being solved -
maybe someone else can fill out some details. There clearly is a need
for an ordering to the buffers presented for rendering on each output
but, so far, I don't see the need for the complexity we have.

*Summary:*

We need an improved Model and Controller for our SceneGraph - and that
the model should support at least three types of node: sessions,
surfaces and outputs (and understand the relationship between them.)

The representation of outputs in the SceneGraph implies that there is
more interaction with the display configuration than at present. (N.b.
any updates to the display configuration should be mediated by the
Controller/shell.)

We're currently distributing knowledge of our "scene" around compositor,
shell, and input. That is going to cause synchronization issues - we
should move all this knowledge into one place.

The above is just a first, high-level, stab at outlining the
requirements - we clearly need to agree the scope and then drill down
into a lot more detail as the next piece of work.

Alan

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 18 Jul 2013 19:39:45 +0800

Sounds quite reasonable and agrees with some thoughts I've had too.

If there is anything I totally disagree with, I'm too tired to see it 
right now.

I think we have a good starting point now that compositor::Scene has 
landed today. However I would only add one note; that is to be careful 
to not move any significant parts around until you need to. This is code 
that a few of us are actively working on, and we want to minimize conflicts.

- Daniel

---

From: Gerry Boland <[email-masked]>
Date: Fri, 19 Jul 2013 01:58:25 +0200

Hi Alan,
mind if I add some questions & comments from the shell PoV?

> So, what do we want from a SceneGraph?

What shell will need is, for every surface, the following properties:
- x, y, z, width, height
- opacity, visible
- scale
- ability to set more complex transformations/shaders, so can desaturate
surface colours, do more advanced animations, blurs etc.

Shell will implement window management through QML. I would like to have
as much flexibility with surfaces as the Qt Scenegraph gives me with
Items [1] [2].

> remarkable accord) there are a number of clients to the SceneGraph:
<snip>
> targets based on position.

I don't follow your last point. Could you give an example use-case please?

> interactions should stay on the C++ side and these probably best fit in

This confused me too. Can you explain to me where the boundary line
between Mir and Unity shell code lies? What exactly are these
"interactions"?

> operations it should perform on all the surfaces associated with the

So I don't follow why the scenegraph needs this. Shell needs to know the
connection between surfaces and sessions, but scenegraph should just be
in charge of displaying surfaces. What use-cases do you see where shell
will need this extra info in SG?

> but, so far, I don't see the need for the complexity we have.

+1 on this. Simple z-order should be sufficient IMO. I'm aware that
DepthID can help with surface ordering (keeps shell on top, sidestage
can stay on top of main stage), but I don't think it is worth the
trouble. I'd favour the simple API of a sole z-order property.

> into a lot more detail as the next piece of work.

Thanks
-Gerry

[1]
https://qt-project.org/doc/qt-5.0/qtquick/qml-qtquick2-item.html#transform-prop
[2] https://qt-project.org/doc/qt-5.0/qtquick/qml-qtquick2-shadereffect.html

---

From: Michał Sawicz <[email-masked]>
Date: Fri, 19 Jul 2013 11:55:12 +0200

> as much flexibility with surfaces as the Qt Scenegraph gives me with

FWIW every time I think of a list of things we'll need to communicate to
the compositor, it feels like we're doing something wrong. We'll
basically have two SceneGraphs (one of the shell, one Mir-internal) and
will have to keep them in sync.

This, I feel, might make it really difficult to achieve some UX that
we're wanting for. It also makes it difficult to think security when
letting others render to the shell (think video/map in dash, custom
infographics, even OSK).

I'm sure it was discussed before, but can you guys please give me a
summary why it's not possible to just keep the shell's scenegraph (in
our case the Qt/QML one) in play? Or if there maybe is another way for
shell's scenegraph to another process's surface for "internal" composition?

Cheers,
-- 
Michał (Saviq) Sawicz <[email-masked]>
Canonical Services Ltd.

---

From: Daniel van Vugt <[email-masked]>
Date: Fri, 19 Jul 2013 18:02:01 +0800

Yes indeed the shell is meant to implement the scene graph. But that 
implementation is still inside libmirserver. It needs to be pulled out 
of the server library.

Interface:       mir::compositor::Scene
Implementation:  mir::surfaces::SurfaceStack

That implementation should go in the shell, whenever feasible. Though 
there is a reasonable argument for keeping a nice fallback SurfaceStack 
implementation in libmirserver for simple shells.

---

From: Michał Sawicz <[email-masked]>
Date: Fri, 19 Jul 2013 12:40:20 +0200

> implementation in libmirserver for simple shells.

Is what you're saying that it *is* planned to be possible to access
out-of-process surfaces in the shell's (Qt/QML for unity8) scenegraph?

---

From: Alan Griffiths <[email-masked]>
Date: Fri, 19 Jul 2013 12:34:53 +0100

> shell's scenegraph to another process's surface for "internal" composition?

Mir components have a dependency on the scenegraph to provide them with
information. This dependency exists in several contexts - e.g. system
compositing and session compositing. There needs to be a better
abstraction of these needs within Mir than exists at present because we
are beginning to encounter issues like
https://code.launchpad.net/~robertcarr/mir/session-transactions/+merge/175669.

As to the suitability of the Qt scenegraph: the compositor, in
particular, needs fast response to its queries - so I'm doubtful about
implementing them across language boundaries. (I also suspect that it
would be hard to satisfy all the functional requirements in Mir this way.)

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 22 Jul 2013 09:45:03 +0800

Sorry, no. It's all in-process and (so far) assumed that the shell pid 
== server pid.

---

From: Gerry Boland <[email-masked]>
Date: Tue, 23 Jul 2013 18:01:12 +0200

> would be hard to satisfy all the functional requirements in Mir this way.)

Qt's Scenegraph is C++. It offers internal APIs to extend it. I have to
ask was it ever actually seriously considered? Just claiming it will be
slow and won't do everything we need isn't much of an argument.

It is a bit late in the day for this discussion, but unless I'm
mistaken, we've never really had it. I'm aware that changing direction
at this stage would be tough though.

I am concerned about having 2 independent scenegraphs, one for Mir and
one for shell (Qt). Here are my reasons:

1. Duplication of effort
Why not use an existing, flexible, well tested and performant
scenegraph, instead of writing our own from scratch?

2. Synchronicity of 2 scenegraphs
Unity8 will be animation heavy. Switching windows will involve animating
both shell elements and mir surfaces. QML will be controlling those
animations. For each frame am I guaranteed that the Qt scenegraph and
the Mir scenegraph will be in perfect sync?

3. Complex visual animations
Right now I wrap Mir's Surface API with a QQuickItem so the surface can
be animated with QML. The surface is rendered by Mir, but properties of
that surface can be manipulated by QML, so it is easily animated. This
works for simple properties like position, opacity... but defining more
complex transformations (for example a shader effect) on surfaces will
be very tough to integrate into QML's animation framework.

One great advantage of QtWayland is that the surface's texture itself is
pulled into the Qt scenegraph, and wrapped in a QQuickItem, so all
graphical effects available to QML can be applied to the surface.

So that's my question: why not emulate QtWayland's approach and use Qt's
scenegraph and renderer?

Thanks
-Gerry

---

From: Thomas Voß <[email-masked]>
Date: Tue, 23 Jul 2013 19:45:14 +0200

>> would be hard to satisfy all the functional requirements in Mir this way.)

Not entirely sure where the notion of a language boundary comes from tbh.

The idea is simple: We collect all of Mir's internal requirements
towards the scenegraph into interfaces. All internal logic works in
terms of these interfaces and the implementation is injected from the
outside. This could be a default implementation for simple shells
(i.e., not Unity), e.g., the system compositor. Unity then is free to
come up with a simple glue layer that implements Mir's interfaces for
the scenegraph leveraging the existing Qt scenegraph. With that,
surfaces would be available as first class citizens to Unity, and Mir
is happy.

> slow and won't do everything we need isn't much of an argument.

I do agree. IIRC, Qt's scenegraph leverages OpenSceneGraph internally,
and that is one fast implementation :)

> be very tough to integrate into QML's animation framework.

+2, I do share these concerns.

> graphical effects available to QML can be applied to the surface.

+1, it will help us to accelerate shell development significantly.

Agreed, I really like this
> So that's my question: why not emulate QtWayland's approach and use Qt's

+1.

Thanks,

  Thomas

> Thanks
> -Gerry
>
> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/mir-devel

---

From: Kevin Gunn <[email-masked]>
Date: Tue, 23 Jul 2013 16:03:35 -0500

hey all -
Using Qt scenegraph sounds good to me....but one (or 2) question(s) though.
In our long term plans of rootless X support, won't this mean the X apps
have to be effectively be Qt windows to be treated as a node on the Qt
scengraph ?
e.g. legacy Xapp -> Xrender -> Qtwindow/node -> mir compositor (arbitrated
by unity8/qt shell)

In the case of unity8 does this preclude native mir clients ? (or at least
will they have to "do something" special to be considered part of the
scenegraph)

maybe its not that big of a concern....from Qt scenegraph doc
"Even though the node tree is mostly built internally by the existing Qt
Quick QML types, it is possible for users to also add complete subtrees
with their own content, including subtrees that represent 3D models."
br,kg

On Tue, Jul 23, 2013 at 12:45 PM, Thomas Voß <[email-masked]>wrote:

> > Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel
>
> --
> Mir-devel mailing list
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel
>

---

From: Gerry Boland <[email-masked]>
Date: Wed, 24 Jul 2013 11:11:45 +0200

> surfaces would be available as first class citizens to Unity, and Mir

I'm  glad to have this confirmed, as I had suspected this was possible,
but I wasn't clear if it was considered the correct "thing to do".

What cast doubt for me is that in Mir, the interfaces and the default
implementation are freely mixed, and so me writing my own replacement
implementation feels like I'm overriding a significant chunk of Mir -
and thus doing something wrong.

I would find a stronger divide between the interfaces and the default
implementation more reassuring - would moving the interfaces into a
separate directory from the implementation be an idea?

<snip>

>> So that's my question: why not emulate QtWayland's approach and use Qt's

Ok, I'm delighted you approve. I think this is a task to pursue.
Thanks!
-Gerry

---

From: Gerry Boland <[email-masked]>
Date: Wed, 24 Jul 2013 11:26:14 +0200

> e.g. legacy Xapp -> Xrender -> Qtwindow/node -> mir compositor (arbitrated

There is no need for X apps to have anything to do with Qt. Xrender will
supply Mir with a pixmap/texture, and that texture will be added as a
node to Qt's scenegraph.

So as far as Qt's scenegraph is concerned, an application is just
another texture to be rendered.

> will they have to "do something" special to be considered part of the

Mir does all the hard work for this, so shell just gets a texture per
client, and it does not matter to shell if that client is X-based or
native Mir.

> with their own content, including subtrees that represent 3D models."

Qt's scenegraph does give us a lot of flexibility. I believe with
QtWayland it merely adds a node with each application texture to the SG
tree. I think we can do the same. I do need to study QtWayland more closely.
-G

---

From: Robert Carr <[email-masked]>
Date: Wed, 24 Jul 2013 16:28:40 -0700

Trying to add my two cents. I agree we need this data structure and set of
interfaces that is outlined (roughly) in the original email. It is true
that in interface presented to the compositor, this component takes on a
scene graph role...I think that is not the problem we are trying to solve
at the moment though.

It seems to me our most presenting problem is synchronization issues due to
distribution of state. Essentially several parts of the system (shell,
surfaces, input) each maintain some internal model which corresponds to a
view of the surface stack. The lack of an authoritative source on state
makes synchronization difficult. Likewise, it makes it unclear how to
implement some really common patterns for the shell (i.e. simple
notify-interfere patterns), and I think has lead to the sort of
'anti-pattern' where the shell has to subclass particular factory
interfaces to view certain state.

So, I think we need to rework the mir interfaces around some sort of multi
index data structure (perhaps with some sort of transactional support) to
fulfill these requirements. The difficult challenge is the design of the
interfaces between this data store, and other mir components/the shell.
Given this I expect the data structure implementation to be trivial.

As for the Qt scene graph.

I think it is easy to say "mir will provide these interfaces, and the Qt
scene graph will implement them". It's difficult for me to understand what
this means though. Are we discussing using the Qt scene graph for
rendering? or only as a data store? Either way what sort of approach can we
use to integrate it with the compositor and/or  renderer (accounting for
features like, hardware overlays, multimonitor, and composition bypass). In
doing so are we expecting performance gains? We should consider that the
usage profile optimized for in open scene graph is likely quite different
than the usage profile exposed by Mir. It would be surprising to me if a
custom renderer tailored to Mir were not more performant.

Another potential benefit thats discussed is ease of interopability with
QML. I think maybe we are forgetting the whole picture here though. If the
idea is that different parts of the mir system communicate through this set
of interfaces implemented by the Qt scene graph, then how can we really
allow QML to skip these interfaces and deal directly with the scene graph
without losing our architecture for synchronization?

I don't mean to raise FUD but it is unclear enough to me that I would not
know the first step if assigned it as a task (My first question would be,
does ms::Surface implement QQuickItem? If not, how do we avoid two copies
of the SurfaceStack). I also think that, ease of applying graphical
effects, is not really something holding us up at the moment, or something
likely to be a significant drain on development resources any time soon.

So anyway, I think we should focus on the data structures and interfaces
around the surface store in Mir before spiking on using the Qt scene graph.
I'll try and write a second email soon on thoughts im starting to develop
about this surface store.

On Thu, Jul 18, 2013 at 3:27 AM, Alan Griffiths <
[email-masked]> wrote:

> <somewhere>". This, so far mythical, SceneGraph fulfills the "model" role
> in our core MVC pattern (see design/Architecture.dia in the source tree)
> and incorporates functionality currently present in a number of parts of
> the system.
>
> The present SurfaceStack implementation is a primitive SceneGraph that was
> "the simplest thing that might work" when it was added, but we've worked
> around its limitations for long enough to know that it is simplistic and
> fails to support the way we want the system to grow. The SurfaceController
> was initially intended as the Controller, but I think it is now clear that
> the Controller should be a part of the shell.
>
> Looking at the architecture diagram and the code (which are in remarkable
> accord) there are a number of clients to the SceneGraph:
>
> *The compositor:*
>
> SceneGraph should be supplying exactly those buffers and meta information
> that the compositor needs.
>
> compositing metadata) that are visible on a specified output to a receiver
> within the compositor.
>
> compositor that changes have happened. We could make this more useful with
> a cleverer SceneGraph - e.g. identifying what has changed for which output.
>
> but it might equally be a association within the SceneGraph - what is clear
> is that it should be part of the way that the SceneGraph passes buffers for
> rendering, not part of the metadata.
>
> *Input:*
>
> Input needs to determine where to route events based upon the input aware
> areas presented by some surfaces. The code here is undergoing a migration
> from an android based input stack to something more integrated into Mir.
>
> Currently the shell conspire with input to keep track of the input target.
> (The shell, in its role as Controller, should control the focus, but the
> focus information should be in the model - i.e. the SceneGraph.)
>
> I'm not sure of the exact state of the mechanism for position based input
> events currently but, to avoid race conditions when the SceneGraph changes,
> it should be the SceneGraph that identifies candidate input targets based
> on position.
>
> *The Shell:*
>
> SceneGraph - it gets to decide the policy for surface placement, ordering,
> focus changes, etc. This is best done by taking the raw request and posting
> the corresponding updates to the SceneGraph.
>
> between the Mir "shell" code and the Unity shell code (the requirements on
> the SceneGraph should be the same). In practical terms a lot of
> interactions should stay on the C++ side and these probably best fit in the
> Mir codebase.
>
> Some examples:
>
> control of the request passed to the SurfaceController: we have two objects
> where we need one.
>
> able to navigate between a session and its associated surfaces. The current
> SceneGraph has no concept of "session" - this relationship is currently
> held within ApplicationSession: this gives ApplicationSession a mixture of
> Controller and Model responsibilities.
>
> The SceneGraph should have a concept of "session" as there are operations
> it should perform on all the surfaces associated with the session.
>
> maybe someone else can fill out some details. There clearly is a need for
> an ordering to the buffers presented for rendering on each output but, so
> far, I don't see the need for the complexity we have.
>
> *Summary:*
>
> We need an improved Model and Controller for our SceneGraph - and that the
> model should support at least three types of node: sessions, surfaces and
> outputs (and understand the relationship between them.)
>
> The representation of outputs in the SceneGraph implies that there is more
> interaction with the display configuration than at present. (N.b. any
> updates to the display configuration should be mediated by the
> Controller/shell.)
>
> shell, and input. That is going to cause synchronization issues - we should
> move all this knowledge into one place.
>
> The above is just a first, high-level, stab at outlining the requirements
> - we clearly need to agree the scope and then drill down into a lot more
> detail as the next piece of work.
>
> Alan
>
> --
> Mir-devel mailing list

---

From: Robert Carr <[email-masked]>
Date: Wed, 24 Jul 2013 16:34:41 -0700

Sorry if there's a bit of a rough-edged tone in the mail. I feel like I
don't understand what people are talking about with using the Qt
scene-graph and it makes me feel like I don't know what I am doing. I am
just insecure, not hostile :)

On Wed, Jul 24, 2013 at 4:28 PM, Robert Carr <[email-masked]>wrote:

> It seems to me our most presenting problem is synchronization issues due
> to distribution of state. Essentially several parts of the system (shell,
>> worked around its limitations for long enough to know that it is simplistic
>> and fails to support the way we want the system to grow. The
>> SurfaceController was initially intended as the Controller, but I think it
>> is now clear that the Controller should be a part of the shell.
>>
>> the model should support at least three types of node: sessions, surfaces
>> and outputs (and understand the relationship between them.)
>>
>> more interaction with the display configuration than at present. (N.b. any

---

From: Michał Sawicz <[email-masked]>
Date: Thu, 25 Jul 2013 03:40:18 +0200

W dniu 25.07.2013 01:28, Robert Carr pisze:
> I don't mean to raise FUD but it is unclear enough to me that I would
> not know the first step if assigned it as a task (My first question
> would be, does ms::Surface implement QQuickItem? If not, how do we avoid
> two copies of the SurfaceStack). I also think that, ease of applying
> graphical effects, is not really something holding us up at the moment,
> or something likely to be a significant drain on development resources
> any time soon.

The biggest benefit that I would see from this is the ability to
composite a surface on top of another, treating it as any other Item in QML.

My usecase: being able to render $your_favourite_part_of_the_shell_ui
out-of-process, while maintaining all of the transitions & animations
designed for that part of the UI.

Reason: stability, security, flexibility. This would mean, that, for
example, the video player could be embedded in the dash video preview
without the risk that it would bring down the shell if the video file
was broken.

There's only so much that we can do to a surface between shell and mir
to not grow the amount of communication. Obviously geometry, opacity,
scale are the most important things, but they're also the easiest. More
complex things like rotation, darkening, desaturation, not to mention
blur, will have to be implemented outside of QML and just "driven" by
some custom properties of the surface item in the QML tree. But we're
limited to whatever mir and/or unity-mir implement.

Don't get me wrong, I'll understand if you say "this can't be done"/"is
not feasible". I just want you to think about why is that, and if maybe
we can think of making it possible / feasible.

g'night,

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 25 Jul 2013 19:45:23 +1000

> without the risk that it would bring down the shell if the video file

This is something that we have to support anyway; roughly all video
players work this way. You'd have an empty dummy Item and have the video
decoder fill that.

Of course, we don't have this implemented, nor do we have a concrete
plan to implement this. It might be implemented with EGLStreams, or by
sharing surface IDs across process, or somehow.

We do need to implement it, though, and it'll solve your
embed-out-of-process-thing use-case.