Subject: mir's internal renderer + other shells

From: Kevin DuBois <[email-masked]>
Date: Fri, 28 Mar 2014 12:48:05 -0700

Hello mir folks,

We have 4 users of mir in-flight right now:
1) unity8 (driving at using QtSG)
2) USC (using the default mir implementation)
3) the demo shell (using the default mir implementation, and overriding its
functionality in some areas)
4) the demo server (using the default mir animation)

I mostly want to think about 2, 3 and 4 at this email discussion, that is,
how do we arrange for sharing the default mir compositor implementation
with USC and the demo shell. (unity8 has its own ball rolling already with
an effective fork of the mir implementation)

Now USC is obviously important because its in production, and the demo
shell is important too, because its making sure its easy to write a
sensible shell from scratch.

Now, 'easy to write a sensible shell from scratch' is a bit vague, we
should hash out what that means in this email chain.

Does it mean that you don't have to write GL? [1]
Does it mean that you don't have to write common GL algorithms? (eg,
tesselation, drawing, texture stuff)
Does it mean that we give some pretty basic, simple interfaces that you can
plug your stuff in?
Does it mean that when mir's compositor is overridden libmirserver.so does
not touch GL state at all?

kdub's opinion:
//begin
Its kinda tricky to think about; I like to approach it from the perspective
of someone who doesn't know much about mir's internals, but has written a
GL game/program before and wants to try their hand at writing a shell, or
perhaps someone with toolkit/GL experience who wants to try a shell. [2]

Given this, what I would want is mir to handle all the junk about clients,
ipc, buffer swapping, etc. I'd just want to write GL; my own shaders, my
own algorithms, my own GL state. [3] I wouldn't really be interested in
using mir's GL stuff (triangle tesselation? bah! I want to write zany
jigsaw tessellations and name the vertex attrib what I want to name it!)

So given this, I'd like to see the demo shell forking the USC rendering
functionality at some point, not sharing very much at all. QtSG/unity8 is
already forking the rendering functionality. This kinda drives against the
'never duplicate code' instinct, but the 3rd party shell writer is probably
/only/ interested in duplicating code.

It also makes the USC/production default renderer implementation the
lightest possible implementation one can get away with, while having the
demo shell's implementation showing a more high power implementation.

So maybe, we have to have 2 vertex/fragment shaders in the mir code base,
as opposed to one set like we have now, but in the end, our interfaces are
better for it, and the 3rd party implementers have an easier time
bootstrapping too.
//end

Interested to hear other people's thoughts, I'd guess there's a few
different 'how easy is easy?" answers floating around. Striking the right
balance will get other shell rolling easily, without us having to support
the myriad things in the USC implementation.

Cheers,
Kevin

[1] we already have an approach aimed at the minimal-GL with
unity8/QtSG/qml effort
[2] There are of course lots of advanced coders who might be interested
(like a phone vendor), but I tend to hearken back to my compiz fusion days
https://www.youtube.com/watch?v=9GtMu5afIKg
[3] Obviously, this is idealistic, the serious shell coder will eventually
wade into more detail about the internal interactions of mir and its scene.
But what gets people started coding is seeing its simple to plug your own
stuff in there and see it hit the screen.

---

From: Mirco Müller <[email-masked]>
Date: Fri, 28 Mar 2014 21:47:17 +0100

Am 28.03.2014 20:48, schrieb Kevin DuBois:
> ...
> Given this, what I would want is mir to handle all the junk about
> clients, ipc, buffer swapping, etc. I'd just want to write GL; my own
> shaders, my own algorithms, my own GL state. [3] I wouldn't really be
> interested in using mir's GL stuff (triangle tesselation? bah! I want
> to write zany jigsaw tessellations and name the vertex attrib what I
> want to name it!)
> ...
    This is pretty much what I would like to encounter, when I'd
consider diving into writing my own shell. Free me from all the boring
stuff and give me all the freedom to go wild with GL.

    Maybe only input could be bit more pre-digested with the option to
also get raw events unprocessed. So by default getting... let's pick an
example... "pinch"-events instead of each single tap and related movements.

Best regards...

Mirco

---

From: Daniel d'Andrada <[email-masked]>
Date: Fri, 28 Mar 2014 19:16:41 -0300

> example... "pinch"-events instead of each single tap and related movements.

I would be careful with that wish list. Before long we can find 
ourselves with a Mir toolkit and I don't think we should be writing 
another nux.
It's pretty clear that any minimally complex/sophisticated shell 
implementation will have to leverage a toolkit to get the job done. You 
simply cannot go very far with pure GL and raw events (you would likely 
end up writing your own micro-toolkit inside your shell code when trying 
to go that way).
I think shell should just get pure GL and raw input events and then the 
implementor can decide which toolkit he is going to plug on top of that: 
be it Qt, clutter, some 3d game engine, the sky is the limit.

Ah, as for the pinch: Qt has PinchArea, clutter has ClutterZoomAction...

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 31 Mar 2014 10:06:28 +0800

This topic sounds like what I've been working on - moving the pieces 
around so that anything and everyone can play nicely with Mir. It's 
really an evolution toward reaching some common interface that Unity8/Qt 
and other pure OpenGL shells could use simultaneously.

 > Does it mean that you don't have to write GL? [1]
At the moment, yes you have to use GL or a toolkit which uses GL (Qt). 
However we are aiming to not even be dependent on GL if something else 
was available.

 > tesselation, drawing, texture stuff)
You can use the built-in drawing methods of GLRenderer (which themselves 
are evolving and subject to change). In the purest sense I think all of 
GLRenderer would become shell-specific. However that's not likely to 
happen while it still contains important boiler-plate logic. I'm slowly 
working to try and push boiler-plate into Mir and GL-bling into the shell.

 > Does it mean that we give some pretty basic, simple interfaces that 
you can plug your stuff in?
Yes, working on that re-"Compositor / Renderer". But I'm conscious that 
we don't want to build a toolkit. We're not building a "toolkit" in the 
normal sense but a set of interfaces for building shells.

 > Does it mean that when mir's compositor is overridden libmirserver.so
 > does not touch GL state at all?
Ideally that should be possible. GL should just be an option anyway. 
Although the GL context is tangled up in the DisplayBuffer design right now.

- Daniel

On 29/03/14 03:48, Kevin DuBois wrote:
> 3) the demo shell (using the default mir implementation, and overriding
> its functionality in some areas)
> I mostly want to think about 2, 3 and 4 at this email discussion, that
> is, how do we arrange for sharing the default mir compositor
> implementation with USC and the demo shell. (unity8 has its own ball
> rolling already with an effective fork of the mir implementation)
>
> Does it mean that we give some pretty basic, simple interfaces that you
> can plug your stuff in?
> Does it mean that when mir's compositor is overridden libmirserver.so
> does not touch GL state at all?
>
> kdub's opinion:
> //begin
> Its kinda tricky to think about; I like to approach it from the
> perspective of someone who doesn't know much about mir's internals, but
> has written a GL game/program before and wants to try their hand at
> writing a shell, or perhaps someone with toolkit/GL experience who wants
> to try a shell. [2]
>
> interested in using mir's GL stuff (triangle tesselation? bah! I want to
> write zany jigsaw tessellations and name the vertex attrib what I want
> to name it!)
>
> functionality at some point, not sharing very much at all. QtSG/unity8
> is already forking the rendering functionality. This kinda drives
> against the 'never duplicate code' instinct, but the 3rd party shell
> writer is probably /only/ interested in duplicating code.
>
> So maybe, we have to have 2 vertex/fragment shaders in the mir code
> base, as opposed to one set like we have now, but in the end, our
> interfaces are better for it, and the 3rd party implementers have an
> easier time bootstrapping too.
> //end
>
> different 'how easy is easy?" answers floating around. Striking the
> right balance will get other shell rolling easily, without us having to
> support the myriad things in the USC implementation.
>
> Cheers,
> Kevin
>
> (like a phone vendor), but I tend to hearken back to my compiz fusion
> days https://www.youtube.com/watch?v=9GtMu5afIKg
> [3] Obviously, this is idealistic, the serious shell coder will
> eventually wade into more detail about the internal interactions of mir
> and its scene. But what gets people started coding is seeing its simple
> to plug your own stuff in there and see it hit the screen.
>
>

---

From: Kevin Gunn <[email-masked]>
Date: Mon, 31 Mar 2014 08:38:38 -0500

On Sun, Mar 30, 2014 at 9:06 PM, Daniel van Vugt <
[email-masked]> wrote:

> around so that anything and everyone can play nicely with Mir. It's really
> an evolution toward reaching some common interface that Unity8/Qt and other
> pure OpenGL shells could use simultaneously.
>
> On kdub's points:
>
>
> However we are aiming to not even be dependent on GL if something else was
> available.
>
>
by this do you simply mean the interface between Compositor & Renderer will
be devoid of GL ?
and the "something else" could be HWC i suppose ?...but even then, we
assume GL as a fallback.

> GLRenderer would become shell-specific. However that's not likely to happen
> while it still contains important boiler-plate logic. I'm slowly working to
> try and push boiler-plate into Mir and GL-bling into the shell.
>
>
yeah, but +1 to what you describe. this actually allows gets that nice
split of "convenience" for those that just need to use boiler-plate....and
then "flexibility" for those that want to go all bling-y.

> Yes, working on that re-"Compositor / Renderer". But I'm conscious that we
> don't want to build a toolkit. We're not building a "toolkit" in the normal
> sense but a set of interfaces for building shells.
>
>
agreed, and i think its good to restate this. I think the creep of "toolkit
into mir" is something we have to be on guard for.

> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/mir-devel
>

---

From: Kevin DuBois <[email-masked]>
Date: Mon, 31 Mar 2014 15:04:29 -0700

I too think we should steer clear of a toolkit, and can assume people want
to write their own GL and GLSL. I'm concerned though that we don't have an
obvious location that we want people to plug in new rendering engines
though.

My big concern right now is that I don't see us rallying around the same
Renderer/Compositor interfaces in order to plug the QtSg and (USC/demo
shells). We're evolving, but divergently. It needs to be obvious what we
want the new-shell-writer to replace. Right now, its not obvious, and we
have the demo shell evolving along the lines of "expand mc::GLRenderer" and
the QtSG stuff evolving more along the lines of "replace the
DefaultDisplayBufferCompositor".

Our examples/ directory is where the new-shell-writer would probably look
first to learn how to do something cool, so we should point them in the
direction we want them to go. There's a split between what we recommend in
our examples, and what we're actually planning on using for the unity8
shell.

The best way I can see to rectify this is to allow for people to rewrite
the DefaultDisplayBufferCompositor, meaning we have a USC compositor
implementation, a fancy demo shell implementation, and a QtSG
implemenation. [1]
We are object-oriented after all :) This lets us arrange our interfaces
flexibly, we have a the different needs between the three.

The next step for driving at the triad of implementations would just be to
1) Continue making DefaultDisplayBufferCompositor smaller and smaller
(which I'm working on anyways to get overlays all hooked up)
2) split the GLRenderer somehow, instead of having one implementation with
different internal parts being overridden. We make it two implementations,
or we make it a cobbling together of different useful GL parts (I prefer
the two implementations, we are object oriented. The end result here would
be that mc::Renderer would get buried into the private headers, or be a
header private to examples/. [2] This also has the added benefit that we
don't have latent demo code in USC's call path.

Cheers
Kevin

[1] Last I've heard about QtSG, DefaultDisplayBufferCompositor is the place
that needs replacement). Correct me if things have changed :)
[2] IIRC, historically, mc::Renderer was originally just to test the
interactions between the compositor and the GL code. At some point it was
made public and overridable, and became a much bigger/more important
interface than we originally intended

On Mon, Mar 31, 2014 at 6:38 AM, Kevin Gunn <[email-masked]>wrote:

> by this do you simply mean the interface between Compositor & Renderer
> will be devoid of GL ?
> agreed, and i think its good to restate this. I think the creep of
> "toolkit into mir" is something we have to be on guard for.
>
>
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel
>
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Mon, 31 Mar 2014 23:37:53 +0011

On Tue, Apr 1, 2014 at 12:38 AM, Kevin Gunn <[email-masked]> 
wrote:
> 
> 
> 
> On Sun, Mar 30, 2014 at 9:06 PM, Daniel van Vugt 
> <[email-masked]> wrote:
>> really an evolution toward reaching some common interface that 
>> Unity8/Qt and other pure OpenGL shells could use simultaneously.
>> 
>> On kdub's points:
>> 
>> 
>> At the moment, yes you have to use GL or a toolkit which uses GL 
>> (Qt). However we are aiming to not even be dependent on GL if 
>> something else was available.
>> 
> 
> by this do you simply mean the interface between Compositor & 
> Renderer will be devoid of GL ?

We assume *something* as a fallback. In the medium term we're going to 
want to have a pure-software, super cut-down unity-system-compositor we 
can shove in an initramfs to provide a full kernel-init-to-shutdown 
system compositor. That's going to want to be GL-less.

>> You can use the built-in drawing methods of GLRenderer (which 
>> themselves are evolving and subject to change). In the purest sense 
>> I think all of GLRenderer would become shell-specific. However 
>> that's not likely to happen while it still contains important 
>> boiler-plate logic. I'm slowly working to try and push boiler-plate 
>> into Mir and GL-bling into the shell.
>> 
> 
> yeah, but +1 to what you describe. this actually allows gets that 
> nice split of "convenience" for those that just need to use 
> boiler-plate....and then "flexibility" for those that want to go all 
> bling-y.
>  
>> 
>> > Does it mean that we give some pretty basic, simple interfaces 
>> that you can plug your stuff in?
>> Yes, working on that re-"Compositor / Renderer". But I'm conscious 
>> that we don't want to build a toolkit. We're not building a 
>> "toolkit" in the normal sense but a set of interfaces for building 
>> shells.
>> 
>  

I think it's a reasonable expectation that people will use an existing 
toolkit (or write a new one) for any non-trivial shell - all the 
existing shells have done so (Nux for Unity, St for GNOME Shell, Qt for 
KDE). We should make this expectation explicit.

Hm. Additionally, how does this affect how our shell interfaces should 
look?

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 01 Apr 2014 09:29:58 +0800

I think this is essentially what I was trying to get at in the 
"Compositor / Renderer" thread on 21 March. So we're on the same page.

It's worth noting that the concepts:
   DisplayBufferCompositor
   DefaultDisplayBufferCompositor
   Renderer
   GLRenderer
are all specific to MultithreadedCompositor. If a shell replaces 
"the_compositor()" then it is replacing all of those classes in a single 
blow. That's exactly what qml-demo-shell does so it doesn't know or care 
what a DisplayBufferCompositor or Renderer are. Although, the price 
qml-demo-shell pays for that is lower performance (no bypass) and no 
multi-monitor support, so it's obviously not a permanent solution.

I agree completely if these are to remain useful abstractions, they 
should become useful to Unity too. And that is at the back of my mind 
(despite what it looks like).

- Daniel

On 01/04/14 06:04, Kevin DuBois wrote:
> I too think we should steer clear of a toolkit, and can assume people
> want to write their own GL and GLSL. I'm concerned though that we don't
> have an obvious location that we want people to plug in new rendering
> engines though.
>
> have the demo shell evolving along the lines of "expand mc::GLRenderer"
> and the QtSG stuff evolving more along the lines of "replace the
> Our examples/ directory is where the new-shell-writer would probably
> look first to learn how to do something cool, so we should point them in
> the direction we want them to go. There's a split between what we
> recommend in our examples, and what we're actually planning on using for
> the unity8 shell.
>
> 2) split the GLRenderer somehow, instead of having one implementation
> with different internal parts being overridden. We make it two
> implementations, or we make it a cobbling together of different useful
> GL parts (I prefer the two implementations, we are object oriented. The
> end result here would be that mc::Renderer would get buried into the
> private headers, or be a header private to examples/. [2] This also has
> the added benefit that we don't have latent demo code in USC's call path.
>
> Cheers
> Kevin
>
> [1] Last I've heard about QtSG, DefaultDisplayBufferCompositor is the
> place that needs replacement). Correct me if things have changed :)
> interactions between the compositor and the GL code. At some point it
> was made public and overridable, and became a much bigger/more important
> On Mon, Mar 31, 2014 at 6:38 AM, Kevin Gunn <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>
>
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         This topic sounds like what I've been working on - moving the
>         pieces around so that anything and everyone can play nicely with
>         Mir. It's really an evolution toward reaching some common
>         interface that Unity8/Qt and other pure OpenGL shells could use
>         simultaneously.
>
>         On kdub's points:
>
>
>     and the "something else" could be HWC i suppose ?...but even then,
>     we assume GL as a fallback.
>
>
>         themselves are evolving and subject to change). In the purest
>         sense I think all of GLRenderer would become shell-specific.
>         However that's not likely to happen while it still contains
>         important boiler-plate logic. I'm slowly working to try and push
>         boiler-plate into Mir and GL-bling into the shell.
>
>
>          > Does it mean that we give some pretty basic, simple
>         interfaces that you can plug your stuff in?
>         Yes, working on that re-"Compositor / Renderer". But I'm
>         conscious that we don't want to build a toolkit. We're not
>         building a "toolkit" in the normal sense but a set of interfaces
>         for building shells.
>
>         Ideally that should be possible. GL should just be an option
>         anyway. Although the GL context is tangled up in the
>         DisplayBuffer design right now.
>
>         - Daniel
>
>
>
>             days https://www.youtube.com/watch?__v=9GtMu5afIKg
>             <https://www.youtube.com/watch?v=9GtMu5afIKg>
>         [email-masked] <mailto:[email-masked]>
>         https://lists.ubuntu.com/__mailman/listinfo/mir-devel
>         <https://lists.ubuntu.com/mailman/listinfo/mir-devel>
>
>
>
>     --
>     Mir-devel mailing list
>     [email-masked] <mailto:[email-masked]>

---

From: Alexandros Frantzis <[email-masked]>
Date: Tue, 1 Apr 2014 14:49:14 +0300

On Mon, Mar 31, 2014 at 03:04:29PM -0700, Kevin DuBois wrote:
<snip>
> that needs replacement). Correct me if things have changed :)

QtSG provides its own rendering threads, which are difficult to manage
externally and to disentangle from the actual rendering operation. So,
as a first step at least, the plan is to override the mc::Compositor
completely.

We will evaluate how well this works, and whether it's worth
trying to make QtSG cooperate with the existing MultithreadCompositor
and DisplayBufferCompositor infrastructure (which I would prefer).

Thanks,
Alexandros

---

From: Kevin DuBois <[email-masked]>
Date: Wed, 2 Apr 2014 14:28:56 -0700

Yeah, I hope it can be tied into multithreaded compositor, although I don't
know how tough this is.

We've been on the same page from a high level for quite some time, I guess
it just comes down to planning the low level details on how the code sholud
shape up.

I've been pouring over this a bit, and here's my make-things-nice plan as
far as the demoshell/usc/android overlay work is concerned

1) DefaultDisplayBufferCompositor is becoming less complex. Step one (in
progress) is to make this work on the list instead of operators/filters on
the scene. This also involves fixing the locking scheme we have around the
Scene/Compositor interactions.

2) once we've transitioned to working solely on the Renderablelist, we
should transition Mesa to doing its hardware optimizations (bypass) from
within the DisplayBuffer. [1]

3) at this point, DefaultDisplayBufferCompositor is pretty simple. We can
write an alternative implementation for the demo shell that adds the drop
shadows by simply adding Renderables to the list that the scene is
producing.

4) simplify GLRenderer so it will just draw the renderables given to it
(and not allow for injecting different primitives via the tessellate
override). Android can then use it to draw the Renderables to the screen
when the Android drivers reject the surface as an overlay

Cheers,
Kevin

On Tue, Apr 1, 2014 at 4:49 AM, Alexandros Frantzis <
[email-masked]> wrote:

> place

---

From: Kevin DuBois <[email-masked]>
Date: Mon, 28 Apr 2014 13:13:23 -0700

I appreciate that its difficult to see the end-goal of my plan here when I
try to propose things in digestible/reviewable chunks of a thousand or so
lines at a time. If you're interested (or concerned) about where this is
all going, check out:

http://bazaar.launchpad.net/~kdub/mir/future-default-display-buffer-compositor/view/head:/src/server/compositor/default_display_buffer_compositor.cpp#L49

This branch still has some kinks to work out (mostly lower level hwc stuff
at this point), but hopefully you get a good idea of how the abstraction
for mesa and android makes writing a DisplayBufferCompositor
straightforward.

We still have that nice 'if/else block' around hardware optimizations, it
just now works for android as well as mesa in this branch.

Cheers,
Kevin

On Wed, Apr 2, 2014 at 2:28 PM, Kevin DuBois <[email-masked]>wrote:

> Yeah, I hope it can be tied into multithreaded compositor, although I
> don't know how tough this is.
>