Subject: Client API philosophy

From: Christopher James Halse Rogers <[email-masked]>
Date: Mon, 10 Nov 2014 13:48:12 +1100

Hey all.

We're getting into the meaty bit of client API support I thought I'd 
write some things about API design philosophy to see if they're 
controversial.

*) An API with lots of functions that do one thing each is simpler than 
an API with few functions that do different things depending on other 
state.

*) A function with unclear semantics is harmful

*) Mir will not have an external reference for window management 
policy. Window management semantics will be associated with the API. 
There is no equivalent for 
http://standards.freedesktop.org/wm-spec/wm-spec-1.3.html outside the 
client API documentation.

*) The client API should be implementing policy, not mechanism.

*) Where state transitions are allowed, it should be possible to 
transition from one valid state to another valid state without 
temporarily being in an invalid state.

Some thoughts.

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 10 Nov 2014 11:31:55 +0800

Definitely controversial; factually false or easily arguable.

Sounds like a response to one of my merge proposals. So please put 
arguments in the code reviews...

On 10/11/14 10:48, Christopher James Halse Rogers wrote:
> *) Mir will not have an external reference for window management policy.
> Window management semantics will be associated with the API. There is no
> equivalent for http://standards.freedesktop.org/wm-spec/wm-spec-1.3.html
> outside the client API documentation.
>

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 10 Nov 2014 12:02:07 +0800

BTW, I do actually agree with half of your points. Just feel that 
over-generalising in a discussion like this one is counter-productive. 
Because it always is (<- over-generalisation).

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Mon, 10 Nov 2014 15:10:57 +1100

> arguments in the code reviews...

It indirectly is, but also in response to some discussion on IRC of one 
of your merge proposals.

I thought it would be valuable to work out what top-level philosophies 
we agree on and discover where we differ. Code reviews make a poor 
venue for this, as they're not as visible and are primarily concerned 
with the concrete code under discussion.

If we disagree on the measurement criteria it's very difficult to reach 
consensus on the code :)

>> policy.
>> equivalent for 
>> http://standards.freedesktop.org/wm-spec/wm-spec-1.3.html
> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/mir-devel

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 10 Nov 2014 12:24:43 +0800

We frequently disagree on the measurement criteria :)

Also, IMHO code reviews provide a more structured place for targeted 
discussions. And that helps to focus on facts, with less hand-waiving.

On 10/11/14 12:10, Christopher James Halse Rogers wrote:
> we agree on and discover where we differ. Code reviews make a poor venue
> for this, as they're not as visible and are primarily concerned with the
> concrete code under discussion.
>

---

From: Alan Griffiths <[email-masked]>
Date: Mon, 10 Nov 2014 09:58:21 +0000

> arguments in the code reviews... 

There's a good reason to discuss this outside of a specific code review:
we need to agree the "big picture".

There is an apparent disagreement about the approach to window
management policy and that affects the review of any and all MPs in this
area.

I've always understood the intent to be that Mir enables shells (in
general and specifically unity8) to implement policies about how things
should be presented. It is far easier for a shell to provide a policy
around, say "menus" if it is asked to "show a menu" than if it is asked
for a window, then asked to "parent" it, then asked to position it, etc.
With this approach there is never any point at which the server knows
what the client intends.

If we intend to push the presentation policy out to the client toolkits
then they will provide inconsistent (a.k.a. incorrect) policy
implementations. (Especially if, as we should hope, there are multiple
shells written using the Mir library that implement policies differently.)

---

From: Thomas Voß <[email-masked]>
Date: Mon, 10 Nov 2014 11:01:13 +0100

On Mon, Nov 10, 2014 at 10:58 AM, Alan Griffiths <
[email-masked]> wrote:

> ​
> management policy and that affects the review of any and all MPs in this
​+1.
​

> shells written using the Mir library that implement policies differently.)
​To add here: Policy implementations have to live server-side. Otherwise,
as Alan points out correctly, enforcing any sort of default behavior is
almost impossible, or tedious at best.

Thomas

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 09:33:54 +0800

We are actually in violent agreement on "policy" and conflating two 
different issues. So please, let's separate them :)

It is indeed up to the server/shell to dictate policy, particularly as 
it can and will vary between shells/modes of a shell. Anything invalid 
is returned as an error to the client API, or in the form of a 
non-blocking API:

   1. asynchronous set feature A = B
   2. optional wait and get feature A, and check it was changed to B or 
something else dictated by shell policy.

What we must not do is try to enforce policy via client function 
prototype design. Because policy changes not just between shells, but 
even between modes of a shell (e.g. we aim to unify Unity8 desktop with 
touch I think).

The confusion here is coming from some people thinking that a flexible 
API prevents strong enforcement of policy. It does not.

P.S. "menus" are explicitly not a window type right now (as copied from 
the WM design docs). So it's possibly assuming too much to mention the 
the word "menu" in the client API. Although we possibly could - rename 
popover?

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 09:48:23 +0800

On the second issue of client API design, it's useful to point out why 
the menu example is not a good argument:

1. Depending on your shell, and its current mode, a "menu" might not 
have a relative position dictated by the window position, but get 
moved/modified elsewhere on the screen. Think about phones where long 
menus/combos get converted to a wheel/widget in the centre of the screen 
(mostly phones).

2. Relative position is useful to other window types too. For example 
the decorations-next design of title bars, but also embedded GL windows 
or accelerated video in a browser. So with multiple features requiring 
relative placement, you propose each new feature gets a new client 
function, which is mostly redundant with the others? That's crazy for 
three reasons:
   (a) Redundancy in the API is unwanted effort, particularly for 
maintenance. It scales poorly.
   (b) Each function, by its name, is tied to a window type (e.g. 
"menu") and so could easily become deprecated as desired 
types/functionality changes.
   (c) Your proposed API that ties a "menu" to (x,y) now has 
unused/ignored parameters in phone (or other) shells that ignore the 
(x,y) for menus (see #1 above).

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 09:56:02 +0800

May I ask if you have a better design for any functionality, please 
demonstrate it. You don't have to go to the trouble of a working merge 
proposal... Just suggest function prototypes.

Then there's less imagination required, less confusion that we're 
arguing when we're agreeing, and more solid reasoning.

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 11 Nov 2014 13:03:27 +1100

On Tue, Nov 11, 2014 at 12:33 PM, Daniel van Vugt 
<[email-masked]> wrote:
> It is indeed up to the server/shell to dictate policy, particularly 
> as it can and will vary between shells/modes of a shell. Anything 
> invalid is returned as an error to the client API, or in the form of 
> a non-blocking API:
> 
>   2. optional wait and get feature A, and check it was changed to B 
> or something else dictated by shell policy.
> 
> even between modes of a shell (e.g. we aim to unify Unity8 desktop 
> with touch I think).

Equally, we must enforce policy via client function prototype. 
Otherwise you can't write a client and expect it to work against any 
sane shell.

Functionally, in the best case¹, this will end up with is an 
out-of-tree specification of window management semantics that clients 
and shells are expected to adhere to.

¹: If Mir is actually used by things other than Unity 8. If Mir 
remains just Unity 8's toolkit then Unity 8's behaviour is obviously 
the specification of Mir window management.

> 
> The confusion here is coming from some people thinking that a 
> flexible API prevents strong enforcement of policy. It does not.

Right, but we want more than strong enforcement of policy. We want to 
be able to write clients that work against any sane shell. If the 
client API doesn't specify what behaviour a client can expect a sane 
shell to have in response to a given API call then that API call can't 
be used by sane clients.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 10:04:57 +0800

I've already explained why that is untrue. Twice.

On 11/11/14 10:03, Christopher James Halse Rogers wrote:
> Equally, we must enforce policy via client function prototype. Otherwise
> you can't write a client and expect it to work against any sane shell.
>
> Functionally, in the best case¹, this will end up with is an out-of-tree
> specification of window management semantics that clients and shells are
> expected to adhere to.
>
> ¹: If Mir is actually used by things other than Unity 8. If Mir remains
> just Unity 8's toolkit then Unity 8's behaviour is obviously the
> specification of Mir window management.
>
>>
> Right, but we want more than strong enforcement of policy. We want to be
> able to write clients that work against any sane shell. If the client
> API doesn't specify what behaviour a client can expect a sane shell to
> have in response to a given API call then that API call can't be used by
> sane clients.
>
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 11 Nov 2014 13:12:20 +1100

On Tue, Nov 11, 2014 at 12:48 PM, Daniel van Vugt 
<[email-masked]> wrote:
> On the second issue of client API design, it's useful to point out 
> why the menu example is not a good argument:
> 
> menus/combos get converted to a wheel/widget in the centre of the 
> screen (mostly phones).

Oh, absolutely. It's fine for the shell to interpret the requested 
coordinates as appropriate for the form-factor. Indeed, we fully expect 
*desktop* shells to interpret the coordinates as appropriate (for 
example, if the requested surface would be mostly offscreen if (x,y) 
were interpreted as its top-left coordinate).

> the decorations-next design of title bars, but also embedded GL 
> windows or accelerated video in a browser. So with multiple features 
> requiring relative placement, you propose each new feature gets a new 
> client function, which is mostly redundant with the others? That's 
> crazy for three reasons:

The APIs are really simple - I'm perfectly happy if they're all 
implemented in terms of integer surface parameters!

And the maintenance effort is traded off between more distinct API 
entry points and easier state validation; if you can't atomically set 
all the necessary attributes on a surface then the surface must 
transition through an invalid state.

If your client API prototypes require all the relevant state to be 
provided up-front it's much easier to verify correctness.

> (x,y) for menus (see #1 above).

Equally, if you *don't* have (x,y) for the menu your client is now tied 
exclusively to the phone and will mysteriously die if you try it on a 
desktop.

The semantic for menus may well be “this pops up a surface associated 
with <rectangle> when appropriate”. That's perfectly fine.

>> as
>> with
>> flexible

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 10:22:21 +0800

There is no invalid state possible, providing all attributes have some 
default value, and all modifications to an attribute are synchronously 
validated (or rejected) on the server side. And it's not even important 
if you allow a slight semantic reordering (i.e. "invalid" state):
    set_parent(a, b);
    set_relative_pos(a, x, y);
vs
    set_relative_pos(a, x, y);  <-- temporary inconsistency
    set_parent(a, b);
so you can treat the inconsistency as an error or support it on the 
assumption that a parent will be provided. Or support it on the 
assumption that parentless relative positioning has meaning (e.g. to 
position a client-rendered panel/dock).

I know the third option is unpopular, but my point is that there are at 
least three options and it really doesn't matter what we choose.

Remember we are writing asynchronous client functions so they don't 
return errors immediately. Rather you can optionally wait for their 
completion and then query what actually happened.

On 11/11/14 10:12, Christopher James Halse Rogers wrote:
> And the maintenance effort is traded off between more distinct API entry
> points and easier state validation; if you can't atomically set all the
> necessary attributes on a surface then the surface must transition
> through an invalid state.
>

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 10:40:35 +0800

So given invalid states are easily rejected by the shell and what mode 
it's in, and given it's unrealistic (predicting the future) and 
undesirable (scales poorly) to design all future shells behaviours into 
separate functions of a client API, I came up with the normalised design 
you see today.

People often get confused about why I mention database normalization in 
API design, but the principles and objectives are the same:
   http://en.wikipedia.org/wiki/Database_normalization

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 11 Nov 2014 13:59:49 +1100

>    http://en.wikipedia.org/wiki/Database_normalization

At least one difference being that the database has a schema which 
enforces the constraints for you. If you take that approach for API 
design it means you need to implement all the constraints yourself; 
you've lost the help of the compiler in doing so.

> entry
> >>>>> management policy and that affects the review of any and all 
> MPs in
> >>>>> this
> >>>>> area.
> >>>>>
> >>>>> around, say "menus" if it is asked to "show a menu" than if it 
> is
> >>>>> asked
> >>>>> for a window, then asked to "parent" it, then asked to position 
> it,
> >>>>> etc.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 11:12:05 +0800

You've forgotten the first point we all agreed on. That is it's up to 
the shell to enforce semantics. The rules will change between shells and 
between modes of a single shell. So you can't use the compiler of your 
client to error-check semantics. They're enforced at runtime according 
to the shell you're connected to.

To try and write a separate client function for each mode of each 
potential type of shell scales very poorly. Client functions would be 
added (and removed) regularly according to the new shells being 
designed. And even in a world with only one shell it still scales poorly 
in number and complexity of client functions. And even if you accept the 
higher number and complexity of client functions, you're cursing the 
client/toolkit author to a larger problem space of "why did my function 
fail?" due to the higher number of parameters to each function.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 11 Nov 2014 15:26:36 +0800

Update: "menu" is indeed a proper type now, in the latest design docs 
(MIR: Surface Management v0.2 (WIP)). So I'll add that and deprecate the 
ambiguous "popover" type.

Turns out when we agreed to scrap the new terminology introduced at the 
start of the year and go back to the old version, that "old" design is 
still different to the even older version that mir_surface_type was 
designed against 1.5-2 years ago.

I also notice a detailed table that shows 7 of the 8 Unity surface types 
need parenting support (see section 3.3). So we should ideally avoid 
writing 7 different parenting functions.

---

From: Alan Griffiths <[email-masked]>
Date: Tue, 11 Nov 2014 14:29:19 +0000

To help resolve this discussion Thomas Voss called a meeting at 9:00UTC
11-11-2014

Those attending:
Thomas Voss
Christopher James Halse Rogers
Alan Griffiths
Alexandros Frantzis
Andreas Pokorny

Apologies:
Daniel van Vugt

1. We started with a recap over some design goals:

1.1 The server should be in control of "policy"
1.2 It should be easy to validate use of the client API

Thomas noted that he saw no need to support re-parenting - there was no
disagreement.

We discussed the several design approaches being discussed - including
the "parenting" proposals from Daniel and the "two-stage-surface-create"
proposal from Chris and variations from various discussions. These were
assessed against the needs of the user of the client API and the needs
of the server to decide presentation policy.

Consensus was reached around the following:

For each type of surface there should be a corresponding "builder" type.
We used creation of a "menu" to illustrate the canonical form of surface
creation (note the names here are illustrative, not prescriptive)...

    // allocate a builder with all required parameters
    MirSurfaceBuilder* menu_builder =
    mir_connection_allocate_menu_builder(connection, /* any required
    parameters (e.g. parent) */);
    ...
    // set any optional attributes
    mir_surface_bulder_set_XXX(menu_builder, xxx);

    // Finally, create the surface
    MirSurface* surface =
    mir_surface_bulder_create_surface_sync(menu_builder);

This approach had the following advantages:

1. The type of surface is determined explicitly
2. Provision of required parameters is validated at compile time
3. MirSurfaceBuilder and MirSurface are opaque types

Note that discovering new "required" parameters is an API break and this
approach ensures affected code is identified at compile time.

Re the "parenting" branches: "parent" is only ever provided as part of
builder creation (and supplied to the server at surface creation time).
Re-parenting isn't to be supported. These branches need reworking
accordingly.

Actions:

Chris: to update his branch to have a separate builder object. (This
also addresses concerns about functions that behave differently between
pre-realize and post-realize.)

Alan: write minutes

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 12 Nov 2014 09:36:24 +0800

Sorry, I got the impression the meeting was off due to nobody replying 
to my meeting replies, and nobody replying to the meeting invite.

Your proposed API is overcomplicated, does not scale and fails to 
support shells other than Unity. I do not endorse it, obviously.

It requires exponentially more effort and maintenance, to achieve a less 
flexible result. I can't stop the team from outvoting me, but I will not 
implement an inferior design that requires unbounded ongoing maintenance 
(supporting new shells, new types, new semantics) when a superior design 
that supports all use cases from day 1 and has zero ongoing maintenance 
is already proposed.