Subject: Clients reading their surface position on screen

From: Gerry Boland <[email-masked]>
Date: Tue, 22 Jul 2014 00:42:42 +0100

Hey folks,
in working on QtCompositor, I stumbled across a problem ([1]).

Autopilot needs to know the position of items in an application
(buttons, etc) in screen coordinates - not surface. It needs that as it
generates inputs via uevent, which are defined in screen coordinates.

To calculate absolute item position, it must know the position of the
application surface in screen coordinates.

However Mir does not give applications this information.

But Autopilot has this requirement, so how can we satisfy it?

Thomi and I chewed this over a bit, and the ideas we had were:

1. unity8 exposes a DBus API which
     a) allows AP to ask "what are the coordinates of the surface with
        PID x
     b) emit signals when surface position changes
   This isn't ideal, as it requires punching holes in client appArmor
confinements just for testing, and would also require a lot of work in
Autopilot to be able to listen to those DBus signals.

2. AP injects events directly into Qt/unity8
   Requires unity8 to be in a "testing" mode. Qt would need lots of work
for it to handle relative coordinates being injected.

3. Mir let the clients find their surface's screen position

But I don't think the Mir team like that final idea (I'm not a fan either).

So I'd like to open this discussion up, and see if the Mir team have
considered this issue and have suggestions/plans.

Thanks!
-Gerry

P.S. for now I've written a hack to fix AP tests with qtComp, so this is
not an urgent topic.

[1] https://bugs.launchpad.net/mir/+bug/1346633

---

From: Thomi Richards <[email-masked]>
Date: Tue, 22 Jul 2014 12:04:54 +1200

Hi,

I commented on the bug, but I may as well comment here:

> 3. Mir let the clients find their surface's screen position

I'm pushing for this solution, for a variety of reasons. The main one is
that the QWidget::mapToGlobal call is what we use to determine global
screen coordinates for each widget. The documentation (
http://qt-project.org/doc/qt-5/qwidget.html#mapToGlobal) says:

Translates the widget coordinate *pos* to global screen coordinates. For
> example, mapToGlobal(QPoint(0,0)) would give the global coordinates of
> the top-left pixel of the widget.
>

There's nothing in there that says "this may not be supported by the
current platform". To my mind, there's a contract here between the
application and Qt, which, if we want to support Qt on mir, has to remain
valid. Autopilot uses this call on X11 platforms, it should be able to use
exactly the same call on mir. Autopilot-qt makes no X11-specific calls, we
*just* use the Qt API.

While autopilot (or more accurately, autopilot-qt) is the broken component
in this case, this is a part of the public Qt API. If it doesn't work on
mir, someone else will have exactly the same problem down the road.

I understand not wanting to expose more information to mir clients than is
strictly necessary. In this case, however, I think the pragmatic approach
is to look at solution #3.

Cheers,
-- 
Thomi Richards
[email-masked]

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 22 Jul 2014 09:15:37 +0800

Sounds like a reasonable request.

As a matter of purity Mir has tried to hide compositor information like 
screen position from clients, which is otherwise a good idea.

Until now we've dithered between two approaches:
   (a) Support arbitrary 3D transformations but only support input 
correctly when surfaces are flat (2D).
   (b) Support arbitrary 3D transformations and support accurate 
coordinate transformation for correct input even when a surface is not 
"flat".

I think we're more going down the road of (b) now -- anpok touched it 
last. In that case you would get a function:
     Point client_coord_to_screen_coord(Point local);
This supports the generalised case where under transformation, a button 
rectangle may in fact no be a rectangle on the screen.

But it's also kind of OK if we just want to assume transformations 
always settle back to 2D and appear flat on the screen for the sake of 
simplified input handling.

- Daniel

---

From: Luke Yelavich <[email-masked]>
Date: Tue, 22 Jul 2014 11:23:09 +1000

> generates inputs via uevent, which are defined in screen coordinates.

Qt's accessibility framework also needs to be able to present this information via at-spi to accessibility tools such as Orca.(1) This is probably something that is more to do with Qt internals working with Mir, but even though this is primarily about autopilot, keeping accessibility requirements in mind when thinking about this is probably also relevant.

Luke

(1) https://blueprints.launchpad.net/ubuntu/+spec/client-unity8-a11y

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 22 Jul 2014 09:26:39 +0800

What we could do is map point-to-point so as to be ready for correctly 
supporting arbitrary 3D transformations of the surfaces being touched, 
but let the shell itself make the assumption that it only ever displays 
things as flat. Thus we can:

    Point top_left = mir_surface_coord_to_screen_coord(0, 0);

to support all future use cases, while also making like maximally simple 
for testers of 2D shells right now :)

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 22 Jul 2014 09:27:47 +0800

"making *life* maximally simple"

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 22 Jul 2014 14:49:30 +1000

On Tue, Jul 22, 2014 at 10:04 AM, Thomi Richards 
<[email-masked]> wrote:
> On Tue, Jul 22, 2014 at 11:42 AM, Gerry Boland 
> <[email-masked]> wrote:
> I'm pushing for this solution, for a variety of reasons. The main one 
> is that the QWidget::mapToGlobal call is what we use to determine 
> global screen coordinates for each widget. The documentation 
> (http://qt-project.org/doc/qt-5/qwidget.html#mapToGlobal) says:
> 
>> Translates the widget coordinate pos to global screen coordinates. 
>> For example, mapToGlobal(QPoint(0,0)) would give the global 
>> coordinates of the top-left pixel of the widget.
> 
> 
> application and Qt, which, if we want to support Qt on mir, has to 
> remain valid. Autopilot uses this call on X11 platforms, it should be 
> able to use exactly the same call on mir. Autopilot-qt makes no 
> X11-specific calls, we *just* use the Qt API

So, Qt actually has a whole bunch of holes where an API just plain 
isn't supported on all platforms but makes no mention of it. This is 
one of them.

That call also fails on Wayland compositors, for example.

This should be fixed in Qt by documenting that this doesn't succeed on 
all platforms, and allowing you to check whether you're on a platform 
where it will.

---

From: Andreas Pokorny <[email-masked]>
Date: Tue, 22 Jul 2014 09:22:03 +0200

Hi,

On Tue, Jul 22, 2014 at 1:42 AM, Gerry Boland <[email-masked]>
wrote:

> [...]

I would like to add a few things that might make this task more problematic.
But I think we can find pragmatic trade-offs and document eventual
limitations.
The compositor could decide to display a surface multiple times on screen.
Consider live previews when switching windows or in task bars. Deform the
surface
in a way that its placement can no longer accurately be described as a 4x4
matrix,
or even two coordinates.

So yes duflus surface_coord_to_screen_coord would solve more of that. Then
the server
can do the calculation - and also allow the compositor to pick one
placement of a surface
or have multiple results..

More limitations:
Since this is used for automated system tests - what happens when the given
location is occluded?
Should we care about the delay between request and response?

Are we doing the right thing here? The system test would then rely on the
positions
reported by applications to trigger an action within that application.

regards
Andreas Pokorny

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 22 Jul 2014 17:52:42 +1000

On Tue, Jul 22, 2014 at 11:23 AM, Luke Yelavich 
<[email-masked]> wrote:
> Qt's accessibility framework also needs to be able to present this 
> information via at-spi to accessibility tools such as Orca.(1) This 
> is probably something that is more to do with Qt internals working 
> with Mir, but even though this is primarily about autopilot, keeping 
> accessibility requirements in mind when thinking about this is 
> probably also relevant.

My memory of our discussions in Malta was that it seemed like a good 
idea for Mir itself to take some part of the role of the at-spi 
registry and have toolkits report the surface-relative positions of 
their widgets. Then Mir can provide (appropriately permissioned) 
accesibility applications the absolute positions of the various widgets.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 22 Jul 2014 15:58:32 +0800

If accessibility and Autopilot can live with a point (centre of a 
widget) rather than needing an accurate region then we can just deal in 
one point per widget. And that keeps anpok's dream alive of perfect 
input mapping in 3D shells :)

On 22/07/14 15:52, Christopher James Halse Rogers wrote:
>> information via at-spi to accessibility tools such as Orca.(1) This is
>> probably something that is more to do with Qt internals working with
>> Mir, but even though this is primarily about autopilot, keeping
> idea for Mir itself to take some part of the role of the at-spi registry
> and have toolkits report the surface-relative positions of their
> widgets. Then Mir can provide (appropriately permissioned) accesibility
> applications the absolute positions of the various widgets.
>
>

---

From: Thomas Voß <[email-masked]>
Date: Tue, 22 Jul 2014 10:01:50 +0200

Hey all,

so reading Gerry's initial mail, it seems to me that autopilot does
not actually want to deal with absolute screen coordinates (and take
on the burden of figuring out multi-monitor, multi-dpi cases on its
own) but instead wants to say: Please Mr. Mir, for this surface with
ID id, send an event of type Y with these attributes. Where Y is most
likely touch events and the attributes are surface-relative
coordinates. With that, we can leverage the existing mir logic, while
keeping AP free of global knowledge.

Cheers,

  Thomas

On Tue, Jul 22, 2014 at 9:58 AM, Daniel van Vugt
<[email-masked]> wrote:
> If accessibility and Autopilot can live with a point (centre of a widget)
> rather than needing an accurate region then we can just deal in one point
> per widget. And that keeps anpok's dream alive of perfect input mapping in
> 3D shells :)
>
>
>
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel

---

From: Thomi Richards <[email-masked]>
Date: Wed, 23 Jul 2014 09:11:38 +1200

Hi,

> keeping AP free of global knowledge.

Actually, that's not the case at all.

We send events through evdev. We do this so events travel through the
entire software stack. We don't want to insert events through mir/unity8
directly.

Anyway, generating events isn't the problem. The problem is figuring out Qt
widget global (screen) coordinates from within the autopilot-qt driver.
Ideally, we'd like to ask Qt "Please Mr Qt, please translate these local
widget coordinates to screen-space coordinates". This is what we do
currently, and it works[1]. I'd really like to not have to deal with mir
directly, since the autopilot-qt driver is complicated enough as it is,
without having to deal with mir directly. We don't currently talk to X11
directly, it'd be a shame to add this low-level code to a library that, to
date, contains reasonably high level code.

RAOF Wrote:

> So, Qt actually has a whole bunch of holes where an API just plain isn't
> supported on all platforms but makes no mention of it. This is one of them.
>
> This should be fixed in Qt by documenting that this doesn't succeed on all
> platforms, and allowing you to check whether you're on a platform where it
> will.
>

This may, or may not all be true, but none of it helps solve the problem :)

I think this conversation is drifting away from the original point. Are
there any strong objections to making the existing Qt API "just work" on
mir?

Cheers,

[1] We already handle multi-monitor configurations. I'm sure some extra
code will have to be written when we have more unusual cases to deal with
(like a tablet plugged in to a large screen, for example), but for now,
this "just works".
-- 
Thomi Richards
[email-masked]

---

From: Luke Yelavich <[email-masked]>
Date: Wed, 23 Jul 2014 08:09:56 +1000

On Tue, Jul 22, 2014 at 05:52:42PM EST, Christopher James Halse Rogers wrote:
> >probably something that is more to do with Qt internals working with Mir,
> >but even though this is primarily about autopilot, keeping accessibility
> >requirements in mind when thinking about this is probably also relevant.
> 
> My memory of our discussions in Malta was that it seemed like a good idea
> for Mir itself to take some part of the role of the at-spi registry and have
> toolkits report the surface-relative positions of their widgets. Then Mir
> can provide (appropriately permissioned) accesibility applications the
> absolute positions of the various widgets.

That was for input event handling IIRC.

Luke

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 23 Jul 2014 09:42:03 +0800

I think we can make QWidget::mapToGlobal() work pretty easily. It just 
requires a new function in libmirclient under the hood first.

That's such a simple task, we should just do it. If it's helpful to 
accessibility then great. But I expect accessibility to require more 
work later.

On 23/07/14 05:11, Thomi Richards wrote:
> Hi,
>
> On Tue, Jul 22, 2014 at 8:01 PM, Thomas Voß <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> Anyway, generating events isn't the problem. The problem is figuring out
> Qt widget global (screen) coordinates from within the autopilot-qt
> driver. Ideally, we'd like to ask Qt "Please Mr Qt, please translate
> these local widget coordinates to screen-space coordinates". This is
> what we do currently, and it works[1]. I'd really like to not have to
> deal with mir directly, since the autopilot-qt driver is complicated
> enough as it is, without having to deal with mir directly. We don't
> currently talk to X11 directly, it'd be a shame to add this low-level
> code to a library that, to date, contains reasonably high level code.
>
>
> RAOF Wrote:
>
>     This should be fixed in Qt by documenting that this doesn't succeed
>     on all platforms, and allowing you to check whether you're on a
>     platform where it will.
>
>
>
> code will have to be written when we have more unusual cases to deal
> with (like a tablet plugged in to a large screen, for example), but for
> now, this "just works".
> --
> Thomi Richards
> [email-masked] <mailto:[email-masked]>
>
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Wed, 23 Jul 2014 13:05:46 +1000

On Wed, Jul 23, 2014 at 7:11 AM, Thomi Richards 
<[email-masked]> wrote:
…
> I think this conversation is drifting away from the original point. 
> Are there any strong objections to making the existing Qt API "just 
> work" on mir?

Fundamentally I think the strong objection is that we cannot make the 
Qt API work *in general*. There is not necessarily a 1-1 mapping of 
client-coordinate-space to screen-space.

Much of the time there will be such a mapping, but we'd prefer to not 
expose an API that's going to mysteriously fail some of the time.

---

From: Thomas Voß <[email-masked]>
Date: Wed, 23 Jul 2014 08:18:01 +0200

On Tue, Jul 22, 2014 at 11:11 PM, Thomi Richards
<[email-masked]> wrote:
> We send events through evdev. We do this so events travel through the entire
> software stack. We don't want to insert events through mir/unity8 directly.
>

Well, if you would only require evdev, we would be fine for Mir.
However, AP leverages a side-channel into
the underlying, whether that is abstracted by Qt or not does not
matter here. We have strong objections supporting such a generic
side-channel by means of adding local <-> global coordinate
translation functions to the Mir client API.

> date, contains reasonably high level code.

Awesome, no need to talk X11 or Mir directly. Adjusting the Ubuntu QPA
should be the way to go here, with the additional benefit of exposing
useful testing functionality to more clients than just Autopilot.

Back to Gerry's original list of alternatives: As far as I understand
it, Unity8 already loads a testability plugin when executed in testing
mode.
That plugin might be a good place to wire up required functionality
and expose a counterpart in the Ubuntu QPA.

@Gerry: What do you think?

> This may, or may not all be true, but none of it helps solve the problem :)

But it does help to understand the reasons for our strong objections:
We consciously want to avoid exposing certain sorts of functionality
via the Mir client API, taking into account the X history. Exposing
global coordinate knowledge is one such area of functionality. I can
see your strive for unification from a testing tool perspective, but
it's not as easy as "just making an API work".

HTH,

  Thomas
> [1] We already handle multi-monitor configurations. I'm sure some extra code
> will have to be written when we have more unusual cases to deal with (like a
> tablet plugged in to a large screen, for example), but for now, this "just
> works".

Well, I was thinking about the upcoming requirements to support
convergence use-cases here.

---

From: Thomas Voß <[email-masked]>
Date: Wed, 23 Jul 2014 08:18:19 +0200

On Wed, Jul 23, 2014 at 5:05 AM, Christopher James Halse Rogers
<[email-masked]> wrote:
> Fundamentally I think the strong objection is that we cannot make the Qt API
> work *in general*. There is not necessarily a 1-1 mapping of
> Much of the time there will be such a mapping, but we'd prefer to not expose
> an API that's going to mysteriously fail some of the time.
>

Seconded!

---

From: Robert Carr <[email-masked]>
Date: Wed, 23 Jul 2014 08:00:45 -0700

Chiming in....but will admit to not having done a super detailed analysis..

I do have an objection to just exposing the global coordinate space.
Without getting in to specifics (so this doesn't become a line item debate
on those specifics), I think we can all agree that a global coordinate
space potentially presents issues and difficulties. The simplest way to
deal with these issues and difficulties is to chunk away the whole concept
:p. I haven't said anything not tautological there, but I felt it needed
stating as I feel in part of the discussion this is ignored. It's not just
a matter of "making an API work".

I don't think this has much to do with accessibility beyond a superficial
level.

Some quick comments on practical solutions to the problem:

I'm a little skeptical of the idea that testing the full input stack in
autopilot application tests is necessary or is a significant contributor to
quality, I think it certainly confuses test scope which has it's own
consequences. My preferred solutions in order:

1. Do it all inside of Qt. Why not?
2. inject_event(client, event) testing API. This doesn't bother me much.
3. query_window_position API (this is neccessarily racy btw...)

---

From: Thomi Richards <[email-masked]>
Date: Thu, 24 Jul 2014 09:57:57 +1200

> consequences. My preferred solutions in order:

Well, the whole point of high-level functional tests is to have as broad a
scope as possible.

This approach has found bugs in mir input event handling before: Remember
the bug about mir not always picking up input devices that were created
after the mir server started? That was found by autopilot, and wouldn't
have been has we been injecting input events into the applications under
test directly. On the unity7 / desktop side, we've found window-stacking
issues with autopilot this way as well. I think I have enough data to say
that "yes, having as broad a scope for functional / user acceptance tests
as possible helps find problems, and increased quality".

Anyway, it sounds like there are several solutions here. I'll wait for
someone to tell me what the proposed fix is. If it requires a change to
autopilot-qt, I really need about 2 weeks notice before it lands in distro
(If you think the mir release process is bad/slow, you should try releasing
autopilot :P ).

Cheers,

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 24 Jul 2014 10:03:07 +0800

While events are being injected into evdev at the kernel level, you do 
have to know screen coordinates of the window in the least, so touches 
match up. That makes Mir's sharing the window position important.

We could take Mir out of the equation if:
   (a) The event injection moved up the stack; or
   (b) All windows tested reside at (0,0).

I'm not sure either of those are as practical as adding a single 
function to the Mir client API. Although I know we don't want it to be 
used in general.

And it's better to have full-stack testing than to not have it.

On 23/07/14 23:00, Robert Carr wrote:
> Without getting in to specifics (so this doesn't become a line item
> debate on those specifics), I think we can all agree that a global
> coordinate space potentially presents issues and difficulties. The
> simplest way to deal with these issues and difficulties is to chunk away
> the whole concept :p. I haven't said anything not tautological there,
> but I felt it needed stating as I feel in part of the discussion this is
> ignored. It's not just a matter of "making an API work".
>
> I don't think this has much to do with accessibility beyond a
> superficial level.
>
> autopilot application tests is necessary or is a significant contributor
> to quality, I think it certainly confuses test scope which has it's own

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 24 Jul 2014 15:52:09 +1000

On Thu, Jul 24, 2014 at 12:03 PM, Daniel van Vugt 
<[email-masked]> wrote:
> While events are being injected into evdev at the kernel level, you 
> do have to know screen coordinates of the window in the least, so 
> touches match up. That makes Mir's sharing the window position 
> important.
> 
> function to the Mir client API. Although I know we don't want it to 
> be used in general.

What do we do for the cases where it's not possible to return a correct 
answer from that function?

Hm. On second thoughts, maybe we should include this, and our existing 
two debug symbols, in an explicit libmirclient-debug library that we 
disclaim responsibility for.

That would still make autopilot need mir-specific code, but not too 
badly. And we wouldn't need to feel bad about making the implementation 
loop over every screen pixel checking whether input at that point hits 
the requested bit. :)

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 24 Jul 2014 14:01:48 +0800

Just remember to map points like the Qt interface does:
    Point screen_coord = mir_surface_coord_to_screen(surface, client_coord);
So we are covered for arbitrary 3D transformations (don't assume windows 
are on screen as rectangles).

That only leaves two problems which are not really problems:
   (1) Races -- Make sure you don't move a window between getting its 
input coordinates and synthesising an event.
   (2) Mirroring e.g. in desktop previews where the same surface is 
composted multiple times in the frame. Actually that's not an issue 
because the input coordinate mapping stuff only cares about the real 
surface location.

So there don't seem to be any practical problems other than us wanting 
to keep private information mostly hidden.

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 24 Jul 2014 16:31:13 +1000

On Thu, Jul 24, 2014 at 4:01 PM, Daniel van Vugt 
<[email-masked]> wrote:
> So we are covered for arbitrary 3D transformations (don't assume 
> windows are on screen as rectangles).
> 

What is the “real” surface location? If we say that it's the 
location where input events will hit it, then it's entirely possible 
that there will be more than one such location - indeed, you can do 
roughly this with Compiz, but obviously without the input bit.

The existence of such an API means that shells must have a concept of 
the canonical location of a surface. That's quite a limitation on a 
shell.

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 24 Jul 2014 14:36:53 +0800

Yeah that's problem (2) mentioned below. And not really a problem.

Autopilot only wants to know one point where it can click a button. The 
outcome is unaffected if the surface is rendered in multiple different 
locations. Just click _one_ copy of the button for the same result.

On 24/07/14 14:31, Christopher James Halse Rogers wrote:
> What is the “real” surface location? If we say that it's the location
> where input events will hit it, then it's entirely possible that there
> will be more than one such location - indeed, you can do roughly this
> with Compiz, but obviously without the input bit.
>
> the canonical location of a surface. That's quite a limitation on a shell.
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 24 Jul 2014 16:45:33 +1000

On Thu, Jul 24, 2014 at 4:36 PM, Daniel van Vugt 
<[email-masked]> wrote:
> Autopilot only wants to know one point where it can click a button. 
> The outcome is unaffected if the surface is rendered in multiple 
> different locations. Just click _one_ copy of the button for the same 
> result.

Unless Autopilot wants to test that each location dispatches input 
correctly, of course :)

I don't object to this being in a libmirclient-debug library. It should 
not be hooked up to Qt's bit, because that's making a promise that 
shells won't necessarily want to keep.

---

From: Gerry Boland <[email-masked]>
Date: Mon, 11 Aug 2014 18:09:30 +0100

Hey folks,
this discussion fizzled out and a conclusion wasn't clear to me. I've
taken the liberty to summarize where the discussion got so far.

The options considered possible (1. was dismissed):

Thomi prefers solution 3. Argues that as Qt exports an API (mapToGlobal)
to do this, Mir should respect it.

RAOF objects to that, saying Qt has lots of platform-specific APIs and
they're poorly documented. mapToGlobal is one of them. Wayland does not
support this call either.

Anpok pointed out that mapToGlobal is not able accurately to encompass
what the compositor is doing to the client's surface (it could be
rotated or weirdly transformed, duplicated...). RAOF & Racarr agree.

TVoss objects too, prefers something like option 2 where AP injects
events into Qt or Mir, instead of closer to the hardware with evdev.
Thomi points out that the current evdev approach found bugs in Mir's
input handling code, so full-stack important benefits.

Duflu had idea of surface_coord_to_screen_coord function available to
the client where the compositor (which is doing the transforming) can
return the correct screen coords to the client. But raised latency
concern & questioned if client occluded, what should it get back? But it
would offer most correct implementation of mapToGlobal.

RAOF open to there being a non-official API for clients
(libmirclient-debug) that AP could use, but does not want Qt's
mapToGlobal connected to it.

After reading the discussion, I (Gerry) have a slightly different proposal:

4. Mir let privileged clients find their surface's screen position.

Shell mediates if a client has permission to ask for its surface's
top-left corner coordinates on screen - which is probably enough for AP.
Or we can be more correct and implement Duflu's suggestion, after
evaluating if the overhead is acceptable.

AP will need a way to tell shell: this app needs to read its position on
screen, so shell will allow it. Else will reject the request. That could
be a DBus call, policed by AppArmor, and shell (unity8) would not need
to be restarted to act upon it.

Personall I don't like the idea of clients being able to ask for their
on-screen position - really there's only a couple of valid use-cases,
which are very special. All apps don't need this.

Sure the other side of the argument is "what is the harm?" - but IMO it
could offer that tiny bit more information to help a malicious app
misbehave - so I'd rather play it safe.

Thoughts?
-Gerry

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 12 Aug 2014 12:33:23 +1000

> to be restarted to act upon it.

I don't think this has any advantage over “shove it in 
libmirclient-debug” if we additionally say “and return garbage from 
everything in libmirclient-debug if Mir isn't started with --debug”?

I'm warming to this idea :). I think it gives everything AP needs 
without too much work (although it does mean AP needs to specifically 
know about Mir) while not allowing apps to depend on information we may 
not be able to provide.

> Personally I don't like the idea of clients being able to ask for 
> their

My primacy concern is not about malicious apps. I'm concerned about 
apps relying on data that the compositor may not be able to provide. 
It's entirely possible that there *isn't* a correct answer for “where 
is my surface on screen”, and any attempt to use such information 
will result in incorrect behaviour.

If we expose this to Qt it'll appear to work (almost) all the time 
under Unity8, apps will find some reason to depend on this working, and 
then when we - or someone else - wants to do something funky we'll find 
that we can't because it breaks clients.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 12 Aug 2014 10:33:26 +0800

There is a 5th option which would satisfy more (not all) of the people 
mentioned:

5. Shell mediates if a client has permission to ask for surface 
coordinates. The API is restricted by default. The API _does_not_ return 
the top-left corner of the surface. Instead it maps a local surface 
coordinate (x,y) to screen coordinates (x',y'). This way we can still 
map correct coordinates for arbitrary 3D transformations (if we want to, 
but don't need to initially).

Also a 6th option:

6. Clients can never ask for their current position but can dictate a 
required creation position. So then they already know how to map between 
their local and screen coordinates (until moved). Such an enhancement 
would also help Mir to support client-side docks and toolbars (which 
someone will ask for eventually).

And a 7th option:

7. Unity8 gets an enhancement to be able to specify position of a 
surface in a way that AP can use. So long as the initial placement is 
honoured, then AP knows how to map between local and screen coordinates 
itself. Until the surface is moved.

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 12 Aug 2014 12:56:34 +1000

On Tue, Aug 12, 2014 at 12:33 PM, Daniel van Vugt 
<[email-masked]> wrote:
> There is a 5th option which would satisfy more (not all) of the 
> people mentioned:
> 
> coordinates. The API is restricted by default. The API _does_not_ 
> return the top-left corner of the surface. Instead it maps a local 
> surface coordinate (x,y) to screen coordinates (x',y'). This way we 
> can still map correct coordinates for arbitrary 3D transformations 
> (if we want to, but don't need to initially).

Again, this assumes that there exists a function whose domain is 
surface coordinates and range is screen coordinates. This function does 
not necessarily exist.

There *is* a unique mapping from screen coordinates to {surfaces} × 
(ℚ × ℚ), but (a) I don't think that's what AP is after - although 
it could implement it - and (b) it would require us to make all 
surfaces visible in some way to all clients.

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 12 Aug 2014 11:13:00 +0800

I think it's mathematically pedantic to say the function does not exist. 
Acknowledging that such a function would theoretically return a set; 
one, multiple or none coordinates (offscreen or occluded), we can dumb 
it down to the simple case that's important -- give one or zero results:

// Returns one screen location where the specified surface coordinate
// is visible. Returns false if the coordinate is not visible on
// screen at all.

MirBool mir_surface_coord_to_screen(MirSurface* surf,
                                     int localx, int localy,
                                     int* screenx, int* screeny);

Admittedly that only works for your own surfaces created in your own 
process.

On 12/08/14 10:56, Christopher James Halse Rogers wrote:
>> can still map correct coordinates for arbitrary 3D transformations (if
>> we want to, but don't need to initially).
>
> Again, this assumes that there exists a function whose domain is surface
> coordinates and range is screen coordinates. This function does not
> necessarily exist.
>
> There *is* a unique mapping from screen coordinates to {surfaces} × (ℚ ×
> ℚ), but (a) I don't think that's what AP is after - although it could
> implement it - and (b) it would require us to make all surfaces visible
> in some way to all clients.
>
>

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 12 Aug 2014 13:17:30 +1000

> I think it's mathematically pedantic to say the function does not 

The *best kind* of pedantry!

> one, multiple or none coordinates (offscreen or occluded), we can 
> dumb it down to the simple case that's important -- give one or zero 
> results:
> 

And I'm perfectly happy for this API to exist in libmirclient-debug, 
and return FALSE when not run with --debug.

What I'm not happy with is hooking this up with any higher-level API 
like Qt's, which does not have the “oh, but it might not work” 
caveat. I'm also much happier if this API doesn't work by default, but 
only in exceptional cases, so client code won't accidentally depend on 
it.

---

From: Gerry Boland <[email-masked]>
Date: Tue, 12 Aug 2014 09:25:27 +0100

On 12/08/14 03:33, Christopher James Halse Rogers wrote:
> > Personally I don't like the idea of clients being able to ask for their
> My primacy concern is not about malicious apps. I'm concerned about apps
> relying on data that the compositor may not be able to provide. It's
> entirely possible that there *isn't* a correct answer for “where is my
> surface on screen”, and any attempt to use such information will result
> in incorrect behaviour.
>
> If we expose this to Qt it'll appear to work (almost) all the time under
> Unity8, apps will find some reason to depend on this working, and then
> when we - or someone else - wants to do something funky we'll find that
> we can't because it breaks clients.
>
This is what is happening right now with X11 though and there's not been any complaints. Since it's really only AP which needs this, it's AP's fault if they do something funky and it breaks.
-G

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 12 Aug 2014 19:27:06 +1000

Right, but you might notice that interacting with transformed windows has been an open feature request on the X server for a decade, and this sort of API is one of the reasons for that :).

Again, I'm ok with it as long as applications don't end up depending on it.

---

From: Daniel d'Andrada <[email-masked]>
Date: Wed, 13 Aug 2014 09:59:31 -0300

> -Gerry

I've yet another suggestion:

unity8 provides a D-Bus API for autopilot to ask "hey, what's the 
position/transformation of surface X?". Then AP would apply that 
transformation to a surface local coordinate and send the resulting 
global coordinate through evdev.

advantages:
  - we don't hassle mir
  - AP still send its input events through evdev, which exercises the 
whole input stack

disadvantages:
  - more work to autopilot and unity8

---

From: Gerry Boland <[email-masked]>
Date: Wed, 13 Aug 2014 14:12:48 +0100

>  - more work to autopilot and unity8

This was option 1 in the original email, and disliked greatly by Thomi.
It requires AP punching open appArmor rules between clients & unity8
when running tests. DBus has much greater latency. Also would require
notion of surface ID, which Mir folks also dislike.

I left it out of further discussion as a result.
-G

---

From: Daniel d'Andrada <[email-masked]>
Date: Wed, 13 Aug 2014 10:19:23 -0300

> I left it out of further discussion as a result.

Ah, right. :)

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 14 Aug 2014 09:31:21 +1000

Ok. I think we've explored all the relevant options

If I implemented

mir_debug_surface_coord_to_screen(MirSurface, int x, int y, int* outx, 
int* outy)

in libmirclient-debug and hooked it up to a --debug flag, who would 
object, and what would those objections be?

---

From: Thomi Richards <[email-masked]>
Date: Thu, 14 Aug 2014 11:41:00 +1200

> notion of surface ID, which Mir folks also dislike.

It'd also require one extra dbus call at a _minimum_ for every call to
Mouse.click_object & Touch.tap_object (those names are from memory, might
not be 100% accurate). While Dbus is fast, it's not fast enough to allow us
to do this without noticing the impact.

If we can avoid it, I'd rather we not rely on dbus for this.

Cheers,

---

From: Thomi Richards <[email-masked]>
Date: Thu, 14 Aug 2014 11:51:03 +1200

> object, and what would those objections be?

Not an objection, but I'd want someone to make a MP against autopilot-qt if
this proposal means not making the existing Qt API 'just work'.

Also, if this involves making unity8 start with the '--debug' command line
argument, then someone (or several people) would have to coordinate
changing all the existing autopilot test suites to make sure that unity8
was started with that command line argument.

Cheers,

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 14 Aug 2014 09:59:42 +0800

The function Chris mentions would indeed allow us to implement the Qt 
function and make it "just work" [QWidget::mapToGlobal()].

Go nuts, Chris :)

On 14/08/14 07:51, Thomi Richards wrote:
>
> On Thu, Aug 14, 2014 at 11:31 AM, Christopher James Halse Rogers
> <[email-masked] <mailto:[email-masked]>> wrote:
>
> Not an objection, but I'd want someone to make a MP against autopilot-qt
> if this proposal means not making the existing Qt API 'just work'.
>
> Also, if this involves making unity8 start with the '--debug' command
> line argument, then someone (or several people) would have to coordinate

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 14 Aug 2014 14:07:12 +1000

Urgh. I know you *can* implement it, I'm just strenuously opposed to it 
being implemented :).

We absolutely should not implement QWidget::mapToGlobal :)

---

From: Alan Griffiths <[email-masked]>
Date: Thu, 14 Aug 2014 09:25:14 +0100

> object, and what would those objections be? 

Returning a bool (or an enum?) to indicate whether the transformation
was successful?

We shouldn't forget that this transform may not be supportable. As
discussed at length there could be 0 or many screen point for a surface
point.

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 14 Aug 2014 16:28:34 +0800

I think a MirBool is enough. Although if anyone does implement the Qt 
function later then that one doesn't have such luxury. You would have to 
return some invalid Point for Qt.

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Fri, 15 Aug 2014 08:55:30 +1000

On Thu, Aug 14, 2014 at 6:28 PM, Daniel van Vugt 
<[email-masked]> wrote:
> function later then that one doesn't have such luxury. You would have 
> to return some invalid Point for Qt.

Which is one reason why we absolutely should not implement the Qt 
function :)

---

From: Gerry Boland <[email-masked]>
Date: Fri, 15 Aug 2014 00:32:20 +0100

> object, and what would those objections be?

And libmirclient-debug is shipped in a separate package which isn't
installed on the device by default? Just thinking how this feature will
not be accessible to all apps.
-G

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Fri, 15 Aug 2014 09:56:32 +1000

> not be accessible to all apps.

Correct. Also, Unity8 won't respond to these requests unless started 
with --debug. Which we obviously won't do by default.

So, it's available to all apps when debugging is enabled, or no apps 
when debugging is not enabled.

---

From: Gerry Boland <[email-masked]>
Date: Fri, 15 Aug 2014 08:14:16 +0100

> when debugging is not enabled.

Thomi, what do you think? Unity8 will need to be restarted with a flag in order to enable this for AP. That ok?
-G

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Fri, 15 Aug 2014 17:17:11 +1000

On Fri, Aug 15, 2014 at 5:14 PM, Gerry Boland 
<[email-masked]> wrote:
> Thomi, what do you think? Unity8 will need to be restarted with a 
> flag in order to enable this for AP. That ok?
> -G

I had a talk with Thomi on IRC this morning; rather than restart Unity8 
with this flag we can just have Unity8 started with this flag on the CI 
devices (and then modify the tests that restart Unity8).

And by “we” I mean “I” ☺.

---

From: Gerry Boland <[email-masked]>
Date: Fri, 15 Aug 2014 08:21:06 +0100

> And by “we” I mean “I” ☺.
Perhaps AP could tell unity8 - go into debug mode - via dbus (hole poked in AppArmor required though)? Saves a restart of the process.

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Fri, 15 Aug 2014 17:27:56 +1000

On Fri, Aug 15, 2014 at 5:21 PM, Gerry Boland 
<[email-masked]> wrote:
>>  >>  >>  mir_debug_surface_coord_to_screen(MirSurface, int x, int y, 
>> int*
>>  >> outx,
>>  >>  >>  int* outy)
>>  >>  >>
>>  >>  > installed on the device by default? Just thinking how this 
>> feature
>>  >> will
> Perhaps AP could tell unity8 - go into debug mode - via dbus (hole 
> poked in AppArmor required though)? Saves a restart of the process.

Possibly? I'm aiming for minimum possible work that cleanly supports 
what people want to do without making API promises we don't want to 
keep.

Is restarting Unity8 a big deal? It doesn't seem so to me; perhaps 
there's some background I'm missing?

---

From: Gerry Boland <[email-masked]>
Date: Fri, 15 Aug 2014 08:32:12 +0100

On 15/08/14 08:27, Christopher James Halse Rogers wrote:
> what people want to do without making API promises we don't want to keep.
>
Only reason I suggested the idea is that restarting unity8 would be a new thing for all app tests, and maybe sending a dbus signal would be less disruptive? But if Thomi things restarting is ok, then I'm happy.

---

From: Michał Sawicz <[email-masked]>
Date: Fri, 15 Aug 2014 12:23:09 +0200

> Only reason I suggested the idea is that restarting unity8 would be a new thing for all app tests, and maybe sending a dbus signal would be less disruptive? But if Thomi things restarting is ok, th…

We're already restarting unity with testability enabled before every 
test suite, to be able to reliably unlock the greeter (in automation 
that is, at home you need to unlock yourself :P).

-- 
Michał (Saviq) Sawicz <[email-masked]>
Canonical Services Ltd.

---

From: Thomi Richards <[email-masked]>
Date: Mon, 18 Aug 2014 11:35:13 +1200

On Fri, Aug 15, 2014 at 10:23 PM, Michał Sawicz <[email-masked]

> Only reason I suggested the idea is that restarting unity8 would be a new
>> thing for all app tests, and maybe sending a dbus signal would be less
>> disruptive? But if Thomi things restarting is ok, then I'm happy.
>>
>
> We're already restarting unity with testability enabled before every test
> suite, to be able to reliably unlock the greeter (in automation that is, at
> home you need to unlock yourself :P).

Right, but that's handled outside the tests (i.e.- by CI infrastructure).
This would add a new requirement that unity8 *must* be restarted with
special options in *all* cases, any time you want to run an autopilot test.
I think that it's quite reasonable to get users to unlock the phone before
an AP test starts if the phone is on their desk, obviously CI needs a
different solution. I don't think it's reasonable to ask users to
_manually_ restart unity8 any time they want to run an AP test, so whatever
we do it should probably be automated.

I'm not sure what the best way to handle that is - you could propose a MP
to autopilot that does the restarting at the correct time, or you could
propose MPs to every test suite that we run on the phone.... there might be
some more options as well (am open to suggestions).

Personally, I think this is an absolutely crazy requirement, but that's
fine, since you'll be implementing the fix, and I'll send any fallout from
the decision to the mir team.

Cheers,