Subject: Untangling EventHub/InputReader

From: Christopher James Halse Rogers <[email-masked]>
Date: Mon, 11 Nov 2013 15:59:23 +1100

Hi all,

As a part of making Mir-on-the-desktop useful we'll need to make
touchpad (and, in general, non-touchcreen) support non-terrible.
Currently it looks like we basically take what evdev sends us and run
with it; to see how terrible that is, uninstall
xserver-xorg-input-synaptics and try to use your touchpad :).

So, we need to do some fairly significant post-evdev-pre-Mir filtering.
In order to make our lives easier this should be done by something like
libtouchpad¹. This means I'm in the market for reworking the ball of
cats that is 3rd_party/.../EventHub.cpp.

As I understand it, the EventHub is responsible for:
1) Discovering new/disappearing devices
2) Emitting events on hotplug
3) Initialising devices
4) Device configuration
5) Reading events from devices and spewing the raw evdev at the caller
6) Describing device capabilities
7) Translating between keycodes and scancodes
8) Getting axis values from devices
9) Vibrating the phone
...
Which seems a /little/ bit like a grab bag.

On top of that, InputReader appears to be responsible for processing the
raw evdev events into something that's actually useful.

This needs cleaning.

For a first go, and to allow me to add proper touchpad support in a sane
way, I think splitting out an honest Device class would be a good first
step in cleaning up.

This Device interface would be responsible for
1) Device initialisation
2) Device configuration
3) Reading events from the device, processing them, and emitting useful
output events
4) Describing device capabilities

If necessary there could be extra interfaces for touchscreens,
touchpads, keyboards, etc, but it looks to me like those are only
necessary because InputReader needs to do device post-processing.

Now, I'm not sure how we want to deal with wide-ranging changes to
3rd_party sources. Presumably we want to pull them into the src/ tree
and test it from there, and have 3rd_party have a weird dependency on
src/?

Chris

¹: https://github.com/whot/libtouchpad / also RAOF/libtouchpad

---

From: Daniel van Vugt <[email-masked]>
Date: Mon, 11 Nov 2013 13:42:20 +0800

We do have recent precedent for putting our own input code (GPL) under 
3rd party, when it belongs next to the existing Android input stuff.

I think it's a given that we've modified Android Input beyond being able 
to pull in new versions. So new code of our own won't get lost.

---

From: Daniel d'Andrada <[email-masked]>
Date: Mon, 11 Nov 2013 10:33:59 -0200

Hi Christopher,

The job of EventHub, in short, is multiplexing the event streams from 
all those /dev/input/event* files into a single output stream of events 
(those events then having device ids to identify from which device they 
come from).

To me it looks like a very clear-cut task. Thus I think that whatever 
you wanna do with those events, that should take place *after* EventHub. 
Just like InputReader, taking those "raw events" from EventHub and 
"cooking" them.

I do agree with you on the vibration API, though. Feels out of place and 
separate from the rest of the API and code. I bet it was added to 
EventHub at a much later time.

So interpreting evdev event triplets (type, code, value) to come up with 
higher-level events or gestures like tap-to-click, two-finger scrolling, 
software button emulation, etc should definitely be done post-EventHub. 
InputReader already does very similar things.

 From your intentions (touchpad gestures), I think you should be looking 
at InputReader, not EventHub. The InputReader::PointerGesture* related 
code seems to be used for mouse pointer emulation. It translates 
multitouch events into pointer (i.e. mouse) movements and actions. Such 
as moving an unpressed pointer around (one-finger movement, aka 
hovering) and moving a pressed mouse around (two-fingers down moving 
around together, aka dragging).

Best Regards,
Daniel

---

From: Thomas Voß <[email-masked]>
Date: Mon, 11 Nov 2013 14:12:33 +0100

On Mon, Nov 11, 2013 at 1:33 PM, Daniel d'Andrada
<[email-masked]> wrote:
> Hi Christopher,
>
> The job of EventHub, in short, is multiplexing the event streams from all
> those /dev/input/event* files into a single output stream of events (those
> events then having device ids to identify from which device they come from).
>
> To me it looks like a very clear-cut task. Thus I think that whatever you
> wanna do with those events, that should take place *after* EventHub. Just
> like InputReader, taking those "raw events" from EventHub and "cooking"
> them.
>
> separate from the rest of the API and code. I bet it was added to EventHub
> at a much later time.
>

Force-feedback support ideally should come in post event-hub, too. For
example via means of a subclass for devices that support
force-feedback.

Thomas

> From your intentions (touchpad gestures), I think you should be looking at
> InputReader, not EventHub. The InputReader::PointerGesture* related code
> seems to be used for mouse pointer emulation. It translates multitouch
> events into pointer (i.e. mouse) movements and actions. Such as moving an
> unpressed pointer around (one-finger movement, aka hovering) and moving a
> pressed mouse around (two-fingers down moving around together, aka
> dragging).
>
> Best Regards,
> Daniel
>
>
> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel

---

From: Robert Carr <[email-masked]>
Date: Mon, 11 Nov 2013 12:11:30 -0800

>> Which seems a /little/ bit like a grab bag.

I think you are right, in terms of implementation it's a total grab bag. I
think Daniel makes a good point that conceptually the EventHub should be
quite clear. It's like a poll manager for the input devices and evdev (to
open devices).

Currently the implementation though has everything from the location of
configuration files, to device details, to strange quirks suited to android
use cases (See if there is a "Q" key to determine if we have an
alphanumeric keyboard).

The way it stands now it's too complex for me to confidently work in.

>> raw evdev events into something that's actually useful.

Yes. As Daniel points out there is some good infrastructure here for the
kind of problem we are trying to solve in the "InputMapper" classes. I'm
not convinced this covers the whole touchpad case though.

>> This needs cleaning.

I agree. Let's tackle the EventHub from just a hygenic perspective of
clarifying responsibilities. We need to be able to work confidently with
this code.

>> way, I think splitting out an honest Device class would be a good first

I think so too. Should I take this on?

Thanks,
Robert

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 12 Nov 2013 11:30:38 +1100

> (those events then having device ids to identify from which device they 

Right, and that's what I'd like to *make* it do. What it *currently*
does is multiplexing the event streams. Oh, and (badly) managing hotplug
of input devices. Oh, and managing the translation from hardware
scancodes to keycodes. Oh, and providing a description for input
devices. Oh, and…

My proposal is that EventHub retain its task of multiplexing the event
streams by watching fds provided by InputDevices. I just want to discard
the other ancillary things that it does.

> Just like InputReader, taking those "raw events" from EventHub and 

I don't think it makes sense to have the uncooked events leave the
device-specific code. What currently happens is that InputReader “cooks”
them, but needs to call back into EventHub in order to do that.

Only the device-specific code really knows how to cook the events
properly.

Also, pragmatically, libtouchpad only provides fully cooked events.

> InputReader already does very similar things.

Doing it on a higher level only makes sense if that higher level allows
us to collect behaviours common to multiple lower-levels. Currently the
upper level deals with at least two entirely disjoint lower levels -
keyboards and touchscreens.

I don't think that touchpads are going to share much behaviour with
touchscreens - they are fundamentally different input devices - and what
little behaviour they do share I don't believe will be easily
abstractable at the higher layer.

Fundamentally I believe that event cooking in InputReader is a
misdesign, at least for our expanded use-cases.

Chris

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Tue, 12 Nov 2013 11:33:13 +1100

> I think so too. Should I take this on?

I'm happy to continue through here; I'll probably need to in order to
collaborate with Peter anyway.

You're welcome to take on hooking up Mir input to XMir, though :)

Chris

---

From: Robert Carr <[email-masked]>
Date: Tue, 12 Nov 2013 11:51:31 -0800

>> Also, pragmatically, libtouchpad only provides fully cooked events.

It seems like even if we start with splitting out the device there are two
sort of directions this could go. On one hand when reading the events, they
could be "cooked", and leave the event hub processed in a device specific
fashion. On the other hand, EventHub could view 'Device' through a simple
'EventSource' interface, and this way the EventHub is modeled as the
'device independent code'. The input reader becomes responsible for
combining the events with the device specific logic.

My intuition kind of lends to the second idea. I think the 'InputReader'
will also depend on some other system state, for example when cooking just
normal cursor input it depends on display state, especially if we start to
introduce things like barriers. This seems like a mismatch with the
EventHub idea of multiplexing the event streams.

Maybe it's not pragmatically the best though? This needs to be a flexible
interface. Both for libtouchpad, and perhaps even other input drivers
further outside of our control in the future.

Robert,

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Wed, 13 Nov 2013 10:57:01 +1100

On Tue, 2013-11-12 at 11:51 -0800, Robert Carr wrote:
> 'device independent code'.

I think in both cases EventHub is device independent code. The
difference is whether the device dependent code comes before, or after,
EventHub.

> introduce things like barriers.

I think we might have different stages of cooking here :). I want the
events leaving EventHub to be a usefully accurate representation of the
user's interaction with the input device. There are further stages of
cooking needed after that - possibly you want to scale to screen size,
then you want the shell to work out whether it needs to handle them,
then you need to identify the client to send the events to, then you
need to translate them into client space using some pretty arbitrary
projective transform.

I don't think we want the shell wants to deal with jittery touchpads,
and the EventHub *certainly* doesn't want to be dealing with projective
transforms into client space.

There are things which might mix the layers, though - you could
implement touchpad acceleration in a way that depends on knowing the
screen size, so an equal movement on the touchpad results in an equal
fraction of the screen traversed¹. And there's the possibility of
probabilistic input stuff, where the higher levels use knowledge about
clients and windows and such to help interpret ambiguous events.

I think those things are best handled as we come to them, though.

> This seems like a mismatch with the

Right. We *will* have external input drivers at some point; there
inevitably turns out to be processing that needs to be done outside the
kernel. We're not going to want to add touchpad, wacom tablet, kinect,
wiimote, $RANDOM_INPUT_DEVICE code directly into Mir.

¹: I suspect this might be better implemented by doing the acceleration
first in fixed-point subpixel-accurate events and then scaling further
up in the stack; after all, this requires knowledge of where the pointer
currently is.

---

From: Daniel d'Andrada <[email-masked]>
Date: Wed, 13 Nov 2013 09:27:09 -0200

> need to translate them into client space using some pretty arbitrary

On the "need to identify the client to send the events to, then you need 
to translate them into client space using some pretty arbitrary 
projective transform" part: that would be done by the qml scene in the 
shell (unity8), right? (gerry's qt scene graph plan) As that scene will 
have qml items representing client windows.

Just to make sure we are all on the same page.

- Daniel

---

From: Christopher James Halse Rogers <[email-masked]>
Date: Thu, 14 Nov 2013 11:06:43 +1100

> Just to make sure we are all on the same page.

Right. I'm just providing a non-exhaustive example of the various stages
of cooking that need to be done to events.