Subject: screenshotting, screencasting & unity8+mir

From: Daniel d'Andrada <[email-masked]>
Date: Tue, 26 Nov 2013 10:31:49 -0200

Hi,

On November 26th I (Unity8), Kevin Gunn, Chris Gagnon (Autopilot) and 
Alexandros Frantzis (Mir) had a meeting on the requirements and 
implementation of screenshotting and screencasting.

Chris told us that what autopilot really wants is screencasting (not 
screen shots) as it records all that happens during a test case and 
publishes the resulting video in case of failure.

Kevin came with the point that application developers already want 
screenshotting and screencasting and therefore the solution should also 
cater to them (as opposed to having a solution that works for autopilot 
now and for other/third-party applications to be done only later).

Thus considering those requirements and the future mir architecture 
(using qt scenegraph), that's the implementation we agreed upon:

Unity8 would provide a D-Bus service for clients (applications, 
autopilot) to request for a screencast or screenshot. In case of a 
screenshot, the requestor would just provide a filename (or directory) 
where the screenshot(s) should be placed and unity8 would do it. For 
screencasts unity8 would provide a shared memory area to the requestor 
that would be kept up to date with the contents of the screen. Access to 
that shared memory area would be controlled with a semaphore and a 
mutex. So the problem of handling codecs, recording and audio stream to 
go along with it, etc is left to the user (i.e. an actual screencasting 
application or tool).

D-Bus was chosen because access control to the screenshot/screencast 
feature can be easily implemented using D-Bus security policies. 
According to Alexandros, Mir probably won't have to be modified for 
screen[shotting|casting]. As a QQuickWindow will be controlling the 
composition now (qt-scenegraph approach), what would have to be done is 
calling glReadPixels right after the scene graph is rendered, which is 
essentially how "QImage QQuickWindow::grabWindow()" is implemented (if I 
understood its code correctly).

So that's where I'm heading with the implementation (didn't start yet). 
If someone has a better idea, sees a problem with this approach or 
otherwise has something to add, please advise.

Feedback already received so far:

Thomi Richards:
  - Clarified that autopilot also wants screenshots.
  - Asked for a command line tool that does all the work of talking to 
that proposed screencasting API and outputs an ogv.

Michał Sawicz:
  - "I'm not sold on the idea that it's unity8's / unity-mir's 
responsibility to provide that interface, though, as the ability to do 
those feels like a very generic need that should be built into the 
display server itself. Maybe the fact that we might be using Qt's 
scenegraph changes something here, though - but maybe that just means 
there's a new project growing on unity-mir's side?"

---

From: Daniel d'Andrada <[email-masked]>
Date: Tue, 26 Nov 2013 10:41:03 -0200

On 26/11/13 10:31, Daniel d'Andrada wrote:
> On November 26th [...]

s/26/25

---

From: Gerry Boland <[email-masked]>
Date: Tue, 26 Nov 2013 13:13:28 +0000

> there's a new project growing on unity-mir's side?"

Hey,
The system compositor will probably not be using the Qt scenegraph, but
instead Mir's own compositor. I don't know if using
QQuickWindow::grabWindow() will work in that case (though if it just
calls glReadPixels, it probably will).

Also if hardware (non-GL) compositing is being used, reading back pixels
via glReadPixels won't be enough as not everything on screen will be
drawn by GL.

As a result, I'm not certain we can rely on QQuickWindow::grabWindow() /
glReadPixels, but would need something internal to Mir.
Thanks
-Gerry

---

From: Thomas Voß <[email-masked]>
Date: Tue, 26 Nov 2013 14:15:52 +0100

On Tue, Nov 26, 2013 at 1:31 PM, Daniel d'Andrada
<[email-masked]> wrote:
> Chris told us that what autopilot really wants is screencasting (not screen
> shots) as it records all that happens during a test case and publishes the
> resulting video in case of failure.
>
> cater to them (as opposed to having a solution that works for autopilot now
> and for other/third-party applications to be done only later).
>
> Thus considering those requirements and the future mir architecture (using
> qt scenegraph), that's the implementation we agreed upon:
>
> Unity8 would provide a D-Bus service for clients (applications, autopilot)
> to request for a screencast or screenshot. In case of a screenshot, the
> requestor would just provide a filename (or directory) where the
> screenshot(s) should be placed and unity8 would do it. For screencasts
> unity8 would provide a shared memory area to the requestor that would be
> kept up to date with the contents of the screen. Access to that shared
> memory area would be controlled with a semaphore and a mutex. So the problem
> of handling codecs, recording and audio stream to go along with it, etc is
> left to the user (i.e. an actual screencasting application or tool).
>

Hmmm, how does the shared memory approach relate to HW accelerated
codecs? Did we investigate into libstagefright to check for the
codec's requirements?

> D-Bus was chosen because access control to the screenshot/screencast feature
> can be easily implemented using D-Bus security policies. According to
> Alexandros, Mir probably won't have to be modified for

How would HW compositors be handled? Are we sure that we can execute a
glReadPixels at the right moment?

I think we should model screencasting and screenshotting as one
potential "post-process" step, right after the compositor has finished
one pass. That very interface could then be made available to
command-line tools via DBus.

One other scenario that might of interest to the list of use-cases:
Ability to screencast a complete boot-up | greeter | session |
shutdown video. If only Unity8 knows about
screenshotting/screencasting, we cannot solve that use case.

Thanks,

  Thomas

> So that's where I'm heading with the implementation (didn't start yet). If
> someone has a better idea, sees a problem with this approach or otherwise
> has something to add, please advise.
>
>  - Asked for a command line tool that does all the work of talking to that
> proposed screencasting API and outputs an ogv.
>
> Michał Sawicz:
>  - "I'm not sold on the idea that it's unity8's / unity-mir's responsibility
> to provide that interface, though, as the ability to do those feels like a
> very generic need that should be built into the display server itself. Maybe
> the fact that we might be using Qt's scenegraph changes something here,
> though - but maybe that just means there's a new project growing on
> unity-mir's side?"
>
> --
> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel

---

From: Alexandros Frantzis <[email-masked]>
Date: Tue, 26 Nov 2013 17:12:37 +0200

> glReadPixels, but would need something internal to Mir.

In general, graphics cards and drivers don't offer access to the final
output buffer (after HW compositing), at least not through a standard
API (I don't know if Android drivers offer such functionality). Even if
we move the screenshoting functionality into Mir, it's probable that the
best we will be able to do is to recomposite everything using OpenGL and
read back the pixels.

Perhaps what could happen is that when Unity8 wants to take a screenshot
it can tell Mir to composite everything with OpenGL for the current
frame.

The original discussion was about autopilot being able to take
screenshots/casts of unity8 for testing and validation purposes, and we
came up with a simple solution for this use case.

However, it seems that people have more use cases for screen capturing
that require additional complexity. I think we need to discuss a bit
more about what we really need and when, check what is feasible in our
time frames and prioritize work. The upcoming sprint seems ideal for
this.

Thanks,
Alexandros

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 27 Nov 2013 09:38:37 +0800

On an implementation note...

I think adding yet more library dependencies (dbus) to Mir would be a 
mistake. It's already quite bloated in that respect.

The key recording functionality in Mir shouldn't care about the 
communications anyway. So it would probably be a cleaner division of 
responsibility to make the bus communication external to libmir*. Either 
attached to unity-mir or some privileged client etc.

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 27 Nov 2013 09:41:08 +0800

Another interesting thought...

Mir already has the code to share buffers with external processes at 
high speed. Think about that for a while :)

---

From: Thomas Voß <[email-masked]>
Date: Wed, 27 Nov 2013 07:32:21 +0100

> came up with a simple solution for this use case.

That's a fair point. As I understand it, there is the immediate
requirement to support QA with screenshotting capabilities and the
presented approach can be used to provide the required functionality.

> time frames and prioritize work. The upcoming sprint seems ideal for

+1.

Cheers,

  Thomas

> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/mir-devel

---

From: Thomas Voß <[email-masked]>
Date: Wed, 27 Nov 2013 07:33:46 +0100

> mistake. It's already quite bloated in that respect.

+1, Mir should expose the required functionality via its server-side interfaces.
Exposure to other clients than the shell is then the responsibility of
the shell from my pov.

  Thomas

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 27 Nov 2013 16:00:50 +0800

In fact, there's probably no reason why we can't just add a special 
function:
     mir_connection_create_readback_surface()

Such a surface would work in reverse for a single DisplayBuffer. You 
could read from it by existing means (like 
mir_surface_get_current_buffer) to get the framebuffer, but swapping it 
just provides you with "the next composited frame". A readback surface 
never affects screen contents, only returns them.

I think I prototyped something that worked like that when experimenting 
in the early days of GBM bypass support.

The only significant challenge then is enforcing security. Deciding how 
to authenticate a privileged client before it can 
mir_connection_create_readback_surface()...

Regardless of the actual implementation details, I think we would be 
silly to not utilize the existing buffer IPC code we already have.

---

From: Daniel d'Andrada <[email-masked]>
Date: Wed, 27 Nov 2013 09:56:32 -0200

> Exposure to other clients than the shell is then the responsibility of

Yes, the D-Bus service would be in Unity8, not mir.

---

From: Ricardo Mendoza <[email-masked]>
Date: Wed, 27 Nov 2013 10:30:37 -0500

> presented approach can be used to provide the required functionality.

Back to this, providing a screenshotting interface with a default file
sink is a matter of... 30 minutes? I believe extending the
com.canonical.Unity.Screen interface; as long as you decide on a place
to put screenshot files and a default format.

I believe we should provide this right away for QA, the rest of the plan
seems more broad and therefore not really committable to a strict
timeframe. I agree that casting would be great and that we need it, just
not sure if its the right time to slot it into the dev plan.

> > time frames and prioritize work. The upcoming sprint seems ideal for

Regards,

     Ricardo

---

From: Michał Sawicz <[email-masked]>
Date: Wed, 27 Nov 2013 16:41:48 +0100

> not sure if its the right time to slot it into the dev plan.

They've clarified that they indeed were talking about screen*casting* - 
a recordmydesktop equivalent. So just screenshotting isn't getting us 
anywhere. And the hardware compositing issue needs solving regardless, too.

-- 
Michał (Saviq) Sawicz <[email-masked]>
Canonical Services Ltd.

---

From: Thomas Voß <[email-masked]>
Date: Wed, 27 Nov 2013 17:03:45 +0100

---------- Forwarded message ----------
From: Thomas Voß <[email-masked]>
Date: Tue, Nov 26, 2013 at 2:15 PM
Subject: Re: screenshotting, screencasting & unity8+mir
To: Daniel d'Andrada <[email-masked]>
Cc: [email-masked]

Thanks,

  Thomas

---

From: Thomi Richards <[email-masked]>
Date: Thu, 28 Nov 2013 08:17:09 +1300

Hi,

On Thu, Nov 28, 2013 at 4:41 AM, Michał Sawicz
<[email-masked]>wrote:

> They've clarified that they indeed were talking about screen*casting* - a
> recordmydesktop equivalent. So just screenshotting isn't getting us

>From a QA point of view, the number one priority is video recording. Number
two priority is taking screenshots. In that order.

Cheers,

-- 
Thomi Richards
[email-masked]

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 28 Nov 2013 09:40:29 +0800

Keeping in mind recording at full frame rate (or near) would solve both 
problems, I think we should have a single approach.

The key thing to remember is bandwidth. Only the buffer sharing logic 
used in client IPC has the bandwidth to cope with recording/screencasting.

Consider: 1920x1200 x 4bytesperpixel x 60FPS
--> 553 megabytes per second

Or even: 1366x768 x 4bytesperpixel x 30FPS
--> 126 megabytes per second

Hopefully it would be encoded more efficiently for storage, but still, 
you need to use buffer IPC that's designed for speed. Fortunately we 
have that already.

---

From: Kevin Gunn <[email-masked]>
Date: Wed, 4 Dec 2013 11:33:20 +0000

After some discussion

   - first goal is to satisfy our QA requirement to capture full screen
   output
      - note, fullscreen only for now, not going to address app windows
   - screencast/screenshot will go through the mir api
      - e.g. mir_connection_output_surface, where the client can indicate
      whether or not the client wants a gl or sw buffer, and then
      mir_output_advance_buffer to indicate reading the next buffer (note names
      subj to change...just proposed ;)
      - which fails for unprivileged clients
      - we did discuss the potential addition of explicit capability
      acquire/release for privileged apps, but it was decided we shouldn't add
      this to the api until we have more concrete need/use cases
   - mir will have a server side check with unity-mir for unity8 to check
   against app armor to determine if the app has the capability to screencast,
   this check will only happen when the call to the api is made

On Thu, Nov 28, 2013 at 1:40 AM, Daniel van Vugt <
[email-masked]> wrote:

> The key thing to remember is bandwidth. Only the buffer sharing logic used
> in client IPC has the bandwidth to cope with recording/screencasting.
>
> Hopefully it would be encoded more efficiently for storage, but still, you
> need to use buffer IPC that's designed for speed. Fortunately we have that
> already.
>
>
>

---

From: Thomi Richards <[email-masked]>
Date: Fri, 6 Dec 2013 07:48:40 +1300

Hi Kevin,

Thanks for the summary.

Might I be a pain, and suggest that if it's going to happen through the mir
API, we also need a simple app that can take screenshots. Writing python
<-> C bindings probably isn't the best way to go here, so I wonder if we
could have a simple app that we can call like so: "mir-screenshot
/path/to/file.png".

That should be easy to write, I think?

On Thu, Dec 5, 2013 at 12:33 AM, Kevin Gunn <[email-masked]>wrote:

>> you need to use buffer IPC that's designed for speed. Fortunately we have
>> that already.
>>
>>
>>

---

From: Michał Sawicz <[email-masked]>
Date: Fri, 06 Dec 2013 02:12:58 +0000

> That should be easy to write, I think?

Oh yeah, there would definitely be a simple implementation of the client.

BUT! Screenshotting very much != screencasting, which was identified as 
the requirement for AP test failures to get more data.

Whatever the case, we discussed a long-term solution, as opposed to a 
quick'n'dirty one, which seems to cater for all use cases we could think of.
-- 
Michał (Saviq) Sawicz <[email-masked]>
Canonical Services Ltd.