Subject: Mir client library thread safety

From: Thomi Richards <[email-masked]>
Date: Fri, 24 May 2013 16:37:00 +1200

Hi Folks,

I found another thread safety issue in mir - this time in the client
library. There's a detailed bug reported here:

https://bugs.launchpad.net/mir/+bug/1183659

I started fixing the code myself, but My C/C++ is a little rusty. My fix is
here:

https://code.launchpad.net/~thomir/mir/fix-client-thread-race

The fix seems to work (for my mir-stress test suite anyway), but I had
problems fixing the failing tests.

It's now end-of-week for me, so I'd appreciate it if someone from the US or
Europe could take a look at the bug and my branch and confirm that I've
correctly identified the issue, and perhaps shepherd a fix into trunk.

Have a great weekend everyone.

Cheers,
-- 
Thomi Richards
[email-masked]

---

From: Alan Griffiths <[email-masked]>
Date: Fri, 24 May 2013 10:43:00 +0100

On 24/05/13 05:37, Thomi Richards wrote:
> I started fixing the code myself, but My C/C++ is a little rusty. My
> fix is here:
>
> <https://code.launchpad.net/%7Ethomir/mir/fix-client-thread-race>
>
> It's now end-of-week for me, so I'd appreciate it if someone from the
> US or Europe could take a look at the bug and my branch and confirm
> that I've correctly identified the issue, and perhaps shepherd a fix
> into trunk.

AFAICS There's only one problem test (which hangs) - and the reason it
isn't easy to fix is that the original logic is broken too.

Investigating.

---

From: Alan Griffiths <[email-masked]>
Date: Fri, 24 May 2013 12:06:55 +0100

> that I've correctly identified the issue, and perhaps shepherd a fix

https://code.launchpad.net/~alan-griffiths/mir/fix-bug-1183659/+merge/165572

---

From: Thomi Richards <[email-masked]>
Date: Mon, 27 May 2013 08:44:54 +1200

Thanks!

On Fri, May 24, 2013 at 11:06 PM, Alan Griffiths <
[email-masked]> wrote:

> Mir-devel mailing list
> [email-masked]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/mir-devel
>

---

From: Thomi Richards <[email-masked]>
Date: Mon, 27 May 2013 17:59:39 +1200

Hey everyone,

I've run into another issue while stress testing mir. The stress test tool
I'm writing currently lives here:

lp:~thomir/+junk/mir-stress

With the mir_demo_server running I can run the stress test with a single
thread just fine:

./mir-stress -t 1 -n 10

But as soon as I try and run with multiple threads:

./mir-stress -t 2 -n 10

roughly 50% of the calls to mir_connect_sync fail, with one of the
following error messages:

"connect: Too many open files"
"eventfd_select_interrupter: Too many open files"

The stress test never opens more connections than it has threads running
(in the above example, two simultaneous connections), so I'm pretty sure
this error state is in fact a bug. I've tried several things to narrow down
the problem, so far without much luck. I'm not even sure if this is en
error state on the client side, or the client api is simply reporting an
error message from the mir_demo_server.

I have tried:

 * Using "strace -e trace=open,close mir_demo_server" and "strace -e
trace=open,close ./mir-stress -t 2 -n 10". The trace log from the demo
server doesn't show any open() calls while the client is trying to connect,
and the client trace log shows balanced open()/close() calls.

 * Running the server and client under valgrind. Valgrind finds a few leaks
on the server side, which all seem to be innocuous (memory leaks, that is,
not file handle leaks), and no leaks in mir-stress.

* Inspecting the output of lsof for both the mir_demo_server and mir-stress
while the stress test is running. The server output looks pretty sane, but
I notice that the mir-stress application has /dev/dri/card0 open a total of
1018 times - which, plus a few more file handles, approaches 1024, which
may be the default limit for the number of open file handles in a single
process... maybe? I'm not sure why these wouldn't show up in my
valgrind/strace adventures earlier however.

Is someone else here able to replicate my results? Has anyone got any ideas
how I can move forwards from here?

Cheers!