[apparmor] Fun with mod_apparmor + keepalive + iOS

[Seth Arnold]

On Thu, Apr 23, 2015 at 12:10:21AM +0200, Walter Hop wrote:
> I made a small script to replay those requests with exact timings to a
> local VM. This reproduces the problem nearly 100% for me. The script is
> here: http://lf.ms/replay.phps

Nice; this is a good concrete step towards figuring out if mod_apparmor is
long-term viable. (Note that some contraption prevents me from actually
viewing the script, but that's not a big deal, I don't have time to
inspect it, it was just curiosity..)

> Various observations:
> 1) "EnableSendfile Off" reduces the problem 75%. My test case stops
> reproducing, but production still has some failures for iOS users.
> 2) "EnableMMAP Off” affects timing, sometimes making the error switch
> between other requested files.
> 3) "KeepAliveTimeout" is unrelated, the problem happens instantly into
> the connection, so the problem is not related to normal connection
> teardown.

Very nice observations, especially the bit about production systems.
That's hard to replicate with tests.

> What is a good next step?

I think catching Apache in the act would be the next best steps; I can
imagine a few different ways to get there. First would be trying to boil
down your application on the server to as minimal as possible. I realize
that's a huge ask, but if it can be reproduced with one small static file
and a ten line script that'd be a huge advantage. Second, try to catch the
series of system calls that Apache is making when the problem happens.
This is easier, though finding the minimal series of systemcalls might be
easier if the application it is serving is simpler.

The Linux syscall tracing tool is strace; you can attach it to running
processes with -p <pid> and store output to a file with -o /path/to/file
-- and you'll probably want -f to follow fork() calls (which will show up
as clone() calls).

You might also want to try reducing the number of listening apache
processes, just to make this tracing more reliable.

Both the tracing and reducing the number of processes might influence the
bug, this feels like a subtle race condition and those are typically very
vulnerable to slight perturbations of timing.

Thanks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20150422/80262609/attachment.pgp>