[apparmor] [patch] make coverage should fail if one of the tests fails

[Christian Boltz]

Hello,

Am Dienstag, 2. Dezember 2014 schrieb Steve Beattie:
> On Sat, Nov 29, 2014 at 09:26:03PM +0100, Christian Boltz wrote:
> > the subject says it all - make coverage should fail if one of the
> > tests fails. Currently it ignores failures and creates the coverage
> > data, so the failed test has a good chance to go unnoticed.
> > 
> > The patch adds a "set -e" to let "make coverage" fail if one of the
> > tests fails.
> 
> In general, I support having tests fail the make target that triggered
> them; however, in this instance I think it's okay that the coverage
> results get generated even if testcases fail, as it can be useful for
> someone in the middle of working on a patch set, where not all the
> tests are passing due to something not having been implemented
> completely, but still wants to see the progress they're making on
> test coverage for another part of their code (I was doing exactly
> this while re-working the capability rules patches). And patches
> should be verified to pass 'make check' anyway, where failing tests
> would get caught.
> 
> So a -1 from me, but I'm open to hearing others' opinions on it.

You have a valid usecase ;-)

OTOH, I'd really like to have it failing because I often (ab)use 
"make coverage-html" to run the tests and get coverage data nearly 
"for free"

How would you like a variable that controls the failure behaviour? 
So you could use something like
    make COVERAGE_IGNORE_FAILURES=true coverage-html

The default could be 
    COVERAGE_IGNORE_FAILURES="set -e"
if we decide to let it fail by default.

(Would "fail by default" be ok?)

It would probably also be possible to do some magic to collect failing 
test-*.py (filenames only) and print a summary after generating coverage 
data, but I'm not sure if that's worth the effort.


Regards,

Christian Boltz
-- 
Böse Zungen behaupten, ein unterschriebenes Zertifikat bescheinigt 
dem Client, daß ein unbekannter Serverbetreiber einem unbekannten
CA-Betreiber Geld bezahlt hat. Das ist natürlich für eine Kommunikation
eine eher nutzlose Garantie.
[http://blog.koehntopp.de/archives/3166-Not-Fixing-SSL.html]