Subject: CI failures

From: Daniel van Vugt <[email-masked]>
Date: Wed, 11 Dec 2013 17:04:17 +0800

It seems CI has become a little over-ambitious recently and no branch 
can pass all the runs. Can we disable the broken ones?

On 11/12/13 17:01, PS Jenkins bot wrote:
> Review: Needs Fixing continuous-integration
>
> FAILED: Continuous integration, rev:1282
> http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-ci/514/
> Executed test runs:
>      FAILURE: http://jenkins.qa.ubuntu.com/job/mir-android-trusty-i386-build/401/console
>      SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-trusty-amd64-build/397
>      FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-trusty-touch/6/console
>      SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-amd64-ci/240
>          deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-amd64-ci/240/artifact/work/output/*zip*/output.zip
>      SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-armhf-ci/243
>          deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-armhf-ci/243/artifact/work/output/*zip*/output.zip
>      SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-trusty-armhf/6
>          deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-trusty-armhf/6/artifact/work/output/*zip*/output.zip
>      FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/45/console
>      SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/2218
>
> Click here to trigger a rebuild:
> http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/514/rebuild
>

---

From: Chris Gagnon <[email-masked]>
Date: Wed, 11 Dec 2013 11:01:58 -0500

The demos say there is an error because of output on stderr, but it won't
fail the build because of it. Currently it will block the build when an
integration test or acceptance test fails.

These tests pass when  I ran CI on the current trunk of the mir-development
branch[1]  and the tests passed, the test also passed before enabling CI
more than one time. It looks like the new code is causing these tests to
fail.

We cannot disable tests when they start to fail.

[1] http://s-jenkins.ubuntu-ci:8080/job/mir-mediumtests-trusty-touch/8/

On Wed, Dec 11, 2013 at 4:04 AM, Daniel van Vugt <
[email-masked]> wrote:

> It seems CI has become a little over-ambitious recently and no branch can
> pass all the runs. Can we disable the broken ones?
>
>
>>          deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-
>> branch-trusty-amd64-ci/240/artifact/work/output/*zip*/output.zip
>>          deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-
>> branch-trusty-armhf-ci/243/artifact/work/output/*zip*/output.zip
>>          deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-
>> trusty-armhf/6/artifact/work/output/*zip*/output.zip

---

From: Alan Griffiths <[email-masked]>
Date: Wed, 11 Dec 2013 16:32:30 +0000

On 11/12/13 16:01, Chris Gagnon wrote:
> These tests pass when  I ran CI on the current trunk of the
> mir-development branch[1]  and the tests passed, the test also passed
> before enabling CI more than one time. It looks like the new code is
> causing these tests to fail. 
>

I've been looking at these failures:

1. The code being changed is unrelated to the failing tests.
2. I cannot reproduce the failures locally.
3. I've found some things to clean up in the test code, but nothing that
I can flag as a race condition.
4. I can't find any timeout issues that look relevant either.

There's obviously something different about the build environment, but I
don't yet know what.

I'll be submitting my cleaned up code shortly and will watch the test
results with interest.

Alan

PS here are my "cuttings" of the failures seen across several MPs:

[ RUN ]
[mSurfaceLoop.creating_a_client_surface_allocates_buffer_swapper_on_server
unknown file: Failure
C++ exception with description "Timeout while waiting for child to
change state" thrown in TearDown().
[ FAILED ]
[mSurfaceLoop.creating_a_client_surface_allocates_buffer_swapper_on_server
(1384463556730 ms)

[ RUN ] ServerDisconnect.client_detects_server_shutdown
unknown file: Failure
C++ exception with description "Timeout while waiting for child to
change state" thrown in TearDown().
[ FAILED ] ServerDisconnect.client_detects_server_shutdown (60188 ms)

[ RUN ] AvailableSurfaceFormatsTest.surface_pixel_formats_reach_client
/tmp/buildd/mir-0.1.2+14.04.20131128.1+autopilot0/tests/integration-tests/test_display_info.cpp:173:
Failure
Value of: StubGraphicBufferAllocator::pixel_formats.size()
Actual: 3
Expected: returned_format_size
Which is: 0
Google Test trace:
/tmp/buildd/mir-0.1.2+14.04.20131128.1+autopilot0/tests/mir_test_framework/testing_process_manager.cpp:147:
Client
/tmp/buildd/mir-0.1.2+14.04.20131128.1+autopilot0/tests/mir_test_framework/testing_process_manager.cpp:169:
Failure
Value of: result.succeeded()
Actual: false
Expected: true
client terminate error=process::Result(child_terminated_normally,
failure(1))
[ FAILED ]
AvailableSurfaceFormatsTest.surface_pixel_formats_reach_client (216

[ RUN ]
DefaultDisplayServerTestFixture.client_library_connects_and_disconnects
/tmp/buildd/mir-0.1.3+autopilot0/tests/acceptance-tests/test_client_library.cpp:129:
Failure
Value of: connection != NULL
Actual: false
Expected: true
Google Test trace:
/tmp/buildd/mir-0.1.3+autopilot0/tests/mir_test_framework/testing_process_manager.cpp:147:
Client
/tmp/buildd/mir-0.1.3+autopilot0/tests/mir_test_framework/testing_process_manager.cpp:169:
Failure
Value of: result.succeeded()
Actual: false
Expected: true
client terminate error=process::Result(child_terminated_normally,
failure(1))
[ FAILED ]
DefaultDisplayServerTestFixture.client_library_connects_and_disconnects
(144 ms)

---

From: Daniel van Vugt <[email-masked]>
Date: Thu, 12 Dec 2013 09:40:07 +0800

Of course I was not suggesting disabling tests. Just to disable broken 
test _machines_.

It appears they're all fixed today, so problem solved :)

On 12/12/13 00:01, Chris Gagnon wrote:
> The demos say there is an error because of output on stderr, but it
> won't fail the build because of it. Currently it will block the build
> when an integration test or acceptance test fails.
>
> On Wed, Dec 11, 2013 at 4:04 AM, Daniel van Vugt
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>
>     It seems CI has become a little over-ambitious recently and no
>     branch can pass all the runs. Can we disable the broken ones?
>
>
>         http://jenkins.qa.ubuntu.com/__job/mir-team-mir-development-__branch-ci/514/
>         <http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-ci/514/>
>         Executed test runs:
>               FAILURE:
>         http://jenkins.qa.ubuntu.com/__job/mir-android-trusty-i386-__build/401/console
>         <http://jenkins.qa.ubuntu.com/job/mir-android-trusty-i386-build/401/console>
>               SUCCESS:
>         http://jenkins.qa.ubuntu.com/__job/mir-clang-trusty-amd64-__build/397
>         <http://jenkins.qa.ubuntu.com/job/mir-clang-trusty-amd64-build/397>
>               FAILURE:
>         http://jenkins.qa.ubuntu.com/__job/mir-mediumtests-trusty-__touch/6/console
>         <http://jenkins.qa.ubuntu.com/job/mir-mediumtests-trusty-touch/6/console>
>               SUCCESS:
>         http://jenkins.qa.ubuntu.com/__job/mir-team-mir-development-__branch-trusty-amd64-ci/240
>         <http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-amd64-ci/240>
>                   deb:
>         http://jenkins.qa.ubuntu.com/__job/mir-team-mir-development-__branch-trusty-amd64-ci/240/__artifact/work/output/*zip*/__output.zip
>         <http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-amd64-ci/240/artifact/work/output/*zip*/output.zip>
>               SUCCESS:
>         http://jenkins.qa.ubuntu.com/__job/mir-team-mir-development-__branch-trusty-armhf-ci/243
>         <http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-armhf-ci/243>
>                   deb:
>         http://jenkins.qa.ubuntu.com/__job/mir-team-mir-development-__branch-trusty-armhf-ci/243/__artifact/work/output/*zip*/__output.zip
>         <http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-armhf-ci/243/artifact/work/output/*zip*/output.zip>
>               SUCCESS:
>         http://jenkins.qa.ubuntu.com/__job/mir-mediumtests-builder-__trusty-armhf/6
>         <http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-trusty-armhf/6>
>                   deb:
>         http://jenkins.qa.ubuntu.com/__job/mir-mediumtests-builder-__trusty-armhf/6/artifact/work/__output/*zip*/output.zip
>         <http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-trusty-armhf/6/artifact/work/output/*zip*/output.zip>
>               FAILURE:
>         http://jenkins.qa.ubuntu.com/__job/mir-mediumtests-runner-__mako/45/console
>         <http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/45/console>
>               SUCCESS:
>         http://s-jenkins.ubuntu-ci:__8080/job/touch-flash-device/__2218
>         <http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/2218>
>
>         http://s-jenkins.ubuntu-ci:__8080/job/mir-team-mir-__development-branch-ci/514/__rebuild
>         <http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/514/rebuild>
>
>

---

From: Daniel van Vugt <[email-masked]>
Date: Tue, 17 Dec 2013 15:49:02 +0800

Actually, they're not fixed. Which is why very little has landed this 
past week...

https://bugs.launchpad.net/mir/+bugs?field.tag=blockingci

One is a minor scripting issue due to changes in libhybris. The other is 
an annoyingly intermittent (Galaxy)Nexus(/4) test failure.

There may be other issues still blocking CI. If you find them, please 
use tag "blockingci".

- Daniel

---

From: Daniel van Vugt <[email-masked]>
Date: Wed, 18 Dec 2013 14:58:59 +0800

Chris,

We still have some weird CI failures that are not the fault of Mir:
https://jenkins.qa.ubuntu.com/job/mir-android-trusty-i386-build/472/consoleFull
https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/112/console

and perhaps:
https://jenkins.qa.ubuntu.com/job/mir-mediumtests-trusty-touch/78/console
(why can't I see any useful log there?)

- Daniel