Devel is broken, we cannot release

[Nate Finch]

I think that's a fair assessment.

Perhaps the easiest fix is to install a switch QA could throw to change the
required merge message to something like !!ThisFixesCI!!


On Tue, Jul 15, 2014 at 9:57 AM, William Reade <william.reade at canonical.com>
wrote:

> On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel <wayne.witzel at canonical.com>
> wrote:
>
>> If we aren't stopping the line when our CI is in the red, then what is
>> the point of even having CI at all? If we are not prepared to adjust the
>> culture of our development. To truly halt forward progress in favor of
>> chasing down regressions then I struggle to find the benefit that CI and
>> testing is giving us at all. Just confirming that master is broken and we
>> are still ignoring it? If master is broken, we all our broken. No
>> development you are doing should proceed that is based on a broken master.
>> That work cannot at any point be considered in good working condition. A
>> problem in master is everyone's problem.
>>
>> Bugs that are found throughout the normal operations and usage of juju
>> should be assigned a priority and queued, but regression is a sign of a
>> greater problem that should be resolved immediately. Allowing regressions
>> to not stop the line is taking the stance that we don't care about
>> deterioration in our code base.
>>
>
> +100 to this. Regressions are a Big Deal and should be treated as such;
> leaving other merges queued until such a time as the regression is fixed
> (or backed out for rework) is entirely reasonable (and I think we've got
> enough evidence that the alternative really doesn't fly effectively).
>
> Cheers
> William
>
>
>>
>>
>> On Tue, Jul 15, 2014 at 9:37 AM, Nate Finch <nate.finch at canonical.com>
>> wrote:
>>
>>> I don't think we need to stop the world to get these things fixed.  It
>>> is the responsibility of the team leads to make sure someone's actively
>>> working on fixes for regressions.  If they're not getting fixed, it's our
>>> fault.  We should have one of the team leads pick up the regression and
>>> assign someone to work on it, just like any other high priority bug.
>>>
>>>
>>>
>>> On Mon, Jul 14, 2014 at 3:05 PM, Curtis Hovey-Canonical <
>>> curtis at canonical.com> wrote:
>>>
>>>> Devel has been broken for weeks because of regressions. We cannot
>>>> release devel. The stable 1.20.0 that we release is actually older
>>>> than it appears because we had to search CI for an older revision that
>>>> worked.
>>>>
>>>> We have a systemic problem: once a regression is introduced, it blocks
>>>> the release for weeks, and we build on top of the regression. We often
>>>> see many regressions.The regression mutate as people merge more
>>>> branches.
>>>>
>>>> The current two regressions are:
>>>> * win juju client still broken with unknown
>>>>   from  2014-06-27 which has varied as a compilation
>>>>   problem or panic during execution.
>>>>   https://bugs.launchpad.net/juju-core/+bug/1335328
>>>>
>>>> * FAIL: managedstorage_test trusty ppc64
>>>>   from 2014-06-30 which had a secondary bug that broke compilation.
>>>>   https://bugs.launchpad.net/juju-core/+bug/1336089
>>>>
>>>> I think the problem is engineers are focused on there feature. They
>>>> don't see the fallout from their changes. They may hope the fix will
>>>> arrive soon, and that maybe someone else will fix it.
>>>>
>>>> I propose a change in policy. When a there is a regression in CI, no
>>>> new branches can be merged except those that link to the blocking bug.
>>>> This will encourage engineers to fix the regression. One way to fix
>>>> the regression is to identify and revert the commit that broken CI.
>>>>
>>>>
>>>> --
>>>> Curtis Hovey
>>>> Canonical Cloud Development and Operations
>>>> http://launchpad.net/~sinzui
>>>>
>>>> --
>>>> Juju-dev mailing list
>>>> Juju-dev at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>>
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev at lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>>
>>
>> --
>> Wayne Witzel III
>> wayne.witzel at canonical.com
>>
>> --
>> Juju-dev mailing list
>> Juju-dev at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140715/7ae9b0e7/attachment.html>