Devel is broken, we cannot release

[Tim Penhey]

On 16/07/14 01:57, William Reade wrote:
> On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel
> <wayne.witzel at canonical.com <mailto:wayne.witzel at canonical.com>> wrote:
> 
>     If we aren't stopping the line when our CI is in the red, then what
>     is the point of even having CI at all? If we are not prepared to
>     adjust the culture of our development. To truly halt forward
>     progress in favor of chasing down regressions then I struggle to
>     find the benefit that CI and testing is giving us at all. Just
>     confirming that master is broken and we are still ignoring it? If
>     master is broken, we all our broken. No development you are doing
>     should proceed that is based on a broken master. That work cannot at
>     any point be considered in good working condition. A problem in
>     master is everyone's problem.
> 
>     Bugs that are found throughout the normal operations and usage of
>     juju should be assigned a priority and queued, but regression is a
>     sign of a greater problem that should be resolved immediately.
>     Allowing regressions to not stop the line is taking the stance that
>     we don't care about deterioration in our code base.
> 
> 
> +100 to this. Regressions are a Big Deal and should be treated as such;
> leaving other merges queued until such a time as the regression is fixed
> (or backed out for rework) is entirely reasonable (and I think we've got
> enough evidence that the alternative really doesn't fly effectively).

Stop the line doesn't mean that everyone stops work, only that trunk
stops accepting new merges until the critical issue is fixed.

This could mean that one or more people actually work on the critical
issue, and others continue with other work, but there are no other trunk
commits until the bug is fixed.  This means that everyone on the team is
aware of the blocker and the progress to fix it because it directly
effects their ability to land their work. This means that there is
internal team pressure as well, and normally this ends up being more
offers to help, review and get the code landed ASAP so it unblocks people.

In the past, we had the landing bot tweaked so it was put into
"emergency only mode" which meant that normal merges were ignored and
only approved critical landings were accepted until the mode was
changed.  This seems like it could be relatively easy to do.

Perhaps a flag like $$ci-fix-merge$$ ?? /me handwaves

Tim