Improving our review criteria

[Ian Clatworthy]

The Bazaar core team has always had a very strong emphasis on quality 
and that's a great thing. My big complaint though is that our review 
criteria favours "developer quality" over "user quality". In particular, 
patches land if:

1. They improve the product.
2. They don't reduce test coverage.
3. They don't reduce internal clarity.

I think the last two things should be taken into account but not be 
blockers to landing as they are now. Why?

 From my perspective, "quality" is how well a product meets user needs. 
Great software teams follow the Agile philosophy of ...

   maximise the rate at which you deliver *value to the customer*

If a change makes life better for thousands of users, we're actually 
*improving* quality of the product vs not landing it, even if that 
change has less tests that it should or it could be implemented using 
"cleaner" layering. We certainly need to closely monitor technical debt, 
  use refactoring, etc. but, right now, I think our review criteria is 
skewed in favour of 10-20 core developers instead of our user base (80k 
at a rough guess) or our potential user base (several million).

It's actually ok to release software with bugs. To give a concrete 
example, Bazaar Explorer gained virtual repositories in 0.9 even though 
they only work locally and not remotely. I knew that before I released 
it but I felt it was ok because 99% of users wanting that feature are 
browsing local disks. Delaying the feature to fix all the issues would 
have reduced quality IMO.

We certainly should focus on overall solution quality but test coverage 
is only one dimension of that. If we're missing features and that's 
stopping projects from considering Bazaar or migrating to Bazaar, we 
aren't meeting user needs regardless of the warm fuzzy feeling we get 
from our fantastic test suite. It's also important to keep in mind that 
there are multiple ways of achieving great quality. These include:

* gaining more users - all bugs are shallow given enough eyeballs
* rapidly turning around "easy" bugs as users submit them.

In summary, I think our emphasis on quality is awesome but I'd like to 
see us put the user more at the center of it. Stuff like better 
packaging, easier migration, cleaner documentation and faster feature 
rollout need higher emphasis in our culture IMO.

Ian C.