Launchpad Report for 2006-03-31

[Christian Robottom Reis]

Just when you thought it was safe to go home! Here's a report of changes
for the second, third and last weeks of March. Apologies for the delay
in sending this report out; this month included a sprint in Launchpad
that threw the scheduled reports slightly off.

The changes listed below were running in production as of Monday morning
at revision 3354 with the following revisions cherry picked in: r3358,
r3362, r3373, r3382.

Extraordinarily this week there was a new rollout done on Tuesday to fix
a few showstoppers on the SFTP server side of things; I'll send out
another report this week that accounts for that.

Important user-visible features or fixes are marked with exclamation
points; other highlights marked with a star.

----------------------------------------------------------------------

Highlights
==========
The list of highlights over these three weeks is too long to list! An
amazing amount of activity happened over the past three weeks, in
particular, during and after the Launchpad sprint.

* On the Malone front, bug watches, branch registration, simplified
  editing and mail notifications are the hot topics. 

* Rosetta had a large number of solid improvements to the import and
  export workflow, and a number of long-standing bugs in this area were
  fixed and placed in production.

* The support tracker now allows setting up contacts to be notified when
  new requests are placed, which was a major requirement for serious
  community support.

* The Soyuz production branch was landed in rocketfuel, greatly
  simplifying the work done in maintaining it. A number of improvements
  and UI fixes were applied to it as well.

* The SFTP server was deployed, allowing people to push branches to
  bazaar.launchpad.net.

* So many infrastructure improvements were made it's hard to highlight
  them: the fix for the Retry top-crasher, XML-RPC support and SQLObject
  prejoining are among the most important.

* The performance work done over this month and the last have brought
  Launchpad to a significantly better level of service.

* I'm going to stop listing improvements now because there are really too
  many to be singled out -- read the exclamation points below if you are
  interested!


Malone
======
! Malone now offers the ability to register bzr branches as fixes for a
  bug. This allows a developer to specify what branch is being used to
  fix a certain bug, and provides a way to track status of that branch
  (bradb, ddaa).
! The second part of bug watch improvements was landed: syncing of
  statuses with Bugzilla bug watches now works as expected, and bugtasks
  that are linked to active bug watches are read-only in the bug pages
  (spec BugWatches, BjornT)
! It is now possible to alter the status of a bug in the main bug
  page; this was by far the #1 user request for Malone and the feature
  makes it a lot easier to update one or more bugs as a set (mpt,
  bradb).
! Mail notifications done by a single person are now batched in 5-minute
  chunks; this should reduce the amount of bug mail Malone sends out to
  end-users, and allows us to capture bug history in a more structured
  format. The bugmail format was also modified to improve batched
  display (BjornT)
* Changing milestones on bugs is now restricted to distribution and
  upstream owners (bradb).
* The code that does comment formatting was updated to further attempt
  to avoid destroying preformatted text (bug 3002, mpt).
* The new buglist format and the advanced search form now include the
  original sort widget (bug 35478, bradb).
* The batch size was tweaked in the production configs to render
  listings with a larger number of bugs (stub).
* Bug listings no longer include Fix Released bugs by default (bug
  34046, matsubara).
* Bugs no longer have short descriptions; they have been merged into
  the main description when they were relevant (mpt).
* Filtering bug queries by component is now possible (bug 33342, bradb).
* Navigation in bug listings was moved to the left, consistently with
  the rest of Launchpad (bradb).
- Personal bug listings now correctly allow searching for specific
  statuses (matsubara).
- Distribution buglists no longer crash when including bugs with no
  source package indicated (bug 33862, BjornT).
- Nicknames are now correctly checked for uniqueness (bug 33904,
  matsubara).
- Remote bug watches have whitespace stripped around them when entered
  by the user (part of bug 34105, BjornT).
- Clearing out the source package of a bugtask no longer fails (bug
  34235, BjornT).
- Using teams with no contact addresses as bug contacts no longer
  crashes (bug 34238, BjornT).
- The message when viewing a bug in a context for which it is not
  reported was improved (bug 30958, mpt).
- We now provide direct links to remote bug watches (bug 29851, mpt).
- The BugTaskSet searching code was unified to avoid duplicating the SQL
  code related to bug privacy (kiko).
- The target milestone layout on the advanced search page was fixed (bug
  36411, bradb).
- Bug menu items were fixed on the bug editing page (bug 32269, mpt).

Rosetta
=======
! Newline character handling in Rosetta was revamped; it now correctly
  handles the various different "standard" newline indicators,
  normalizing the format entered by the end-user, and warning when
  encountering content that is inconsistent (bug 1681, carlos).
! The translation import queue was reimplemented, and many translation
  export fixes were made to the export code. This was a major change in
  the Rosetta infrastructure that allows processing a large number of
  imports and exports in faster and less error-prone manner. The new
  queue now implements filtering options and algorithms that attempt to
  minimize the manual work to be done (bug 33020, carlos).
! Handling of Big5 -- and other encoding formats that can include bytes
  which are literal escapes -- was fixed by converting to UTF-8 early in
  the Rosetta parser (jamesh).
! When parsing a translation file that lacks a proper header we now
  raise POSyntaxError instead of crashing miserably. This was the second
  most common crash when importing new translations (bug 1892, carlos).
! When exporting a translation file, if we are unable to encode the file
  using the declared character set, we now export to UTF-8. This works
  around the fact that many translation files have incorrect character
  sets indicated, and it also accommodates for Rosetta translations that
  include extended characters (bug 1572, carlos).
* A bug that caused fuzzy translations to be accepted as official in
  certain circumstances was tracked down and fixed; it related to
  SQLObject-level caching (bug 1660, carlos).
* Language packs are now always exported in UTF-8 format (carlos).
* A crash in the translation page related to verification of plural
  forms was fixed (bug 31381, matsubara).
- A bug that caused subsequent imports issued by the same person to be
  ignored was fixed (bug 36044, carlos).

Support
=======
! Distributions, source packages and upstream products now offer the
  ability to indicate support contacts, who are automatically subscribed
  when new support requests are filed. This allows the support tracker
  to be used much more effectively, providing support communities with a
  notification system to enhance their work (BjornT)

Soyuz
=====
! The Soyuz production branch was finally landed in production. This was
  a heroic review and code update effort that can not unnoticed: a diff
  of over 15,000 lines was analyzed and reviewed; tests were added; code
  was fixed; fixes from tip were backported; eye and hand were not
  spared getting this done, but in the end it was done -- and all
  rejoiced (cprov, Kinnison, jamesh).
! The second, final part of the initial mirror management feature was
  implemented. This includes the script for probing mirrors and
  verifying how up-to-date their content is, along with a host of other
  fixes to the code (spec MirrorManagement, salgado)
* The personal package page was further improved: it now displays in two
  columns, allows in-table sorting, and displays only the latest version
  uploaded of all relevant packages in the same table; this reduces the
  general size of the page while still displaying the information that
  is most useful to the end-user (kiko).
* A number of UI improvements to Soyuz were implemented, including a
  web view into the upload queue, improvements to the build queue pages,
  and more (cprov).
* The change-override tool was fixed (cprov).
- The package search page no longer returns results in multiplicity (bug
  35905, kiko).
- Traversal was fixed to binary package releases that are not in the
  PUBLISHED status (bug 36208, kiko).
- The binary package publishing history listing no longer crashes when
  displaying packages with removals requested (bug 32605, kiko).
- A minor traversal issue was fixed for DistroReleaseBinaryPackage
  (kiko).
- The distribution page no longer offers a link to Change Members if the
  user does not have the permissions to (bug 1001, mpt).

Specifications
==============
! A large number of improvements were implemented in the specification
  tracker. This includes the ability to target specifications to
  distribution releases and upstream product series, based on a
  nomination and approval system. The new system allows one to track
  implementation of the specification separately from its elaboration
  and approval. Sorting, filtering, menus and the layout of the
  specification listings was also improved. (mark).
! The specification tracker now implements an email listener that
  forwards notifications sent by a specification hosting system (such as
  Moin) to the specification's subscribers (BjornT).
! Email notifications are now sent out to related people when changing a
  specification (BjornT).
- Requesting feedback twice for the same person no longer crashes (bug
  29934, matsubara).

The Bazaar (Code)
=================
! The code for the SFTP server was finally integrated and deployed to
  bazaar.launchpad.net. This allows end-users to push bzr branches to
  this location, and Launchpad will pick up and display changes to these
  branches publically (spec SupermirrorFilesystemHierarchy, spiv).
* It is now possible to fetch a branch by URL, and the branch's URL is
  now a unique identifier for it; this allows us to use URLs to identify
  branches throughout Launchpad (ddaa).
- The branch scanner now correctly detects and handles uncommits (ddaa)
- Branch listings were improved (ddaa)
- Page titles for branches were updated to properly handle branches with
  no title (ddaa)

Launchpad
=========
! Launchpad now offers a default 2-column template, with one column of
  portlets and the main content area expanding to the right. This change
  allows us to accomodate tables well and also promotes better
  accessibility for clients with reduced browser widths. A number of
  pages have been or are in the process of being converted to this
  template (mpt).
! Karma values are now normalized per category: this means that the
  maximum karma available from working on Malone and Rosetta is the
  same; this should make for a more level playing ground, and it rewards
  people that work across multiple appliations (stub).
* Karma is no longer calculated for teams (stub).
* Given and family name columns, largely unused, were removed from the
  Person table (stub).
* The code of conduct pages were reworked to make their meaning
  clearer to end-users (part of bug 31933, mpt).
- Error pages now correctly indicated that the word "bug" needs to be
  included when emailing the system-error address (mpt).
- Adding events to the top-level calendar no longer crashes (bug 2112,
  matsubara)
- A broken "Unsubscribe to Bounty" link was removed (mpt).
- Menuitems for support requests and people were simplified (mpt).
- We no longer explicitly suggest registering a new product when
  searching for a product fails. The product search results were also
  reformatted slightly (mpt).
- The team page no longer offers a link to edit Members if the
  user does not have the permissions to (bug 31839, mpt).
- The FOAF RDF export now includes a correctly-calculated mbox_sha1sum
  (bug 35198, stub).

Infrastructure
==============
! Retry exceptions, the single most recurring cause of crashes, are now
  properly handled in the Zope publisher code (bug 31478, stub).
! The upgrade to PostgreSQL 8.1 was completed, giving us further
  improvements in locking and performance. This upgrade required minor
  updates to our test and database schema code (stub, SteveA).
! An SQLObject feature that allows for selective optimization through
  premeditated joining and caching was implemented. This feature allows
  us to dramatically reduce the number of queries issued against the
  database in most cases, reducing and eliminating timeouts in all but
  the most severe cases. This optimization was applied to all
  problematic pages in Launchpad, and a very small number of pages now
  remain as performance issues (spiv, kiko).
! XMLRPC support was added to the Launchpad infrastructure, making it
  very easy to offer public APIs through a browser class. This will
  allow building very useful external APIs with minimal developer
  effort. Navigation components were additionally made available to the
  XMLRPC request object (SteveA).
! The standard error message display Launchpad offers was significantly
  improved; there are now distinct notification styles for error,
  warning, informational, and debug messages (mpt).
! Client-side certificates are no longer negotiated for any of our
  sites; this fixes an issue that caused some users and some browsers to
  be unable to access the Launchpad sites (Znarl).
! PQM was revamped to include progress output and filter submissions
  according to which project it is servicing. Test output was also
  modified to output incrementally to better support this feature
  (lifeless, stub).
* The batch navigator API was refactored, reducing the amount of work
  necessary to implement batching, and avoiding duplication and
  confusion in handling its arguments. Much refactoring and optimization
  was done, and a number of smaller bugfixes were also implemented. All
  callsites were updated to use the new API (bugs 33290, 1235 and
  others, kiko).
* The hang-detection logic in our test runner infrastructure was
  previously inverted, which caused it to never actually kick in. This
  is now fixed (spiv).
* Full-text search quoting was fixed to handle a number of special
  symbols (bugs 32071, 33920, stub).
- We now try harder to avoid leaving signal handlers behind when tearing
  down test cases (spiv).
- The librarian now calculates MD5 sums for files stored in it; this
  implements a requirement for implementing PersonPackageArchives
  (spiv).
- SQLObject now copes with single-column tables (bug 35952).
- The ZODB was completely disabled and references to it removed, in an
  attempt to avoid threading deadlocks in production (SteveA).
- dbschema items now include a sortkey field that allows specifying an
  integer value for sorting dbschema items (SteveA).
- Menus now have a request instance variable, like views do. This allows
  creating dynamic menus (SteveA).
- The duration of the entire request, instead of just SQL time, is now
  properly reported in OOPS reports (jamesh).
- A minor fix to count() queries on set operations was implemented
  (kiko).
- SelectResults.__nonzero__ only issues one query now, and avoids
  including any prejoined columns, which are irrelevant to it (kiko).
- The distrorelease vocabulary no longer uses the object database ID to
  refer to the object, fixing its use in autogenerated forms (carlos).
- The GeneralFormView now includes facet support (mpt).
- A number of validation functions were cleaned up (matsubara).
- Queries on alternate IDs now cause SQLObject to issue only one query,
  caching the result, instead of two (spiv).
- The sqlvalues function now correctly handles SQLObject and DBSchema
  items (kiko)
- All SQLObject tests now pass, and are now required when merging
  SQLObject code into RF (salgado, spiv).
- The backported, intelligent version of SQLMultipleJoin was now added
  to all database classes that still used the old version, which is a
  performance enhancement on many levels given that the old version
  issued a large number of queries when joining (bug 33625, matsubara)
- The system gettextpo library is used instead of our copy of it; we
  used a copy because it contained updates that have now been rolled
  into the released versions (jamesh).
- shortlist() now correctly indicates what callsite is using it
  incorrectly (stub).
- Database diagrams were updated (stub).

Upcoming Changes
================
- ShipitForDapper (salgado)
- POMsgSetPage (carlos)
- Fixes for session concurrency crashes (stub)
- launchpad-server and launchpad-developer packages (jblack, jbailey)

Planned Changes
===============
- Zope 3.2 update (stub, SteveA)
- CrowdControl (SteveA)

-- 
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125