[MERGE][ping] urllib keeps connections alive

[Vincent Ladeuil]

Hi all,

More work since last submission, thanks to reviewers (past and
future), here is another round that will hopefully find its way
into 0.13.

This is a new implementation of bzrlib/transport/http/_urllib.py
but most of the work is done in the new file
bzrlib/transport/http/_urllib2_wrappers.py. 

Also note, that most of the test code have been moved from
transports/http/_urllib.py to tests/HttpServer.py and
tests/HTTPUtils.py.

On top of keeping connections alive, it also attempt one retry of
the request when errors occur, so that should provide a more
robust behavior in case of transient errors at the socket
level. It also implements some adaptive behavior when web
servers misbehave (in bzr POV) regarding ranged GET requests.

It provides fixes for bugs #53654, #57644, #62276, #62029,
#42383, #44647, #48527 <mickey mouse voice>Seven in a blow !
</mickey mouse voice>

I include a patch because the bundle is too heavy for the list
(~1.5MO), but a branch is available on launchpad:

  https://launchpad.net/people/bzr/+branch/bzr/bzr.urllib.keepalive

and the direct link to the branch:

  http://bazaar.launchpad.net/~bzr/bzr/bzr.urllib.keepalive

Look for revision 2041.

To ease review and per John's request, I tried to provide
incremental patches, but after discussing it on IRC, it appears
that it's more complicated than it seems at first sight.

The main problem being that informations from intermediate merges
from bzr.dev pollute the diffs and render them useless for
review.

So instead of further delaying the potential reviews, I send only
the complete diff and will try other approaches to provide better
incremental diffs (any help on the task, even just ideas, will be
very welcome).

Anyway, here are some revnos pointers to help reviewers :

- 2005: First working* implementation based on _urllib2_wrappers,

- 2013: Add basic authentification manager,

- 2020: Add robustness tests around http status line,

- 2023: Add robust code around http status line,

- 2026: Add tests code for range handling in GET requests

- 2029: Move test code outside of
  transport/http/__init__.py. This one is big in size but small
  in intent. It brings up a circular dependency between 'basic
  test framework' using 'hhtp server' as a read-only server and
  'http test framework' defining http server and needing 'basic
  test framework', which was solved by creating
  tests/HttpServer.py. So part of the code went to HttpServer and
  the rest to HTTPUtils.

- 2033: Add tests for ranged GET requests.

- 2034: Provide a more robust http range handling.

- 2041: Cosmetic and minor bug fixing.

Any comments appreciated,

        Vincent

* As in: passing selftest and basic real life tests.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: first_working_urllib2_implementation.patch
Type: text/x-patch
Size: 35702 bytes
Desc: First working _urllib2_wrappers implementation
URL: <https://lists.ubuntu.com/archives/bazaar/attachments/20061025/587d375e/attachment.bin>