launchpad proxy support

[Vincent Ladeuil]

>>>>> "Gordon" == Gordon Tyler <gordon at doxxx.net> writes:

    Gordon> Vincent Ladeuil wrote:
    >> So a couple of general remarks: what you really want to reuse is
    >> the HTTPConnection objects in _urllib2_wrappers.
    >> 
    >> SO forget about the http_transport file entirely (we'll need some
    >> copyright assignments anyway so let's keep it simple too).
    >> 
    >> The idea is that you define some HTTP/HTTPS classes inside
    >> lp_registration and override their _connection_class attributes
    >> with the _urllib2_wrappers HTTPConection objects.  Doing that
    >> will avoid having to redefine the request method too.

    Gordon> I don't think just re-using the _urllib2_wrappers HTTPConection
    Gordon> is going to do it. It needs the whole stack of the ProxyHandlers,
    Gordon> RedirectHandlers, etc. that come from using a urllib2 URLOpener
    Gordon> (appropriately modified by the wrapper).

Sure, I know that :-) That's where the proxy env variables are
analyzed, the redirection handled, etc.

What I meant was that the HTTPTransport class doesn't match the
httplib, urllib2 and _urllib2_wrappers classes design and not
respecting that design is the cause of some of the problems
you're encountering.

    Gordon> However, if it makes you more comfortable, I can
    Gordon> rewrite the HTTPTransport class into
    Gordon> lp_registration.py. It bears very little resemblance
    Gordon> to the original file posted by Bill on the Python bug
    Gordon> report. I have rewritten it from scratch, pretty
    Gordon> much.

I can see tat, but keep two separate paths for http and https
please.

    Gordon> I've just recently pushed another change to
    Gordon> lp:~doxxx/bzr/lp_proxy to fix the underlying problem in the way
    Gordon> urllib2 handles proxying connections to ports other than 80
    Gordon> (i.e. https' 443). I had to apply a workaround/hack to
    Gordon> _urllib2_wrappers' Request and _ConnectRequest so that they could
    Gordon> remember the original URL's port and supply that in the CONNECT
    Gordon> request to the proxy.

+        # Use scheme default if no port specified in URL
+        if not self.proxied_port:
+            if self.get_type() == 'http':
+                self.proxied_port = httplib.HTTPConnection.default_port
+            elif self.get_type() == 'https':
+                self.proxied_port = httplib.HTTPSConnection.default_port

That is the controversial part. That code will not be needed if
you define two classes instead of one, each using the right
HTTP[S]Connection attribute.

    Gordon> This combined with using a custom xmlrpclib Transport to use
    Gordon> _urllib2_wrappers to make the HTTP requests appears to have fixed
    Gordon> the problem completely for me.

Congrats ! You made it work ! Let's try to make it better now :)

    >> Then, we will need tests, which has been the main blocking point
    >> for me so far.... I think the simplest here is to go with some
    >> fake server with pre-canned responses tailored to one request
    >> where the proxied server has one response and the not proxied one
    >> another response just to ensure we reach the good one.
    >> 
    >> We'lll have to test for the various proxies combinations too
    >> http[s] -> http[s].

    Gordon> I'll have to look at how _urllib2_wrappers tests and launchpad
    Gordon> tests are written.

You want to look at test_http among others, but be aware that we
*don't* have a proxy test server, we fake one for test purposes
but I don't think you can reuse it.

    Vincent