Did we break "public-bucket-url" ?

[Ian Booth]

On Sun 15 Sep 2013 20:50:41 EST, John Arbash Meinel wrote:
> I'm trying to sort through the ssl stuff, and I noticed the recent
> changes stopped calling "openstack.NewPublicClient".
>
> This means that PublicStorage is returning an authenticating client
> (s.client).
>

The code is this area has not changed since May. NewPublicClient is still called
when creating a public storage instance based on the configured
public-bucket-url for openstack providers. Note though that it is a requirement
that public-bucket-url be defined for this to happen; the fallback is to simply
use the current user's credentials to create a named container on the user's
account.

> Which I *think* means that if you had been setting public-bucket to
> point to an account which isn't yours, we'll be making requests for
> that data with your creds (rather than no creds).
>

What has changed though, is that the process to find tools for bootstraping juju
first tries using simplestreams. simplestreams does not use the
public-bucket-url setting which is eventually going away. It looks in order at:

1. tools-url setting
2. control bucket
3. keystone juju-tools endpoint (for openstack)
4. public tools repository at https://juju.canonical.com/tools

If none of the above work it falls back to the old way which uses the public bucket.

The simplestreams work is still in progress, so documentation has not been
communicated, but as of Friday last week, for Canonistack you can use this
setting in your env yaml:

tools-url : <public-bucket-url>/tools

where <public-bucket-url> is the originally configured public-bucket-url
setting. The above tools url has metadata and tools for released versions of
Juju up to 1.13.3

simplestreams tools lookup almost always just performs http gets against
configured urls (no credentials required), except for when it tries looking in
the control/private bucket in which case is uses an authenticated storage reader.

> Even with the changes to DataStore, you still go via the StorageReader.
>

This is *only* for the control/private bucket which requires credentials.

> I just tested it with Canonistack and public-bucket-url set (and
> FORCE-VERSION=1.13.3).
>
> With those bits, I see it try to connect to my private bucket, and
> then fall back to what seems to be public-bucket-url. Both of which
> fail. Then it tries the same public URL again and it succeeds.
>
> So I'm pretty sure that public-bucket-url is just broken now, and
> bootstrap is only succeeding because it falls back to the keystone
> entry which it then accesses as a Raw URL rather than a Swift bucket.
>
> Also, with all of the URL probing it now does, bootstrap is *much*
> slower. It took about 1m40s for bootstrap to complete. See
> http://pad.lv/1219582 where I added some debug logs.

public-bucket-url is still used for legacy tools lookup which is invoked if
simplestreams fails. So public-bucket-url is not broken, but it is ignored for
simplestreams.

simplestreams tools lookup does do a bit of file probing via http get, because
it needs to first check for signed metadata and then fallback to unsigned
metadata, and it does this for each possible URL. But the URL probing is pretty
fast (< 1 second for each). It's the control bucket lookup that is slow (see below).

The 10s "timeout" is only relevant when trying to load non-existent files from
the control/private bucket via swift, which is one of the 3 (or 4 on openstack)
places both simplestrams and legacy tools lookup uses. I didn't realise this
delay was there. But it is now made worse because each simplestreams tools
lookup looks for both signed and unsigned metadata. And it happens 3 times
during the bootstrap sequence:

1. look for tools to see if sync is required
1.1 (perform sync)
2. look for tools again after sync to ensure they are there
2.1 (start bootstrap)
3. look for tools during bootstrap

We can do some things to optimise this eg let the bootstrap process know where
the tools are as a result of step 2, or reduce the swift timeout for tools
lookup. Or only look for unsigned metadata in the control bucket. There's a few
possibilities. This issue is only in trunk so we'll need to get it sorted before
the next release.