Juju on MAAS agent tools upgrade mechanism

[Peter Grandi]

> https://bugs.launchpad.net/juju-core/+bug/1447899 The bug is
> fixed in the recently released 1.25-alpha1.  What you can try on
> your system is to explicitly specify the version you want to
> upgrade to:

Had a look and forgot to mention that I tried that and got "ERROR
no matching tools available", that's why I tried to build a local
cache with known metadata content.

As it is so typical ("cannot read file" style...), the error
message does not say which component it came from and which
version it was trying to match against, and in which list it was
trying to match it.

> You appear to have set up a valid tools collection with the
> correct metadata. The tools tarballs themselves which the
> metadata refers to reside in a blobstore managed by Juju.

That's interesting, because I can't find that blobstore, and the
local setup was upgraded at least from 1.23.2 to 1.23.3 without
any special work (or at least those who did it don't remember it).

So far my impression was that:

* For public-cloud-providers the '.tgz' files are fetched from
  either:
  * the URL in the setting 'agent-metadata-url' (used to be
    'tools-metadata-url') if non-empty or unset.
  * else the 'streams.canonical.com' site.

* For MAAS amd private cloud providers:
  * Create a local cache with:
    * 'juju sync-tools --local-dir juju-tools' (downloads the
      '.tgz's to the '.../tools/released/' subdirectory
    * 'juju metadata generate-tools -d juju-tools''
    * 'juju sync-tools --source juju-tools' copies the downloaded
      tools to the '.../tools/releases/' subdirectory.
  * Make it available to unit-daemons with ???.

As to the latter point it seems that the local 'jujud' instances
fetch from 'streams.canonical.com':

  http://askubuntu.com/questions/555281/ubuntu-maas-juju-bootstrap-stuck-on-fetching-tools

    Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz 'https://streams.canonical.com/juju/tools/releases/juju-1.20.11-trusty-amd64.tgz'

at least at bootstrap time.

> Which bug report are you referring to with regard to building
> a local tools cache?

  https://bugs.launchpad.net/juju-core/+bug/1442719 (comment #2)

I wonder whether MAAS provider based systems are impacted by
this change:

  https://bugs.launchpad.net/juju-core/+bug/1484833/comments/1

and other entries listed here:

  https://bugs.launchpad.net/juju-core/+bugs?field.tag=sync-tools

> Regardless of any local cache, or use of sync-tools etc, if
> your environment has internet access then an upgrade request
> like the one above will go to
> https://streams.canonical.com/juju/tools and retrieve from the
> the requested tools (1.24.5).

But there are a couple of mentions that this does not happen for
MAAS managed machines:

* 'juju help upgrade-juju' 1.24.5:

    «Both of these depend on tools availability, which some
    situations (no outgoing internet access) and provider types
    (such as maas) require that you manage yourself; see the
    documentation for "sync-tools".»

* https://help.ubuntu.com/lts/clouddocs/en/Installing-MAAS.html

    «Preparing MAAS for Juju and OpenStack using Simplestreams»
    «This necessary information is stored in a json metadata
    format called "simplestreams". For supported public cloud
    services such as Amazon Web Services, HP Cloud, Azure, etc,
    no action is required by the end user. However, those
    setting up a private cloud, or who want to change how things
    work (eg use a different Ubuntu image), can create their own
    metadata, after understanding a bit about how it works.
    «Setting up metadata for a private (eg Openstack) cloud
    requires metadata to be generated using tools which ship
    with Juju.»

* http://maas.ubuntu.com/docs/juju-quick-start.html

    «$ juju sync-tools
    $ juju bootstrap»

> If there is no outgoing internet access, then sync-tools is
> used to populate the environment tools cache with tools source
> from a local directory or elsewhere.

That's a bit vague. I also looked at:

* https://help.ubuntu.com/lts/clouddocs/en/Installing-MAAS.html

    «If no matching metadata is found in the user specified URL,
    environment's cloud storage is searched. No user configuration
    is required here - all Juju environments are set up with cloud
    storage which is used to store state information, charms etc.
    Cloud storage setup is provider dependent; for Amazon and
    Openstack clouds, the storage is defined by the
    "control-bucket" value, for Azure, the "storage-account-name"
    value is relevant.»

* http://wiki.cloudbase.it/juju-centos

    «Now you should edit the environments.yaml file in
    ~/.juju. Make maas the default provider. Then in the section
    for maas configuration set: agent-metadata-url:
    http://<MaaS-Machine-IP>/tools»

My impression overall is that Juju with MAAS or private cloud is
not a priority ("abandonware"? "scratch-your-itch"?) and therefore
I need to reverse engineer it and find a workaround based on
understanding how it is *supposed* to work and how it *actually*
works, considering that the 'juju-core' package is 1.24.5 and the
'jujud' instances in the units are still 1.23.3.

So my request for help in understanding how it is *supposed* to
work for MAAS.