Juju on MAAS agent tools upgrade mechanism

Wed Sep 2 03:01:08 UTC 2015

Hi Peter

There's a lot to respond to in your email - I'll summarise the key points up front.

Firstly, let me briefly explain a little about how the tools tarballs are
handled in Juju. I'll cover bootstrap as well as upgrades, just for
completeness, even though you are just upgrading.

Each machine in a Juju environment runs a jujud binary. The binary is packaged
in the so-called tools tarball. The bootstrap process needs to download the
tools from somewhere to the initial Juju Server. For deployments with internet
access, the tools come from https://streams.canonical.com/juju/tools/. This is
the simplest case and doesn't require any agent-metadata-url or sync-tools usage.

I may have missed it in your emails, but I'm assuming your environment does have
internet access?

In any case, during the bootstrap process, the tool tarballs are retrieved and
cached in a blobstore maintained by the Juju Server. This blobstore is a Mongo
database called "blobstore". This is separate to the "juju" database where the
collections representing the environment model and tools metadata etc are stored.

Note - this is where some of the older material you have found online might be a
little out of date. We have dropped the requirement that cloud providers *must*
provide a blob storage mechanism for Juju to use. In it's place, we now store
charms and tools in the environment blobstore. The MAAS provider still has
access to MAAS storage, but doesn't use it any more for tools, charms etc.

If your environment does not have internet access at the time of bootstrap, then
we need a way to provide the initial Juju Server with the tools. There's 2 ways:
1. Host the tools at a location pointed to by agent-metadata-url config setting
2. Have the tools available in a local directory and pass to bootstrap

For the latter case, the sync-tools utility can be used to pull tools tarballs
from streams.canonical.com to a local directory

If you have no internet access, and tools are available in a local directory,
you bootstrap Juju with the metadata-source argument. This uploads the tools to
the newly deployed environment from that directory rather than fetched from an
online source.

eg
$ mkdir test
$ juju sync-tools --local-dir ~/test --version 1.24
$ juju bootstrap --metadata-source ~/test

The sync-tools above is actually unnecessary if the machine used has internet
access. But if it didn't you'd run sync-tools on a machine which did and copy
the entire ~/test directory across to the machine used to bootstrap.

So the above is for bootstrap.

For upgrades, if the machines in your environment have internet access, then

$ juju upgrade-juju --version=1.24.5

should just work.

If your Juju Server machine does not have internet access, then what you need to
do is copy the new tools to which we want to upgrade into the Juju Server's
blobstore. The sync-tools command does this.

$ juju sync-tools --version 1.24

Once the above is run, the upgrade command should be able to find the latest
1.24 tools in the Juju Server blobstore.

There should be no special processing needed for MAAS. That may have been the
case some time ago, before sync-tools and the local caching of the tools in the
Juju Server blobstore, but not now. The only time bootstrap or upgrade requires
extra steps is for deployments without internet access.

I will start the ball rolling to get that outdated documentation you came across
fixed up.

On your question about whether Juju on MAAS or private clouds is abandonware -
definitely NOT. We have a whole lab set up which tests MAAS and Juju working
together on a daily basis. The fact you've had issues upgrading is most
unfortunate and you've clearly found issues with the documentation which hasn't
helped.

There's lots of folks on the #juju-dev IRC channel on Freenode who would be able
to give you assistance in "real time" if you wanted to hop on there and ask for
help. We'd love to get your issues solved.

On 01/09/15 21:44, Peter Grandi wrote:
> 
>> https://bugs.launchpad.net/juju-core/+bug/1447899 The bug is
>> fixed in the recently released 1.25-alpha1.  What you can try on
>> your system is to explicitly specify the version you want to
>> upgrade to:
> 
> Had a look and forgot to mention that I tried that and got "ERROR
> no matching tools available", that's why I tried to build a local
> cache with known metadata content.
> 
> As it is so typical ("cannot read file" style...), the error
> message does not say which component it came from and which
> version it was trying to match against, and in which list it was
> trying to match it.
> 
>> You appear to have set up a valid tools collection with the
>> correct metadata. The tools tarballs themselves which the
>> metadata refers to reside in a blobstore managed by Juju.
> 
> That's interesting, because I can't find that blobstore, and the
> local setup was upgraded at least from 1.23.2 to 1.23.3 without
> any special work (or at least those who did it don't remember it).
> 
> So far my impression was that:
> 
> * For public-cloud-providers the '.tgz' files are fetched from
>   either:
>   * the URL in the setting 'agent-metadata-url' (used to be
>     'tools-metadata-url') if non-empty or unset.
>   * else the 'streams.canonical.com' site.
> 
> * For MAAS amd private cloud providers:
>   * Create a local cache with:
>     * 'juju sync-tools --local-dir juju-tools' (downloads the
>       '.tgz's to the '.../tools/released/' subdirectory
>     * 'juju metadata generate-tools -d juju-tools''
>     * 'juju sync-tools --source juju-tools' copies the downloaded
>       tools to the '.../tools/releases/' subdirectory.
>   * Make it available to unit-daemons with ???.
> 
> As to the latter point it seems that the local 'jujud' instances
> fetch from 'streams.canonical.com':
> 
>   http://askubuntu.com/questions/555281/ubuntu-maas-juju-bootstrap-stuck-on-fetching-tools
> 
>     Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz 'https://streams.canonical.com/juju/tools/releases/juju-1.20.11-trusty-amd64.tgz'
> 
> at least at bootstrap time.
> 
>> Which bug report are you referring to with regard to building
>> a local tools cache?
> 
>   https://bugs.launchpad.net/juju-core/+bug/1442719 (comment #2)
> 
> I wonder whether MAAS provider based systems are impacted by
> this change:
> 
>   https://bugs.launchpad.net/juju-core/+bug/1484833/comments/1
> 
> and other entries listed here:
> 
>   https://bugs.launchpad.net/juju-core/+bugs?field.tag=sync-tools
> 
>> Regardless of any local cache, or use of sync-tools etc, if
>> your environment has internet access then an upgrade request
>> like the one above will go to
>> https://streams.canonical.com/juju/tools and retrieve from the
>> the requested tools (1.24.5).
> 
> But there are a couple of mentions that this does not happen for
> MAAS managed machines:
> 
> * 'juju help upgrade-juju' 1.24.5:
> 
>     «Both of these depend on tools availability, which some
>     situations (no outgoing internet access) and provider types
>     (such as maas) require that you manage yourself; see the
>     documentation for "sync-tools".»
> 
> * https://help.ubuntu.com/lts/clouddocs/en/Installing-MAAS.html
> 
>     «Preparing MAAS for Juju and OpenStack using Simplestreams»
>     «This necessary information is stored in a json metadata
>     format called "simplestreams". For supported public cloud
>     services such as Amazon Web Services, HP Cloud, Azure, etc,
>     no action is required by the end user. However, those
>     setting up a private cloud, or who want to change how things
>     work (eg use a different Ubuntu image), can create their own
>     metadata, after understanding a bit about how it works.
>     «Setting up metadata for a private (eg Openstack) cloud
>     requires metadata to be generated using tools which ship
>     with Juju.»
> 
> * http://maas.ubuntu.com/docs/juju-quick-start.html
> 
>     «$ juju sync-tools
>     $ juju bootstrap»
> 
>> If there is no outgoing internet access, then sync-tools is
>> used to populate the environment tools cache with tools source
>> from a local directory or elsewhere.
> 
> That's a bit vague. I also looked at:
> 
> * https://help.ubuntu.com/lts/clouddocs/en/Installing-MAAS.html
> 
>     «If no matching metadata is found in the user specified URL,
>     environment's cloud storage is searched. No user configuration
>     is required here - all Juju environments are set up with cloud
>     storage which is used to store state information, charms etc.
>     Cloud storage setup is provider dependent; for Amazon and
>     Openstack clouds, the storage is defined by the
>     "control-bucket" value, for Azure, the "storage-account-name"
>     value is relevant.»
> 
> * http://wiki.cloudbase.it/juju-centos
> 
>     «Now you should edit the environments.yaml file in
>     ~/.juju. Make maas the default provider. Then in the section
>     for maas configuration set: agent-metadata-url:
>     http://<MaaS-Machine-IP>/tools»
> 
> My impression overall is that Juju with MAAS or private cloud is
> not a priority ("abandonware"? "scratch-your-itch"?) and therefore
> I need to reverse engineer it and find a workaround based on
> understanding how it is *supposed* to work and how it *actually*
> works, considering that the 'juju-core' package is 1.24.5 and the
> 'jujud' instances in the units are still 1.23.3.
> 
> So my request for help in understanding how it is *supposed* to
> work for MAAS.
> 