Juju on MAAS agent tools upgrade mechanism

[Peter Grandi]

Apologies for the late reply, I spent most the time in between
reverse engineering some issues with other ("hipsterish")
clusterized services.

In the meantime I have written and just now uploaded my own
draft overview of how Juju is structured, at a very high level:

  https://wiki.ubuntu.com/ServerTeam/JujuConcepts

Needs to be update it a bit with some of the information below.

> Each machine in a Juju environment runs a jujud binary. The
> binary is packaged in the so-called tools tarball.

I seem to have noticed 'jujud' is per-unit rather than
per-machine, but then I think that there is a very noticeable
bias of Juju development towards one-unit-per-node on dynamic
public "cloud" providers... :-)

> The bootstrap process needs to download the tools from
> somewhere to the initial Juju Server.

That would be I guess the Juju "controller" machine, which is
not necessarily any of the MongoDB repset.

> For deployments with internet access, the tools come from
> https://streams.canonical.com/juju/tools/. This is the
> simplest case and doesn't require any agent-metadata-url or
> sync-tools usage. I may have missed it in your emails, but I'm
> assuming your environment does have internet access?

The Juju controller, the Juju state machines and the Juju nodes
I am dealing with all have Internet access. They are on various
private subnets, but the Juju controller also has a public
address, and the others are NAT'ed.

> [ ... ]  we now store charms and tools in the environment
> blobstore.

Thanks for the details!

> So the above is for bootstrap.

So far so good, and it looks like bootstrap worked around May
this year.

> For upgrades, if the machines in your environment have
> internet access, then juju upgrade-juju --version=1.24.5
> should just work.

That's a bit vague. though. I would run 'juju upgrade-juju' on
the control node, which has got 1.24.5 and then "somehow" the
~70 units with 'jujud' 1.23.3 deployed on the local 12 nodes
would then download the '.tgz' for their architecture of version
1.24.5, but that does not happen and I got instead the error
message "ERROR no matching tools available" which seems to be
coming from the 'juju' command running on the control node
itself.

The reason I am looking for details is to know a bit in advanced
at to what 'strace' or where to 'tcpdump' to see what is
actually broken.

This quote tells me that at least in some cases it is the unit's
'jujud' that fetches its own replacement:

  >> http://askubuntu.com/questions/555281/ubuntu-maas-juju-bootstrap-stuck-on-fetching-tools
  >> Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz 'https://streams.canonical.com/juju/tools/releases/juju-1.20.11-trusty-amd64.tgz'

and in that example is that it is fetched direct from the
"simplestream" at the Canonical site.

> $ juju sync-tools --version 1.24

Ahhh spotted that the ".5" is not used here. Indeed I add it
indeed does not work. Unfortunately right now the Juju setup is
in an unfavourable state because of a MAAS upgrade issue.

> Once the above is run, the upgrade command should be able to
> find the latest 1.24 tools in the Juju Server blobstore.

I'll have a look later, I have a present problem with MAAS that
is blocking me, about to send another email about that.