Juju on MAAS agent tools upgrade mechanism

[Peter Grandi]

>> [ ... 1.23.3 has some excessive lease/txns traffic fixed in
>> 1.24.5 ... ]
> [ ... ] You may find you have a problem upgrading away from
> 1.23 (again, due to problems with the new lease feature). I
> created a Juju plugin to help work around this. [ ... ]

I have acquired the plugin and found in the "cheatsheet" a hint
on how to get it and have it recognized, but I haven't been able
yet to use it because of a different problem.

The Juju infrastucture that 1.23.3 is running on is managed by
MAAS, and perhaps because of that I haven't been able to get the
1.24.5 tools to get installed, while the '.deb' for 1.24.5 was
installed without issue on the "control" node.

The first issue was that 'juju upgrade-juju' reported no newer
version and that version 1.24.5 was unknown. The colleague who
upgraded from 1.23.2 to 1.23.3 some months ago thinks it all
"just worked", but after much searching I figured out that there
have been a few changes and MAAS is a special case as 'juju help
upgrade-juju' states that:

  «Both of these depend on tools availability, which some
  situations (no outgoing internet access) and provider types
  (such as maas) require that you manage yourself; see the
  documentation for "sync-tools".»

and indeed some tutorials for Juju on MAAS throw in a
'sync-tools' line. However that did not work for me, and I got
an error message with '1.24.5--amd64', and then during my
attempts to work around by building a local 'tools' cache as
suggested by a bug report, with SHA256 mismatches (the report of
such mismatches was wrong).

Eventually I managed to build a local 'tools' cache that seems
to work containing just:

  local-tools/tools/releases/juju-1.24.5-trusty-amd64.tgz
  local-tools/tools/streams/v1/index.json
  local-tools/tools/streams/v1/index2.json
  local-tools/tools/streams/v1/com.ubuntu.juju-released-tools.json

and I also fixed the 'toolsmetadata' collection as indicated in
a bug report and now it looks like:

  { "_id" : "1.23.2-trusty-amd64", "version" : "1.23.2-trusty-amd64", "size" :
		  NumberLong(11555177), "sha256" :
		  "acabf7b8f9d9a9d718a083f80355dfbdce228bb2f8c4e9cfab7899c730f7290b",
		  "path" :
		  "tools/1.23.2-trusty-amd64-acabf7b8f9d9a9d718a083f80355dfbdce228bb2f8c4e9cfab7899c730f7290b",
		  "txn-revno" : NumberLong(2), "txn-queue" : [
		  "55534945705cc83c16000038_2232d0f1" ] }
  { "_id" : "1.23.3-trusty-amd64", "path" :
		  "tools/1.23.3-trusty-amd64-007c62a742c974c3f082964f37b04c28d46345e4816a926c31f8bdef53000552",
		  "sha256" :
		  "007c62a742c974c3f082964f37b04c28d46345e4816a926c31f8bdef53000552",
		  "size" : NumberLong(11566458), "txn-queue" : [
		  "558c54bc705cc881670003fd_9cb7835d",
		  "558c54bc705cc881670003fe_d9e05d7b" ], "txn-revno" :
		  NumberLong(2), "version" : "1.23.3-trusty-amd64" }
  { "_id" : "1.24.5-trusty-amd64", "version" : "1.24.5-trusty-amd64", "size" :
		  NumberLong(16649545), "sha256" :
		  "e080a20aed15abb1e131dec2bafa227ac395cfb5710b10c05d82f9c50243a497",
		  "path" :
		  "tools/1.24.5-trusty-amd64-e080a20aed15abb1e131dec2bafa227ac395cfb5710b10c05d82f9c50243a497",
		  "txn-revno" : NumberLong(2), "txn-queue" : [
		  "55e0745a705cc8ff13002089_6cf89b22" ] }

That seems "consistent" to me. But 'juju upgrade-juju' still
tells me that no suitable version is available.

Given the numerous changes and bug fixes I don't know if there
is *any* documentation that can help, so I have been a bit
reverse engineering the various mechanisms involved.

But it is still not clear to me *how* the agent's tools are
supposed to be updated and from where for MAAS and in other
cases. "Something" must download a certain '.tgz' and unpack it
in every unit directory when this is triggered by 'juju
upgrade-juju'.

How is the agent tools upgrade mechanism is supposed to work,
for MAAS at least?