Landing bot back online

[Martin Packman]

Our landing bot is operational again after the canonistack lcy02
maintenance earlier today.

There were quite a few hitches along the way, some of which are
probably attributable to the fact we're still running an ancient juju
1.11.0 in that environment.

First up the state server came up without mongo from running out of
disk space... which made double sure by filling up the logs with
attempting to connect to mongo.

Then the provider-state file somehow didn't have anything in it, so I
manually uploaded a reference to the state server id to swift. That
got status working, but with "instance-state: error". I added logging
to the client, mongo instance data claimed not to find the machines.
But a restart of the second machine and some other fiddling and
mysteriously everything was fine.

I then got one successful config-changed through to update the tarmac
conf - unfortunately missing one part that had been updated locally
and with a bad change to the test commands. From there, no more `juju
set` invocations actually resulted in the hook firing, though the unit
agent did pick up the first step.

(Misc note, not being able to `juju set` what you `juju get` is just
frustrating).

So, I looked at upgrading the juju version to something current... Not
a good idea. There have been so many tools location specification
changes in the mean time, I couldn't get a combination that both a
recent client and the old agents could understand. That also left the
upgrade stuck in an uncompleteable state, so I hacked the
agent-version back to what it was before.

After that, for whatever reason config-changed started firing again,
so I finally fixed up the tarmac config and confirmed changes could
land again.

We should probably set up a new environment with a current juju after
fixing up the tarmac charm to have some more of the things we needed
to manually hack in last time around.

Martin