What is the proper procedure to upgrade juju (from 1.18 to 1.24)

[Mario Splivalo]

Hi, lads (and ladies!)

I need to upgrade customer who's running juju 1.18 (on a fairly large
setup, 100+ nodes), and I'm not sure what's the proper procedure to do so.

I tried (in our lab, local OpenStack) upgrading juju-core package to
1.24, and then running 'upgrade-juju', but that just left my environment
unusable.

Then I opted for a 'intermediate' step - I'll upgrade to 1.20 (which is
in trusty-updates, and from there I'll upgrade to 1.24.

So, I installed 1.18 from trusty (apt-get install
juju-core=1.18.1-0ubuntu1), bootstraped, and deployed two simple
services, one-unit ubuntu service (u1) and three unit ubuntu service (u2).

After services were installed I run upgrade-juju (with juju-core still
being 1.18)

juju upgrade-juju

That upgraded agents on all the installed units to 1.20.14. (juju status
confirmed that, as well as symlinks in /var/lib/juju/tools on each of
the deployed units).

After that I upgraded juju-core (on my control node) to 1.20.11. But,
after that running 'juju upgrade-juju' yielded with "no upgrades
available". As I was running 1.20 agent on all of the installed units I
went further and upgraded juju-core to 1.24 (from juju stable ppa), and
tried 'juju upgrade-juju' again. But again I was greeted with "no
upgrades available".

I then run 'juju upgrade-juju --dry-run --version 1.24.0', and this time
juju showed me available versions and suggested that 1.24 was the best one:

ubuntu at mariosplivalo-bastion:~$ juju upgrade-juju --dry-run
no upgrades available
ubuntu at mariosplivalo-bastion:~$ juju upgrade-juju --dry-run --version 1.24.0
available tools:
    1.24.0-precise-amd64
    1.24.0-precise-armhf
    1.24.0-precise-i386
    1.24.0-trusty-amd64
    1.24.0-trusty-arm64
    1.24.0-trusty-armhf
    1.24.0-trusty-i386
    1.24.0-trusty-ppc64el
    1.24.0-utopic-amd64
    1.24.0-utopic-arm64
    1.24.0-utopic-armhf
    1.24.0-utopic-i386
    1.24.0-utopic-ppc64el
    1.24.0-vivid-amd64
    1.24.0-vivid-arm64
    1.24.0-vivid-armhf
    1.24.0-vivid-i386
    1.24.0-vivid-ppc64el
best version:
    1.24.0
upgrade to this version by running
    juju upgrade-juju --version="1.24.0"
ubuntu at mariosplivalo-bastion:~$ juju upgrade-juju --dry-run
--version="1.24.0"
available tools:
    1.24.0-precise-amd64
    1.24.0-precise-armhf
    1.24.0-precise-i386
    1.24.0-trusty-amd64
    1.24.0-trusty-arm64
    1.24.0-trusty-armhf
    1.24.0-trusty-i386
    1.24.0-trusty-ppc64el
    1.24.0-utopic-amd64
    1.24.0-utopic-arm64
    1.24.0-utopic-armhf
    1.24.0-utopic-i386
    1.24.0-utopic-ppc64el
    1.24.0-vivid-amd64
    1.24.0-vivid-arm64
    1.24.0-vivid-armhf
    1.24.0-vivid-i386
    1.24.0-vivid-ppc64el
best version:
    1.24.0
upgrade to this version by running
    juju upgrade-juju --version="1.24.0"


So I did what was suggested:

ubuntu at mariosplivalo-bastion:~$ juju upgrade-juju --version="1.24.0"
available tools:
    1.24.0-precise-amd64
    1.24.0-precise-armhf
    1.24.0-precise-i386
    1.24.0-trusty-amd64
    1.24.0-trusty-arm64
    1.24.0-trusty-armhf
    1.24.0-trusty-i386
    1.24.0-trusty-ppc64el
    1.24.0-utopic-amd64
    1.24.0-utopic-arm64
    1.24.0-utopic-armhf
    1.24.0-utopic-i386
    1.24.0-utopic-ppc64el
    1.24.0-vivid-amd64
    1.24.0-vivid-arm64
    1.24.0-vivid-armhf
    1.24.0-vivid-i386
    1.24.0-vivid-ppc64el
best version:
    1.24.0
ubuntu at mariosplivalo-bastion:~$

Although the output from the command didn't suggest it, actually the
upgrade started in the background. After some time 'juju status'
returned output, and everything seemed to be ok, but the agent version
was still 1.20. Inside /var/lib/juju/tools there was 1.24.0-trusty-i386
directory, but the symlinks for machine-X etc were still pointing to 1.20:

ubuntu at juju-mariosplivalo-machine-1:/var/lib/juju/tools$ ls -al
total 20
drwxr-xr-x 5 root root 4096 Jun 30 17:11 .
drwxr-xr-x 5 root root 4096 Jun 30 16:51 ..
drwxr-xr-x 2 root root 4096 Jun 30 16:51 1.18.4-trusty-i386
drwxr-xr-x 2 root root 4096 Jun 30 17:02 1.20.14-trusty-i386
drwxr-xr-x 2 root root 4096 Jun 30 17:11 1.24.0-trusty-i386
lrwxrwxrwx 1 root root   19 Jun 30 17:02 machine-1 -> 1.20.14-trusty-i386
lrwxrwxrwx 1 root root   19 Jun 30 17:02 unit-u1-0 -> 1.20.14-trusty-i386


However, when I try to add another service (juju deploy ubuntu u3), that
one never finishes - the machine is brought up, I can ssh into it (using
ssh keys from ~/.juju/), but juju status for that service shows this:
  u3:
    charm: cs:trusty/ubuntu-3
    exposed: false
    service-status:
      current: unknown
      message: Waiting for agent initialization to finish
      since: 30 Jun 2015 17:25:36Z
    units:
      u3/0:
        workload-status:
          current: unknown
          message: Waiting for agent initialization to finish
          since: 30 Jun 2015 17:25:36Z
        agent-status:
          current: allocating
          since: 30 Jun 2015 17:25:36Z
          version: 1.24.0
        agent-state: pending
        agent-version: 1.24.0
        machine: "5"
        public-address: 10.5.2.30



Am I doing something wrong here? Is this the proper procedure to upgrade
juju across a live deployment?

	Mario