Updating state on agent upgrade

[William Reade]

On Wed, Sep 25, 2013 at 9:42 AM, Andrew Wilkins <
andrew.wilkins at canonical.com> wrote:

> On Wed, Sep 25, 2013 at 1:22 PM, Andrew Wilkins <
> andrew.wilkins at canonical.com> wrote:
>
>> On Wed, Sep 25, 2013 at 10:54 AM, Tim Penhey <tim.penhey at canonical.com>wrote:
>>
>>>  > All the above are pieces of info about the environment we could record
>>> > clearly in state, and which should apply to any manager node we start
>>> in
>>> > an HA context. Furthermore, across even non-management nodes, we can
>>> > know it's not even worth bothering to run any non-environ provisioner
>>> if
>>> > the environment can't supply new addresses; if we've got environment
>>> > capabilities recorded in state, we can know what needs to be done at
>>> the
>>> > time of machine creation.
>>>
>>> I'm not even sure that this information needs to be in state.  At least
>>> for the first cut of it.
>>>
>>

Well, *maybe*. The various methods of adding a machine will all need access
to this information or we *will* screw it up, and we don't have a single
place where we can enforce sanity: we've got InjectMachine in
bootstrap-state, and IIRC we've got AddMachine accessible both via the API
and directly on state, not to mention (effectively) internally to state
when we assign a unit to a new machine. We can either keep it in state or
we can contort ourselves to

If we keep the information in state, we can always get it at the right
moment inside addMachineOps (or whatever it is); if we rely on clients to
always calculate the jobs in the same way then we take on an awful lot of
distributed sanity-maintenance work, and we all know that's a bad idea.
ISTM that state is the only reasonable place to store this.


> Also, we do have a problem with considering that having multiple environ
>>> provisioners as they are currently defined is going to cause race
>>> conditions on starting / stopping containers unless we add extra
>>> metadata to state so one provisioner doesn't try to stop a machine
>>> another is starting.  Actually given the HA story, it is better to have
>>> two working collaboratively than a fail-over we have to manage.
>>>
>>
*Container* provisioning should not be affected by HA, but env machine
provisioning will indeed be.

I agree that it's better to write tasks that can work collaboratively than
to have to manage failover ourselves; the question is whether it's easier
to update N tasks such that they can all work collaboratively, or to slap a
giant shared failover mechanism on top of the group. I'm open to arguments
either way.

I think that conceptually, "capability" makes sense for some things more
>> than job/role. In particular, "has the ability to manage firewalls" seems
>> better expressed as a capability than as a job. However, I don't think it's
>> really worthwhile changing code to match. A capability can be expressed as
>> a job, even if it's *slightly* awkward. The fact that we're giving a
>> machine-agent the job "ManageFirewall" implies that it has that capability.
>>
>
It's not so much how we express the fact as *where* we express it. If the
only way we store important *environment* properties is by tacking them
onto a (particular, special) *machine* I think we're setting ourselves up
for trouble.


> Yep. As discussed on IRC, this could just as well be done with the
>> key/value map. I kind of don't like adding required things into a key/value
>> map, but on on the other hand this is bootstrap-specific, and not something
>> the machine-agent proper cares about. Not changing the format is good, too.
>>
>
I don't like the key/value map either tbh, but I think it' the least worst
way of getting the data where we need it. I just worry that lots of other
things may be able to get at that data too and may start to abuse it -- I
honestly feel that all the k/v data is borderline abusive anyway, and I'd
prefer not to entrench its use unless we really can't figure out a better
solution. I *do* feel that k/v would be great if it were used exclusively
to feed bootstrap-state, and was elided from the machine agent config --
because, really, the source of truth for pretty much *everything* should be
the API.

We don't need to add the capabilities to the config.  We could add them
>>> to the information that the machine api gets back.  However, since the
>>> machine agents don't know what the environment is, it takes us back to
>>> storing the roles (jobs) in state.
>>
>>
Agreed; and I remain pretty much steadfast that state is where we *should*
be storing the capability information.

We need a state side, server upgrade process defined.  Enough of this
>>> ad-hoc jiggery-pokery.
>>>
>>
The jiggery-pokery seems to me to be inescapable while clients and manager
nodes are still making uncontrolled DB connections. It's evil and it sucks
and everyone hates it, but the alternatives seem no better (and rather more
prone to possible corruption).


> We also need a defined process for upgrades.  I'm not sure how close we
>>> are to this right now, but I think we need something like this:
>>>
>>> 1) Put the API server into a state where it continues to serve requests,
>>> but doesn't accept new connections.
>>> 2) The tool version is updated causing all machine agents to kill
>>> themselves.
>>> 3) We need some form of state-side lock to allow only one state server
>>> to modify the underlying structure, and a defined process of functions
>>> to run to modify the state documents to the next version. [1]
>>>
>>> This process needs to be defined, and stable, such that we don't delete
>>> it all when the next minor branch commit is done.
>>>
>>> 4) When the state servers have been upgraded, we then kick off the api
>>> servers, which the machine agents can then connect to.
>>>
>>
>> This sounds sane to me.
>>
>
Yeah: that sounds like a very sane starting point. Do you see a way to get
this to work started usefully without isolating state completely?

Yep. After I sent the email yesterday, I began thinking that this upgrade
>> functionality is going to be exactly what's needed for updating the state
>> schema. I've got a few things to finish off (authenticated httpstorage is
>> half down; still need to document manual provisioning). Pending
>> cloud-installer work, I can start looking into this in a bit more detail.
>>
>
That sounds good to me if we've got a path forward.


> Vague ideas at the moment:
>> - Add a version to the state database (I suppose there'd need to be some
>> kind of metadata document collection), to track required schema changes.
>> - Add a state/upgrade package, which keeps a full history of
>> point-to-point schema updates required. We iterate through version changes,
>> applying upgrade steps one at a time. Everything must be done in a
>> transaction, naturally.
>>
>
FWIW, everything will I think have to be done in a long series of
transactions, especially if we're considering arbitrary changes. Shouldn't
be a big deal so long as the various changes are idempotent, I think.


> - One API server will (with a global lock):
>>    * First upgrade the state database. All other code can be written to
>> assume the current database schema.
>>    * Invoke an EnvironUpgrader interface method, optionally implemented
>> by an Environ. This interface defines a method for upgrading some
>> provider-specific aspects of the environment (e.g. going through and adding
>> jobs to all of the state-server machines). The EnvironUpgrader will
>> similarly need to keep track of versions, and point-to-point upgrades.
>>
>
Strong -1 on getting the environ involved directly in this case. Juju
should itself be able to discover the capabilities of the environment, and
react to them, rather than having the environment control the contents of
state. The idea of an EnvironUpgrader *is* itself worthwhile, but it should
be used to fix things that are part of the environment, rather than part of
state, and I don't think the need is so pressing right now.


> [1] I think the stance of only supporting upgrades to the next public
>>> release is crackful.  Consider a user that has a working juju install
>>> and has not needed to poke it for ages.  They are on 2.2.0.  They read
>>> the latest juju docs that show new amazing juju-gui stuff that requires
>>> 2.10.0.  We should not make them go 2.2 -> 2.4 -> 2.6 -> 2.8 -> 2.10 as
>>> that is clearly a terrible end user experience.
>>>
>>
>> From a user POV, that sounds pretty horrible.
>>
>
I strongly agree that when we're in a position to manage upgrades sanely,
we should by default apply all upgrades until we reach the latest version.
The requirement to advance in .2s is only intended to last as long as the
exposed state problem.

Cheers
William


>
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20130925/04bab82e/attachment-0001.html>