Peers vs Members vs Terminology vs Usability

[Stuart Bishop]

On 30 January 2017 at 02:55, James Beedy <jamesbeedy at gmail.com> wrote:


> I've been using elasticsearch-base as a playground for developing this, as
> (for the moment) the units of the elasticsearch-base charm only need to
> know what units of an application exist when a unit joins or departs the
> relationship so it can update elasticsearch.yml and restart elasticsearch.
> I started out with a working implementation where all units of the
> application could know about eachother by updating the unitdata with their
> ip when they join, following which all units of the application grab the
> new unit data and update their configs. Where I am hitting contention is
> trying update the unitdata cache with the current data for all existing
> members when a unit departs the relationship. Basically a diff of what it
> was before. It sounds like what you've suggested, using
> context.Relations().peer might be the pre-existing functionality that
> you've written that I can use to derive the functionality I'm looking for.
>
> Heres what I've got going on now using the leader to do the coordinating
> to some degree... I feel like its becoming a hack fest though ... possibly
> you could take a look?
>
> https://github.com/jamesbeedy/juju-layer-elasticsearch-base/
> blob/use_master_for_member_add/reactive/elasticsearch_base.py#L126
> <https://github.com/jamesbeedy/juju-layer-elasticsearch-base/blob/use_master_for_member_add/reactive/elasticsearch_base.py>
>

refresh_cache() is a worry.  Its a handler being run on the leader, but it
stores state locally. This state will be lost if the leader changes. The
only safe place for the leader to store state needed for leadership
decisions is leadership settings, because that is the only place it can be
found when a new leader is elected.




>
> I personally would not have found it useful, as when I've needed a peer
> relation I've needed a lot more than just knowing who the peers are and
> their addresses.
>
> - To this extent, possibly it would of made the path to what you were
> trying to accomplish a more pleasant experience if you had an api to access
> to the relational data for all of the units of its application type, and
> were able to just focus on what you were trying to do (looks like you have
> created context.Relations() specifically for this purpose)?
>

Yeah. The peer relation and the data on it is just the medium for the
protocol you are building (or some people prefer the term interface). The
context module was just my attempt at a Pythonic wrapper to the data
structure, so my implementation was more readable and maintainable.



>
> - This is very possibly exactly what I'm looking for. Does
> context.Relations().peer give you access to the relational data of your
> 'peer' in the context of Juju - such that a unit's only peer is the leader,
> and the leaders peers are all members that are not the leader? Or does this
> return relation data for all peers of the relation?
>

I don't think Juju has an idea of a unit's only peer being the leader and
the leader's peers being all the non-peer units. My understanding is that a
units peers are all the other units in the service/application, and it
doesn't matter if you are talking about the leader or not.

context.Relations().peer gives you access to the peer relation data of all
the other units in the application/service.



>
> and context.Relations().peer.local is a dictionary of the local peer
> relation data that you can update
>
> - Update with the information that context.Relations().peer returns? How
> is this different/better/worse then using leader to store the data, or unit
> data even? Because its data on the relation, which makes it fit the
> reactive model better/less cluttered?
>
> - I seem to be hitting an issue with layer leadership where only the
> leader is able to set data to the leader. Is this intended by design, or is
> this (what appears to be) a bug?
>

In Juju, only the leader can write to leadership settings by design.

If you want to use Juju leadership to coordinate operations over your units
(good idea - it works great), then the only reliable channel the leader has
to communicate with the other units is the leadership settings. If you try
to use the peer relation to communicate from the leader to the other units
then you will fail. This is because the leader can change, and the other
units have no way of knowing who is the leader (if they are not the leader,
the best they can tell is who *was* the leader - any information they get
about leadership may be out of date by the time they see it).

If your units need to communicate back to the leader or between each other,
the only channel they have to do so is the peer relation.

So, more practically using the leadership layer:

@when('leadership.is_leader')
@when_not('leadership.set.master')
def anoint_master():
    leadership.leader_set(master=hookenv.local_unit())

@when('leadership.is_leader')
@when('leadership.set.master')
def maybe_failover():
    master = leadership.leader_get('master')
    peer_rel = context.Relations().peer
    if master != hookenv.local_unit() and master not in peer_rel:
        leadership.leader_set(master=hookenv.local_unit())

@when('leadership.changed.master')
def follow_master():
    master = leader_get('master')
    rels = context.Relations()
    rels.peer.local['following'] = master
    reconfigure(master=master)  # maybe master is us

@when('leadership.set.master')
def update_status():
    master = leader_get('master')
    peer_rel = context.Relations().peer
    for peer, reldata in rels.peer_rel.items():
        if reldata.get('following') != 'master':
            hookenv.status_set('waiting', 'Waiting for cluster
coordination')
            return
    if master == hookenv.local_unit():
        hookenv.status_set('active', 'Master')
    else:
        hookenv.status_set('active', 'Secondary to {}'.format(master))

It seems to work best if the handlers making the leadership decisions are
separate to the handlers that act on them. Structured this way, your charm
does not care which unit is leader and it can change between any two hooks
and you don't care.


>
> (but note that the design has a bug in
> it, as it assumes there is a single peer relation while in reality charms
> can have multiple peer relations. You will consistently get the same peer
> relation, but it might not be the one you expect).
>
> - Uggg, possibly I've already been hitting this.
>

Only if you have more than one peer relation declared in metadata.yaml.
Which is pretty unusual.




-- 
Stuart Bishop <stuart.bishop at canonical.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20170130/58f610d8/attachment.html>