Using Openstack scheduler hints with juju

[Martin Packman]

In cases where you care about which physical host a juju machine is
deployed on, some specific openstack configuration is required, and an
amendment to the juju code landed on trunk.


We currently have production services which have multiple units all
located on the same host, what we want is those units spread across
multiple hosts so a single hardware failure is not fatal.

Openstack supports this through passing a 'scheduler hint' on server
boot, and experimental support was added[bug#1049858] for this in juju
a while back.  This week we had cause to try it out in production, and
ran into a few complications.


The basic idea is to set a constraint, with a json string value which
is passed through to the server boot api call to inform the nova
scheduler.

So, if you want the next unit to be on a different physical machine
from a another, you can pass or set a constraint like so:

    juju set-constraint
'os-scheduler-hints={"different_host":"<some-server-uuid-here>"}'

If you want it on the same machine as another:

    juju set-constraint
'os-scheduler-hints={"same_host":"<some-server-uuid-here>"}'

A list of string uuids can also be given rather than a single string
uuid. Spaces are accepted by the json parser, but are a little tricky
to handle in constraint specification so are carefully omitted in
these examples.


However the landed juju code gets the api slightly wrong, the hints
key should appear in the root dictionary rather than under the
'server' key. Note though using the 'OS-SCH-HNT:' prefix rather than
the generic 'os:' present in the documentation is fine, and mildly
preferred.

There are several configuration options that are involved on the
openstack deployment side, if any are set incorrectly nova will
happily schedule you a server anyway, but without understanding or
acting on any hints given.

Firstly, a compute api extension is required otherwise the key won't
be passed through. The default enables all builtin extensions and is
fine:

    osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions

Appending nova.api.openstack.compute.contrib.scheduler_hints is
required for deployments that instead selectively enable extensions.
The config option osapi_compute_ext_list also exists, I'm not sure how
these two interact.

Secondly, FilterScheduler must be used which can be done for compute
only if MultiScheduler is the base scheduler, as in the defaults:

    scheduler_driver=nova.scheduler.multi.MultiScheduler
    compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

One of our deployments uses nova.scheduler.chance.ChanceScheduler
instead, so hints are silently ignored there.

Then the relevant filters need to be enabled, by default they're all available:

    scheduler_available_filters=nova.scheduler.filters.all_filters

But the ones we need are not used and must be added to the list:

    scheduler_default_filters=AvailabilityZoneFilter,RamFilter,ComputeFilter

Appending ',DifferentHostFilter,SameHostFilter' should be enough, but
awaits testing.

Getting that that configuration value changed is the last wrinkle that
we're stuck on for now. We deploy Openstack itself using juju, and the
charms[nova-compute] used have a neat option
'config-flags'[config-flags] that should allow (live?) nova.conf
updating.

However, it's defined as a 'string' when it's really a dict of comma
separated key-pairs. As the config value we need to change also
contains commas, and both the charm[shell-parsing] and nova are naive
about parsing. Is there any reason we can't make this option a yaml
dict and teach the charm how to interpret that?

One further possibility is in availability zones which could be used
to similar effect, and for which the filter is enabled by default.
This is for logical partitioning of services however, so is somewhat
abstracted from the actual hardware.

It's also not to be confused[availability-zone-confusion] with the old
Zones code, which was broken in diablo and removed in essex. The
replacement Cells concept[nova-compute-cells], which is landing for
grizzly, is potentially also of interest for juju, though aimed at a
larger scale, splitting regions into distinct, but connected,
locations.

So, we can nearly do what we want with the current state of juju and
openstack, but hopefully we can get something less fiddly out as a
more general solution.


Martin



[bug#1049858] "Juju should support physical placement on OpenStack"
<https://bugs.launchpad.net/juju/+bug/1049858>

[nova-compute] One of the charms we use to deploy openstack using juju
<lp:~canonical-sysadmins/canonical-marshal/nova-compute>

[config-flags] From the nova-compute charm config.yaml we can update nova.conf
  config-flags:
    default: None
    type: string
    description: Comma separated list of key=value config flags to be
set in nova.conf.

[shell-parsing] The parsing in hooks/lib/nova/nova-common can't unescape commas
  function set_config_flags() {
    # Set user-defined nova.conf flags from deployment config
    juju-log "$CHARM: Processing config-flags."
    flags=$(config-get config-flags)
    if [[ "$flags" != "None" && -n "$flags" ]] ; then
      for f in $(echo $flags | sed -e 's/,/ /g') ; do
        k=$(echo $f | cut -d= -f1)
        v=$(echo $f | cut -d= -f2)
        set_or_update "$k" "$v"
      done
    fi
  }

[availability-zone-confusion] "Implement Availability Zone scheduler"
<https://answers.launchpad.net/nova/+question/194109>

[nova-compute-cells] "Nova Compute Cells"
<http://wiki.openstack.org/blueprint-nova-compute-cells>
<https://blueprints.launchpad.net/nova/+spec/nova-compute-cells>