Network Space Binding Changes After Deployment or Upgrades

Tue Mar 28 12:00:52 UTC 2017

John,

Thank you for the clarification!

Haven't thought about the SDN part but this is definitely something
that occurs in practice. VPNs come to mind as well: besides making a
tunnel via network infrastructure and setting up routing (or a
separate vlan for that VPN traffic), one could spin up tunnels
on-demand at endpoints themselves - that would require MAAS to gain
VPN support in such a way that images deployed by it would be able to
create tun/tap or GRE interfaces on demand much like you can do with
VLAN interfaces now.

I can also think of a scenario where one charm is a VPN service which
is deployed onto nodes with a good crypto acceleration and other
charms are its clients which dynamically learn which network space to
bind to for their application traffic based upon the information
provided by a VPN unit. Clients would then use some form of an
auto-discovery mechanism to connect to services via a newly learned
network. It is a very high-level idea but it may make sense
nevertheless.

As for rebinding and containers, since Juju is gaining support for new
container types (qemu/kvm VMs are 'containers' in Juju's terminology
AFAIKS as the container interface in Go is implemented for them) it
might be worthwhile to think about how this will be done at the kernel
level.

https://github.com/juju/juju/blob/master/instance/container.go
https://github.com/juju/juju/blob/master/container/interface.go

The decision here I think is whether to do an interface hot-plug or
reuse an existing interface for binding changes. E.g. you are no
longer bound to a particular network space - do you hot-unplug an
interface or do you leave it hanging? Or do you plug the same
interface into a different bridge?

Changing a bridge for an existing interface on a live container or a
VM can be complicated.

For containers created by lxd I checked that it is possible to do:

lxc launch ubuntu:xenial ct
brctl dellif <old_br_name> <iface_name>
brctl addif <new_br_name> <iface_name>
lxc exec ct bash
dhclient
# ping via a new bridge works without ct restart

As for VMs I'd look at the vhost reconnect patches that were pushed
into QEMU in 2016 and see if this works not only the "reconnect to the
same bridge" case but also for changing a bridge to a different one.

https://www.mail-archive.com/qemu-devel@nongnu.org/msg384411.html
https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg02927.html
(an old thread)

src/qemu $ git log
4616e3592ba62dd44d62babd84b23f84f6b33ddb~10..4616e3592ba62dd44d62babd84b23f84f6b33ddb

http://git.qemu-project.org/?p=qemu.git;a=commit;h=4616e3592ba62dd44d62babd84b23f84f6b33ddb
Best Regards,
Dmitrii Shcherbakov

Dmitrii Shcherbakov  | Canonical
Field Software Engineer
dmitrii.shcherbakov at canonical.com
IRC (freenode): Dmitrii-Sh

On Tue, Mar 28, 2017 at 12:09 PM, John Meinel <john at arbash-meinel.com> wrote:
>
>
> On Mon, Mar 27, 2017 at 3:31 PM, Dmitrii Shcherbakov
> <dmitrii.shcherbakov at canonical.com> wrote:
>>
>> Hi everybody,
>>
>> As far as I can see, there is no way to change a network space binding or
>> add a new one after a charm has been deployed.
>
>
> The one caveat here is that in 2.1.2+ if you used a 'default' binding (juju
> deploy --bind foo) then all new endpoints for a charm will also be bound
> into whatever the default was for that application.
>
>
> That is currently correct. We have been discussing it, and how we want to
> concretely model the evolution of spaces over time. One of the particular
> points of concern is whether we would ever want to allow multiple units of
> the same application to be configured differently.
> For example:
>   $ juju deploy app --constraints mem=4G
>   $ juju set-constraints app mem=2G
>   $ juju add-unit app
>
> You'll now have 2 units, where they have different expected memory. However,
> we feel that supporting that for bindings would be a net negative, because
> it complicates the *user's* mental model of where everything exists. An
> other applications that relate to the app need more complex thoughts when it
> comes to "am I able to talk to all of the units?".
>
> Complications like that are what led to us not implementing rebinding from
> the beginning.
>
> Our current plan is to have some sort of "rebind an endpoint" command
> (name-to-be-determined juju set-binding, juju rebind, etc). This would
> include a sanity check that all of the units that are currently deployed
> have access to the space that you are requesting. eg:
>  juju deploy app --bind foo --constraints spaces=baz
>  juju rebind app blah=baz
> Would be allowed, because all units of "app" must be in both "foo" and "baz"
> spaces.
>
> We should also support "--bind" as a parameter to "juju upgrade-charm",
> since the set of endpoints that we know about can be changing, and you
> should be able to specify values for the new entries, rather than inherit
> the default and then need to be updated.
>
> The caveat is that if all units don't have access to the new space, then we
> should fail. Which would still let you "remove units not in the space I
> want" (which may be all of them), "rebind", "add-units back to however many
> I want".
>
> The other complexity that I know is lurking around the corner is that as
> soon as we have the ability to rebind, people are going to want to do:
>   juju add-machine --constraints spaces=foo,bar
>   juju deploy app --bind foo --to lxd:0
>   juju rebind app bar
>
> Which will require us to change the network bindings of a running container.
> Which is ultimately something I think we want, but will also likely not be
> in the first implementation of rebind. (For anything like an SDN we should
> be able to rebind to whatever spaces that machine can have access to, rather
> than what spaces it is currently connected to.)
>
>>
>> What I encountered is a situation where you either want to add a new
>> endpoint (and bind it to a space) after a charm upgrade or modify an
>> existing one. With the current approach this is not possible without a full
>> redeployment of an application (remove-application -> deploy & bind ->
>> add-unit).
>>
>> However, for deployments with large uptime requirements this is not a
>> feasible solution.
>>
>> My view of the problem:
>>
>> Juju doesn't know how a charm code have used binding information during a
>> deployment;
>> Juju can expose an event which should be handled in the charm code;
>> There is a distinction between "binding-changed" and "upgrade" events as
>> the latter may include binding changes but needs a mechanism to learn which
>> bindings have changed.
>>
>> --
>> Only the deployment-related code has handling of bindings:
>>
>> https://github.com/juju/juju/blob/master/cmd/juju/application/deploy.go#L615
>>
>> https://github.com/juju/juju/blob/staging/cmd/juju/application/bundle.go#L382
>>
>> Nothing about bindings here:
>>
>> https://github.com/juju/juju/blob/master/cmd/juju/application/upgradecharm.go
>> ---
>>
>> I realize that this is a complex feature and will require code on the
>> charm side but I think it is important as it makes deployments less rigid
>> and more update-friendly.
>>
>> Has it been discussed before?
>
>
> It has certainly been discussed, we're just not there yet.
>
> John
> =:->
>
>>
>>
>>
>>
>> Any feedback is appreciated - thanks in advance.
>>
>> Best Regards,
>> Dmitrii Shcherbakov
>>
>> Field Software Engineer
>> IRC (freenode): Dmitrii-Sh
>>
>> --
>> Juju-dev mailing list
>> Juju-dev at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>