Trying to remove and redeploy quantum-gateway on a different server

[Daniel Bidwell]

On Sun, 2015-09-20 at 11:59 +0100, Mark Shuttleworth wrote:
> Hi Jeff
> 
> Glad you found the issue! It's pretty cool that you could just
> reconnect the bits with a different mapping to VMs and voilà it
> works :)
> 
> Generally, Juju will provide you with the fastest way to "do what you
> want", and to the extent the charms have seen your situation before, a
> best practice result. In this case we can enhance that charm to error
> helpfully (i.e. with a message) saying "I need two network interfaces,
> please put me on an appropriate VM or real machine".
> 
> For those people who really want to understand every aspect of
> OpenStack, this is a great way to do what you want, and experiment
> quickly, to get the large scale cloud you want. We just added detailed
> support for Contrail 2.2 for example, which is fun to test as a real
> SDN at scale.
> 
> But we think most people do not need or want to be OpenStack experts;
> they want OpenStack to "Just Work" so they can focus on their apps and
> workloads.
> 
> That's why we also built the Autopilot, which is free for up to 10
> nodes (thereafter free with out standard support offering) and it
> will:
> 
>  * study the hardware you provide it
>  * offer you a choice of SDNs and storage and other components
>  * create the appropriate reference architecture of containers, metal
> and VMs
>  * monitor the resulting cloud
> 
> Once you've deployed, you can feed it more machines to grow, and it
> will evolve the reference architecture accordingly.

Does this mean that I can use a local install of landscape and grow it
to more than 10 nodes without getting support for it?  I stopped using
landscape because I can't afford to use something that will require
software support when it grows larger than a small cluster.

We are a small, private university without much funding, but would very
much like to build a private openstack (the easy way).

> 
> If you're going to be building lots of these clouds, or need to focus
> at a higher level, please consider the autopilot.
> 
> Mark
> 
> On 19/09/15 16:48, Jeff McLamb wrote:
> 
> > Solved! TL;DR - You must deploy quantum-gateway or neutron-gateway to bare
> > metal. It will not work in an LXC container.
> > 
> > It appears as though you cannot deploy the {quantum,neutron}-gateway charm
> > to an LXC. I eventually tracked down the stuck config-changed state due to
> > the openvswitch vswitchd daemon not being able to start because it cannot
> > load the kernel module properly, which causes a loop whereby it keeps
> > attempting to add flows to the br-int OVS but fails because the daemon is
> > not running.
> > 
> > I removed the service from the LXC and deployed it to a bare metal machine
> > (co-hosted with Ceph services that don’t seem to do anything fancy with
> > bridges or OVS) and everything worked just fine.
> > 
> > Thanks,
> > 
> > Jeff
> > 
> > On Fri, Sep 18, 2015 at 4:29 PM, Jeff McLamb <mclamb at gmail.com> wrote:
> > 
> > > Thanks for the info, Mike! I will provide a bit more detail as to what’s
> > > going on in case any Juju experts have seen something similar.
> > > 
> > > Where I last left off, I had deleted all units of anything
> > > {neutron,quantum}-gateway related from juju’s perspective. However, the
> > > original bare metal node was still providing the neutron gateway (i.e.
> > > network node) services, so I manually issued a stop on all of those (e.g.
> > > stop neutron-dhcp, stop neutron-metadata-server, stop neutron-metering,
> > > etc.) The only thing I left running on that server was openvswitch because
> > > it is also a compute node, so it needs to have the openvswitch plugin
> > > running, as I understand it.
> > > 
> > > I then issued a `juju deploy neutron-gateway —to lxc:3` in order to
> > > install the gateway in an lxc container.
> > > 
> > > The lxc container is provisioned and everything seems to be going OK until
> > > it gets stuck in agent status -> message: running config-changed hook
> > > 
> > > If I ssh into the container itself and look at processes, I can see it is
> > > in a seemingly infinite loop where the neutron rootwrapper is constantly
> > > issuing ovs-vsctl add-br br-int calls, along with adding and removing
> > > flows. It’s as though it keeps trying to add and remove the br-int ovs, and
> > > just sits there doing that forever.
> > > 
> > > I manually killed the config-changed hook and then issued a `juju
> > > debug-hooks neutron-gateway/0 config-changed` followed by `juju resolved
> > > neutron-gateway/0 —retry`, which has dumped me into a config-changed tmux
> > > session where I manually issued ./hooks/config-changed.
> > > 
> > > There is not much other than the following output, after which it hangs:
> > > 
> > > root at juju-machine-3-lxc-11:/var/lib/juju/agents/unit-neutron-gateway-0/charm#
> > > ./hooks/config-changed
> > > Reading package lists... Done
> > > Building dependency tree
> > > Reading state information... Done
> > > Reading package lists... Done
> > > Building dependency tree
> > > Reading state information... Done
> > > Reading package lists... Done
> > > Building dependency tree
> > > Reading state information... Done
> > > python-dbus is already the newest version.
> > > The following packages were automatically installed and are no longer
> > > required:
> > >   libfreetype6 os-prober
> > > Use 'apt-get autoremove' to remove them.
> > > 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> > > 
> > > At this point everything hangs. I have not yet Ctrl-C’d the process yet.
> > > 
> > > In the /var/log/juju/unit-neutron-gateway-0.log it looks pretty standard,
> > > finishing all log output with Creating bridge br-int, Creating bridge
> > > br-ex, Creating bridge br-tun.
> > > 
> > > At this point there is no more streaming log output either.
> > > 
> > > Has anyone else seen this or have an idea what might be going on?
> > > 
> > > Thanks,
> > > 
> > > Jeff
> > > 
> > > On Fri, Sep 18, 2015 at 2:22 PM, Mike McCracken <
> > > mike.mccracken at canonical.com> wrote:
> > > 
> > > > Hi Jeff!
> > > > 
> > > > On Fri, Sep 18, 2015 at 11:12 AM, Jeff McLamb <mclamb at gmail.com> wrote:
> > > > 
> > > > > Hi all -
> > > > > 
> > > > > * Sorry for the cross-post on openstack-installer, but I think this is
> > > > > more of a juju question than an openstack-installer issue*
> > > > > 
> > > > > As I originally deployed my current test setup via openstack-installer,
> > > > > I am wondering how I am supposed to use it and/or juju directly for future
> > > > > changes/additions.
> > > > > 
> > > > Yes, the openstack-installer is intended to be a first step and not a
> > > > full purpose juju front-end. For future changes of the sort you're
> > > > describing, you'll want to get familiar with using juju directly. Note that
> > > > the official Juju web GUI is one of the services that you can install via
> > > > the manual placement UI in the installer.
> > > > 
> > > > I am trying to understand neutron networking, investigating all of the
> > > > > OVS and Linux bridges, how traffic flows between them and the network node
> > > > > etc. In my initial deployment I co-located a compute node and the network
> > > > > node on bare metal via manual placement in openstack-installer.
> > > > > 
> > > > > Now, in order to better see the separation of stuff (i.e. not conflate
> > > > > what the compute node is doing with the network node) I decided to `juju
> > > > > remove-service quantum-gateway` followed by a `juju add-service
> > > > > quantum-gateway —to lxc:3` in order to remove it from the original node and
> > > > > install it into its own container on the controller node that hosts lots of
> > > > > container-based services.
> > > > > 
> > > > > I have received lots of errors in the process during which various hooks
> > > > > get stuck. I have been able to get around it by manually killing hooks and
> > > > > then issuing a `juju resolved quantum-gateway` until finally the service
> > > > > goes away. Not to mention the fact that a new deployment seems to be in a
> > > > > “blocked” state saying I should upgrade to neutron-gateway. I tried that
> > > > > with similar results.
> > > > > 
> > > > > At this point I managed to get rid of everything — i.e. I do not have a
> > > > > quantum-gateway or a neutron-gateway as reported by juju.
> > > > > 
> > > > > However, I have noticed that the original server (colocated compute and
> > > > > quantum-gateway) still has all of the neutron services running. Issuing a
> > > > > `neutron agent-list` shows everything connected and happy as it ever was.
> > > > > It seems as though my juju fiddling has basically been just manipulating
> > > > > the juju database without actually removing the underlying services. I have
> > > > > noticed that when removing services on an lxc container it does indeed kill
> > > > > the container (so obviously the service too)… however, maybe because this
> > > > > original service was deployed on bare metal with another service, it
> > > > > doesn’t bother to actually stop the services and remove their /etc/init
> > > > > files?
> > > > > 
> > > > This does sound unexpected, hopefully someone more directly involved with
> > > > Juju can comment here.
> > > > 
> > > > 
> > > > > I have tried to redeploy via openstack-status, but it seems like it is
> > > > > useless for future deployment even though it has that nice curses-looking
> > > > > GUI from which I can deploy. It’s understanding of where everything is
> > > > > placed is completely broken, and it does not seem to honor any requests to
> > > > > place stuff in the first place.
> > > > > 
> > > > I'll follow up with you about the openstack-status tool on the installer
> > > > list, to keep this thread juju-specific.
> > > > 
> > > > 
> > > > > Any ideas?
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > Jeff
> > > > > 
> > > > > --
> > > > > Juju mailing list
> > > > > Juju at lists.ubuntu.com
> > > > > Modify settings or unsubscribe at:
> > > > > https://lists.ubuntu.com/mailman/listinfo/juju
> > > > > 
> > > > > 
> > 
> > 
> 

-- 
Daniel Bidwell <drbidwell at gmail.com>