Trying to remove and redeploy quantum-gateway on a different server

[Jeff McLamb]

Mark, Mike -

Thanks for all of this. Indeed, it was pretty cool that I could just tear
things down and point it elsewhere. Mike's been very helpful on the
ubuntu-openstack-installer list with the Autopilot, as well as many other
issues. I've decided to go with the manual Juju approach for now because
I'm trying to understand things at a lower level, but once that's done and
I'm comfortable with all the pieces and how they fit together, I'll revisit
the Autopilot.

Thanks,

Jeff

On Sun, Sep 20, 2015 at 6:59 AM, Mark Shuttleworth <mark at ubuntu.com> wrote:

> Hi Jeff
>
> Glad you found the issue! It's pretty cool that you could just reconnect
> the bits with a different mapping to VMs and voilà it works :)
>
> Generally, Juju will provide you with the fastest way to "do what you
> want", and to the extent the charms have seen your situation before, a best
> practice result. In this case we can enhance that charm to error helpfully
> (i.e. with a message) saying "I need two network interfaces, please put me
> on an appropriate VM or real machine".
>
> For those people who really want to understand every aspect of OpenStack,
> this is a great way to do what you want, and experiment quickly, to get the
> large scale cloud you want. We just added detailed support for Contrail 2.2
> for example, which is fun to test as a real SDN at scale.
>
> But we think most people do not need or want to be OpenStack experts; they
> want OpenStack to "Just Work" so they can focus on their apps and workloads.
>
> That's why we also built the Autopilot, which is free for up to 10 nodes
> (thereafter free with out standard support offering) and it will:
>
>  * study the hardware you provide it
>  * offer you a choice of SDNs and storage and other components
>  * create the appropriate reference architecture of containers, metal and
> VMs
>  * monitor the resulting cloud
>
> Once you've deployed, you can feed it more machines to grow, and it will
> evolve the reference architecture accordingly.
>
> If you're going to be building lots of these clouds, or need to focus at a
> higher level, please consider the autopilot.
>
> Mark
>
>
> On 19/09/15 16:48, Jeff McLamb wrote:
>
> Solved! TL;DR - You must deploy quantum-gateway or neutron-gateway to bare
> metal. It will not work in an LXC container.
>
> It appears as though you cannot deploy the {quantum,neutron}-gateway charm
> to an LXC. I eventually tracked down the stuck config-changed state due to
> the openvswitch vswitchd daemon not being able to start because it cannot
> load the kernel module properly, which causes a loop whereby it keeps
> attempting to add flows to the br-int OVS but fails because the daemon is
> not running.
>
> I removed the service from the LXC and deployed it to a bare metal machine
> (co-hosted with Ceph services that don’t seem to do anything fancy with
> bridges or OVS) and everything worked just fine.
>
> Thanks,
>
> Jeff
>
> On Fri, Sep 18, 2015 at 4:29 PM, Jeff McLamb <mclamb at gmail.com> <mclamb at gmail.com> wrote:
>
>
> Thanks for the info, Mike! I will provide a bit more detail as to what’s
> going on in case any Juju experts have seen something similar.
>
> Where I last left off, I had deleted all units of anything
> {neutron,quantum}-gateway related from juju’s perspective. However, the
> original bare metal node was still providing the neutron gateway (i.e.
> network node) services, so I manually issued a stop on all of those (e.g.
> stop neutron-dhcp, stop neutron-metadata-server, stop neutron-metering,
> etc.) The only thing I left running on that server was openvswitch because
> it is also a compute node, so it needs to have the openvswitch plugin
> running, as I understand it.
>
> I then issued a `juju deploy neutron-gateway —to lxc:3` in order to
> install the gateway in an lxc container.
>
> The lxc container is provisioned and everything seems to be going OK until
> it gets stuck in agent status -> message: running config-changed hook
>
> If I ssh into the container itself and look at processes, I can see it is
> in a seemingly infinite loop where the neutron rootwrapper is constantly
> issuing ovs-vsctl add-br br-int calls, along with adding and removing
> flows. It’s as though it keeps trying to add and remove the br-int ovs, and
> just sits there doing that forever.
>
> I manually killed the config-changed hook and then issued a `juju
> debug-hooks neutron-gateway/0 config-changed` followed by `juju resolved
> neutron-gateway/0 —retry`, which has dumped me into a config-changed tmux
> session where I manually issued ./hooks/config-changed.
>
> There is not much other than the following output, after which it hangs:
>
> root at juju-machine-3-lxc-11:/var/lib/juju/agents/unit-neutron-gateway-0/charm#
> ./hooks/config-changed
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> python-dbus is already the newest version.
> The following packages were automatically installed and are no longer
> required:
>   libfreetype6 os-prober
> Use 'apt-get autoremove' to remove them.
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> At this point everything hangs. I have not yet Ctrl-C’d the process yet.
>
> In the /var/log/juju/unit-neutron-gateway-0.log it looks pretty standard,
> finishing all log output with Creating bridge br-int, Creating bridge
> br-ex, Creating bridge br-tun.
>
> At this point there is no more streaming log output either.
>
> Has anyone else seen this or have an idea what might be going on?
>
> Thanks,
>
> Jeff
>
> On Fri, Sep 18, 2015 at 2:22 PM, Mike McCracken <mike.mccracken at canonical.com> wrote:
>
>
> Hi Jeff!
>
> On Fri, Sep 18, 2015 at 11:12 AM, Jeff McLamb <mclamb at gmail.com> <mclamb at gmail.com> wrote:
>
>
> Hi all -
>
> * Sorry for the cross-post on openstack-installer, but I think this is
> more of a juju question than an openstack-installer issue*
>
> As I originally deployed my current test setup via openstack-installer,
> I am wondering how I am supposed to use it and/or juju directly for future
> changes/additions.
>
>
> Yes, the openstack-installer is intended to be a first step and not a
> full purpose juju front-end. For future changes of the sort you're
> describing, you'll want to get familiar with using juju directly. Note that
> the official Juju web GUI is one of the services that you can install via
> the manual placement UI in the installer.
>
> I am trying to understand neutron networking, investigating all of the
>
> OVS and Linux bridges, how traffic flows between them and the network node
> etc. In my initial deployment I co-located a compute node and the network
> node on bare metal via manual placement in openstack-installer.
>
> Now, in order to better see the separation of stuff (i.e. not conflate
> what the compute node is doing with the network node) I decided to `juju
> remove-service quantum-gateway` followed by a `juju add-service
> quantum-gateway —to lxc:3` in order to remove it from the original node and
> install it into its own container on the controller node that hosts lots of
> container-based services.
>
> I have received lots of errors in the process during which various hooks
> get stuck. I have been able to get around it by manually killing hooks and
> then issuing a `juju resolved quantum-gateway` until finally the service
> goes away. Not to mention the fact that a new deployment seems to be in a
> “blocked” state saying I should upgrade to neutron-gateway. I tried that
> with similar results.
>
> At this point I managed to get rid of everything — i.e. I do not have a
> quantum-gateway or a neutron-gateway as reported by juju.
>
> However, I have noticed that the original server (colocated compute and
> quantum-gateway) still has all of the neutron services running. Issuing a
> `neutron agent-list` shows everything connected and happy as it ever was.
> It seems as though my juju fiddling has basically been just manipulating
> the juju database without actually removing the underlying services. I have
> noticed that when removing services on an lxc container it does indeed kill
> the container (so obviously the service too)… however, maybe because this
> original service was deployed on bare metal with another service, it
> doesn’t bother to actually stop the services and remove their /etc/init
> files?
>
>
> This does sound unexpected, hopefully someone more directly involved with
> Juju can comment here.
>
>
>
> I have tried to redeploy via openstack-status, but it seems like it is
> useless for future deployment even though it has that nice curses-looking
> GUI from which I can deploy. It’s understanding of where everything is
> placed is completely broken, and it does not seem to honor any requests to
> place stuff in the first place.
>
>
> I'll follow up with you about the openstack-status tool on the installer
> list, to keep this thread juju-specific.
>
>
>
> Any ideas?
>
> Thanks,
>
> Jeff
>
> --
> Juju mailing listJuju at lists.ubuntu.com
> Modify settings or unsubscribe at:https://lists.ubuntu.com/mailman/listinfo/juju
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-openstack-installer/attachments/20150922/1c38c23b/attachment.html>