Stalled deployment while using --edit-placement

Thu Aug 13 08:37:18 UTC 2015

Hi Jeff, I'll take another look at this in the morning, but I'll try to be
useful now too:

At first glance it looks like keystone wasn't assigned, and it somehow let
you go ahead and deploy despite that.
Or keystone was assigned but that assignment got lost somehow. I'll need to
look into that further in the light of day.

I'm not too surprised that the experience of adding a unit of keystone
after the deploys were done was buggy - it's expected that keystone is
always deployed initially so I bet the add unit UI makes some assumptions
that didn't hold there.

It also looks like mysql is not deployed either, despite having been
assigned to an LXC on
 /MAAS/api/1.0/nodes/node-3603315a-3498-11e5-9aef-a0481ca53e55/.

Those two services are the first ones that should be deployed, because most
of the others depend on them.

After thinking about it, I think you've found a problem with the code that
attempts to re-use the placements.yaml file.
If you do openstack-install -u to uninstall, it will leave the config
directory around, with the idea that you may want to save the log files or
reuse config for a follow-on attempt.

If the placements.yaml is from a fully completed install, a follow-on
install will reuse it cleanly.
However, if the placements.yaml comes from a partially completed placement,
it will probably get confused and do things like think that mysql is
already deployed.

I suggest trying again with openstack-install -u, then moving or deleting
~/.cloud-install/ to avoid any stale config/placement data from the
previous install.

Thanks for your patience!
-mike

On Wed, Aug 12, 2015 at 10:31 PM, Jeff McLamb <mclamb at gmail.com> wrote:

> OK so I started over, this time making sure only to assign services to
> LXC containers or bare metal nodes that were in the “Ready” state.
> There are 4 of these and they are physical servers.
>
> This time everything looked good at first. All machines and services
> came up, I got a green check mark next to everything, and then at the
> bottom it showed the login IP and credentials for both Horizon and the
> Juju GUI.
>
> I tried to login to Horizon and got an error occurred authenticating.
> Please try again later.
>
> I looked back at the openstack-installer results and realized that
> Keystone had never been deployed! It was listed as chosen when I went
> to add services. I had remembered to put it in an LXC container like
> the rest. And there were no more required services to deploy.
>
> So I re-added it in the (A)dd Services section, and it was already
> selected to be deployed in a container to the proper host. I clicked
> Deploy and it did add Keystone into the list of services, now the only
> one pending.
>
> However, it chose to deploy it to an LXC container on “Machine 5”. It
> failed because there were no more new machines to deploy. I had
> specifically chosen it to go in an LXC container on a different host
> though.
>
> Here is commands.log:
>
> http://paste.ubuntu.com/12068748/
>
> And here is placements.yaml:
>
> http://paste.ubuntu.com/12068766/
>
> Any ideas?
>
> Getting really close now…
>
> Thanks,
>
> Jeff
>
>
> On Wed, Aug 12, 2015 at 4:35 PM, Mike McCracken
> <mike.mccracken at canonical.com> wrote:
> > ok, yeah the juju status is telling us what happened, the installer
> tried to
> > add the machine with the maas ID
> node-ac253c98-3547-11e5-921d-a0481ca53e55,
> > which I'm guessing is double-juice, to juju. It uses the maas ID as a
> tag to
> > identify it.
> > Juju couldn't add the machine, most likely because it wasn't available
> > because it was already deployed to be used as the juju bootstrap node.
> >
> > The solution is not to try to deploy to that node, but really the
> installer
> > shouldn't have shown it as available.
> > I'll look into that.
> >
> > Unfortunately there isn't a way to just tell it to skip the current
> charm's
> > deployment. In our experience, there are relatively few errors that can
> be
> > safely ignored in this process and still end up with a working cloud, so
> we
> > generally try to just surface errors as clearly as possible and stop.
> I'll
> > also look into how to make this a more clear error, since it's definitely
> > fatal and shouldn't just hang.
> >
> > I recommend just trying again either without juju gui or placing it
> > differently. Stop the deployment and use openstack-install -u to clear
> out
> > the juju environment and release the machines from maas.
> >
> > -mike
> >
> > On Wed, Aug 12, 2015 at 1:20 PM, Jeff McLamb <mclamb at gmail.com> wrote:
> >>
> >> At this point it would just be nice to kill the stalled attempt at
> >> deploying Juju GUI and go on to the next deployment. Is there any way
> >> to do that? The (A)dd Services menu does not seem to respond to
> >> removals.
> >>
> >> Jeff
> >>
> >> On Wed, Aug 12, 2015 at 4:20 PM, Jeff McLamb <mclamb at gmail.com> wrote:
> >> > Hey Mike -
> >> >
> >> > Here is the whole thing:
> >> >
> >> > http://paste.ubuntu.com/12065381/
> >> >
> >> > Here is juju status:
> >> >
> >> > http://paste.ubuntu.com/12065399/
> >> >
> >> > Thanks!
> >> >
> >> > Jeff
> >> >
> >> > On Wed, Aug 12, 2015 at 2:59 PM, Mike McCracken
> >> > <mike.mccracken at canonical.com> wrote:
> >> >> Hi Jeff, that error from the log is telling you that it tried to
> find a
> >> >> machine in the juju environment that matched the MAAS ID for the
> >> >> machine you
> >> >> chose for juju-gui, and it couldn't.
> >> >> The installer should take care of adding machines to juju that you
> >> >> choose in
> >> >> the placement UI, so this should only happen if there was an error in
> >> >> that
> >> >> process, which will be slightly earlier on in the log.
> >> >>
> >> >> If you'd rather not share the whole log, grep for a log line with
> >> >> "calling
> >> >> add_machines with params", and see if there are errors after that.
> But
> >> >> if I
> >> >> could see the whole log, it might be faster. You can mail it to me
> off
> >> >> list
> >> >> if you like.
> >> >>
> >> >> The output of 'juju status' would be useful here too, to see what it
> >> >> knows
> >> >> about double-juice
> >> >>
> >> >> Thanks,
> >> >> -mike
> >> >>
> >> >> On Wed, Aug 12, 2015 at 10:33 AM, Jeff McLamb <mclamb at gmail.com>
> wrote:
> >> >>>
> >> >>> Hey Mike -
> >> >>>
> >> >>> My juju deployment host is trusty and it is commissioning trusty
> >> >>> nodes. I am using openstack kilo via the experimental ppa. Juju is
> >> >>> version 1.24.5.
> >> >>>
> >> >>> The installation line is:
> >> >>>
> >> >>> $ sudo openstack-install --edit-placement --http-proxy <PROXY>
> >> >>> --https-proxy <PROXY> --debug
> >> >>>
> >> >>> Here is my placements.yaml:
> >> >>>
> >> >>> http://paste.ubuntu.com/12063466/
> >> >>>
> >> >>> Here is the relevant portion of commands.log, indicating that the
> node
> >> >>> I selected for juju gui is unsuitable I guess? Like I said, I guess
> I
> >> >>> can understand this, because the machine double-juice (which is a
> KVM
> >> >>> instance that I manually deployed via MAAS) is already deployed (it
> is
> >> >>> the juju deployment machine) and not in a Ready state like all the
> >> >>> other machines:
> >> >>>
> >> >>> http://paste.ubuntu.com/12063514/
> >> >>>
> >> >>> Thanks,
> >> >>>
> >> >>> Jeff
> >> >>>
> >> >>>
> >> >>> On Wed, Aug 12, 2015 at 1:12 PM, Mike McCracken
> >> >>> <mike.mccracken at canonical.com> wrote:
> >> >>> > Hi Jeff, I'll try to answer inline.
> >> >>> >
> >> >>> > In general, it's helpful to know which ubuntu release you're
> working
> >> >>> > with
> >> >>> > and what openstack release you're trying to install, as the
> >> >>> > installer
> >> >>> > does
> >> >>> > try to support a few combinations.
> >> >>> > If you don't specify a release, the current default is kilo.
> >> >>> >
> >> >>> > Also, are you using the installer from the 'testing' PPA or the
> >> >>> > newer
> >> >>> > 'experimental' one?
> >> >>> > If you are using the testing PPA, then switching to the newer one
> >> >>> > may
> >> >>> > fix
> >> >>> > your bug:
> >> >>> >
> >> >>> > $ sudo apt-add-repository ppa:cloud-installer/experimental
> >> >>> > $ sudo apt-get update
> >> >>> > $ sudo apt-get install openstack
> >> >>> >
> >> >>> >
> >> >>> > On Wed, Aug 12, 2015 at 9:37 AM, Jeff McLamb <mclamb at gmail.com>
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> Hi everyone,
> >> >>> >>
> >> >>> >> I am playing around with openstack-installer using
> --edit-placement
> >> >>> >> to
> >> >>> >> manually assign services to my 4 physical nodes.
> >> >>> >>
> >> >>> >> Due to having only 4 nodes, I have to use several servers as
> >> >>> >> multi-use. My first question is, are there any functional
> >> >>> >> limitations
> >> >>> >> on using nodes with multiple services? Will openstack-installer
> >> >>> >> detect
> >> >>> >> this if there are? I realize that for performance reasons it
> might
> >> >>> >> not
> >> >>> >> make sense, but right now I just want to get a working setup with
> >> >>> >> what
> >> >>> >> I have.
> >> >>> >
> >> >>> >
> >> >>> > There are some limitations, and the installer does try not to let
> >> >>> > you do
> >> >>> > some things that will break, but it can't guarantee a working
> setup.
> >> >>> > In
> >> >>> > particular, nova-compute and quantum/neutron need to be deployed
> >> >>> > onto
> >> >>> > bare
> >> >>> > metal or KVM, not LXC, IIRC because of how they interact with the
> >> >>> > host
> >> >>> > kernel.
> >> >>> >
> >> >>> >>
> >> >>> >> I am trying to deploy 2 bare metal compute nodes, 2 bare metal
> Ceph
> >> >>> >> nodes (shared across compute), 1 bare metal Ceph OSD (shared on
> one
> >> >>> >> of
> >> >>> >> the compute nodes). Most of the required controller-like services
> >> >>> >> are
> >> >>> >> in LXC containers on a single node. Neutron services are in LXC
> >> >>> >> containers on another node.
> >> >>> >
> >> >>> >
> >> >>> > The installer shouldn't allow placing neutron in a container. Can
> >> >>> > you
> >> >>> > send
> >> >>> > along the contents of ~/.cloud-install/placements.yaml so we can
> see
> >> >>> > how
> >> >>> > the
> >> >>> > services are laid out?
> >> >>> >
> >> >>> > Also, if you could include the log at
> ~/.cloud-install/commands.log,
> >> >>> > we
> >> >>> > can
> >> >>> > look there for errors explaining why it's hanging at juju gui.
> >> >>> >
> >> >>> > In general if you don't want the juju gui, you can skip it, either
> >> >>> > just
> >> >>> > not
> >> >>> > placing it or removing it from the default placement.
> >> >>> > It's not necessary for the operation of any other services.
> >> >>> >
> >> >>> > Thanks,
> >> >>> > -mike
> >> >>> >
> >> >>> >>
> >> >>> >> I also chose to install Juju-GUI in a container on the Juju
> >> >>> >> deployment
> >> >>> >> node, which itself is a manually deployed KVM node hosted on the
> >> >>> >> MAAS
> >> >>> >> server, managed by MAAS.
> >> >>> >>
> >> >>> >> Using the latest installer, deployment has stalled during Juju
> GUI.
> >> >>> >> It
> >> >>> >> successfully deployed Keystone, MySQL, NTP to containers, but has
> >> >>> >> been
> >> >>> >> sitting at Juju GUI. I am suspicious that this is because the
> >> >>> >> chosen
> >> >>> >> node is one that Juju is not deploying itself?
> >> >>> >>
> >> >>> >> Is there any way to (A)dd Services and remove Juju GUI during
> this
> >> >>> >> stage? I tried to do that, but Remove does not seem to register
> >> >>> >> anything. And juju-gui does not yet appear in `juju status` so I
> >> >>> >> can't
> >> >>> >> remove it that way either.
> >> >>> >>
> >> >>> >> Basically, I'd like to move along with the deployment and forgo
> >> >>> >> Juju
> >> >>> >> GUI for now, but I can't figure out how to tell it to skip over
> >> >>> >> this
> >> >>> >> part of the deployment.
> >> >>> >>
> >> >>> >> Thanks,
> >> >>> >>
> >> >>> >> Jeff
> >> >>> >>
> >> >>> >> --
> >> >>> >> ubuntu-openstack-installer mailing list
> >> >>> >> ubuntu-openstack-installer at lists.ubuntu.com
> >> >>> >> Modify settings or unsubscribe at:
> >> >>> >>
> >> >>> >>
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-openstack-installer
> >> >>> >
> >> >>> >
> >> >>
> >> >>
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-openstack-installer/attachments/20150813/afd8a28b/attachment-0001.html>