[Bug 1630891] Re: unable to start lxd container instances after host reboot
James Page
james.page at ubuntu.com
Wed Nov 9 13:09:10 UTC 2016
from stable/newton branch:
commit c3ed3574a5c0aa59c9e68e6ac4ab2bff1fce3ca0
Author: Chuck Short <chuck.short at canonical.com>
Date: Thu Oct 6 09:08:37 2016 -0400
Reset vifs after reboot
When a host is rebooted instances are set into a shutdown instance
state, which is the correct thing.
However when a user tries to restart an instance, LXD will fail
to restart the container because the bridge is not present.
This is due to the fact that when the instance status transition
from shutdown to active the compute daemon tries to run the
plug_vifs method, which was not present in the driver.
To correct this we re-add the the plug_vifs and unplug_vifs
method.
LP: #1630891
Change-Id: Iae80f4667debab4d1e21d67d882ed52ef59cf15b
Signed-off-by: Chuck Short <chuck.short at canonical.com>
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1630891
Title:
unable to start lxd container instances after host reboot
Status in Ubuntu Cloud Archive:
New
Status in nova-lxd:
Fix Released
Status in nova-lxd package in Ubuntu:
Confirmed
Bug description:
After a compute hypervisor is rebooted, instances revert to SHUTDOWN
state (which is the correct default); however, issuing a nova start
<uuid> does not start the instance up:
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 75, in wrapped
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 66, in wrapped
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 188, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 157, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 613, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 216, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 204, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2475, in start_instance
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self._power_on(context, instance)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2445, in _power_on
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server block_device_info)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 747, in power_on
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server container.start(wait=True)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/container.py", line 159, in start
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server wait=wait)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/container.py", line 144, in _set_state
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.client, response.json()['operation'])
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/operation.py", line 29, in wait_for_operation
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server operation.wait()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/operation.py", line 51, in wait
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server raise exceptions.LXDAPIException(response)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server LXDAPIException: Missing parent 'qbrd853dc7e-2c' for nic 'qbrd853dc7e-2c'
It looks like the neutron network setup is not being re-created prior
to creating the container - this must be done by the compute driver
before attempting to start the instance:
root at juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether fa:16:3e:9d:8d:e1 brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether fa:8d:01:a5:24:0a brd ff:ff:ff:ff:ff:ff
4: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether 36:1b:99:1f:fd:4f brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether 26:e8:f0:7c:58:43 brd ff:ff:ff:ff:ff:ff
6: gre0 at NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/gre 0.0.0.0 brd 0.0.0.0
7: gretap0 at NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
8: gre_sys at NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 86:a0:50:4a:1b:b0 brd ff:ff:ff:ff:ff:ff
9: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether ae:20:44:00:f9:4b brd ff:ff:ff:ff:ff:ff
10: br-data: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether ee:3d:ef:52:b3:45 brd ff:ff:ff:ff:ff:ff
11: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 62:46:66:1a:af:04 brd ff:ff:ff:ff:ff:ff
root at juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10:~# sudo ovs-vsctl show
86ce50a2-d54f-4d1a-92f2-655ff728d897
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "gre-0a051413"
Interface "gre-0a051413"
type: gre
options: {df_default="true", in_key=flow, local_ip="10.5.20.25", out_key=flow, remote_ip="10.5.20.19"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-0a051412"
Interface "gre-0a051412"
type: gre
options: {df_default="true", in_key=flow, local_ip="10.5.20.25", out_key=flow, remote_ip="10.5.20.18"}
Port br-tun
Interface br-tun
type: internal
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Bridge br-data
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-data
Interface br-data
type: internal
Port phy-br-data
Interface phy-br-data
type: patch
options: {peer=int-br-data}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port int-br-data
Interface int-br-data
type: patch
options: {peer=phy-br-data}
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
ovs_version: "2.6.0"
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nova-compute-lxd 14.0.0~rc1-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.4.0-38.57-generic 4.4.19
Uname: Linux 4.4.0-38-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
CrashDB:
{
"impl": "launchpad",
"project": "cloud-archive",
"bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
}
Date: Thu Oct 6 08:26:12 2016
Ec2AMI: ami-0000044a
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.medium
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
TERM=screen-256color-bce
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: nova-lxd
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1630891/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list