[Bug 1471022] Re: [SRU] race between nova-compute and neutron-ovs-cleanup

[Edward Hope-Morley]

So it does appear that this a result of the openvswitch-switch service
taking a while to startup on boot/reboot. I have added some retry logic
to nova-compute and it appears that openvswitch is taking around 80s to
start:

neutron-ovs-cleanup start/pre-start, process 3055
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 1s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 3s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 3s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 5s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 5s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 7s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 7s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 9s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 9s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 11s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 11s
neutron-ovs-cleanup start/pre-start, process 3055
Attempting to start neutron-ovs-cleanup
start: Job is already running: neutron-ovs-cleanup
Recheck neutron-ovs-cleanup status in 13s
neutron-ovs-cleanup start/running
2015-07-03 17:51:40.660 15690 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.7/dist-packages/nova/servicegroup/api.py:65
2015-07-03 17:51:43.053 15690 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-07-03 17:51:44.320 15690 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.7/dist-packages/stevedore/extension.py:156
2015-07-03 17:51:44.602 15690 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.7/dist-packages/stevedore/extension.py:156
2015-07-03 17:51:44.603 15690 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'

I'll have an SRU patch up shortlty

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1471022

Title:
  [SRU] race between nova-compute and neutron-ovs-cleanup

Status in nova package in Ubuntu:
  In Progress
Status in nova source package in Trusty:
  New
Status in nova source package in Utopic:
  New
Status in nova source package in Vivid:
  New

Bug description:
  This issue appears to be a consequence of
  https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we
  added a 'wait-for-state running' to the nova-compute upstart so as to
  ensure that neutron-ovs-cleanup has finished before nova-compute
  starts.

  I have started to spot, however, that on some hosts (metal only) there
  is now a race between the two whereby nova-compute sometimes fails to
  start on system boot/reboot with the following in /var/log/upstart
  /nova-compute.log:

  ...
  libvirt-bin stop/waiting
  wait-for-state stop/waiting
  neutron-ovs-cleanup start/pre-start, process 3084
  start: Job failed to start

  If I manually restart nova-compute all is fine. So this looks like a
  race between nova-compute's wait-for-state and neutron-ovs-cleanup's
  pre-start -> start/running.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1471022/+subscriptions