[Bug 1420572] Re: race between neutron-ovs-cleanup and nova-compute

[Edward Hope-Morley]

** Description changed:

+ [Impact]
+ 
+  * We run neutron-ovs-cleanup in startup if neutron installed. If nova-compute
+    does not wait for completion it will try to use veth/bridge devices that may
+    may be in the process of bring deleted.
+ 
+ [Test Case]
+ 
+  * Create neutron (ovs) network and boot an instance with this network
+ as --nic
+ 
+  * Check that creation was successful and network is functional. Also make a
+    note corresponding veth and bridge devices (ip a).
+ 
+  * Reboot system, check that expected veth and bridge devices are still
+    there and that nova-compute is happy e.g. try sshing to your instance. Also
+    check /var/log/upstart/nova-compute.log to see if service waited for
+    ovs-cleanup to finish.
+ 
+ [Regression Potential]
+ 
+  * None
+ 
+ 
+ ---- ---- ---- ----
+ 
  There is a race when both neutron-ovs-cleanup and nova-compute trying to
  do operations on the qvb*** and qvo*** devices. Below is a scenario I
  recently met,
  
  1. nova-compute was started and creating the veth_pair for VM instances
  running on the host -
  https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1298
  
  2. neutron-ovs-cleanup was kicked off and deleted all the ports.
  
  3. when nova-compute tried to set the MTU at
  https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1280
  , Stderr: u'Cannot find device "qvo***"\n' was reported. Because the
  device that was just created was deleted again by neutron-ovs-cleanup.
  
  As they both operate on the same resources, there needs a way to
  synchronize the operations the two processes do on those resources.

** Description changed:

  [Impact]
  
-  * We run neutron-ovs-cleanup in startup if neutron installed. If nova-compute
-    does not wait for completion it will try to use veth/bridge devices that may
-    may be in the process of bring deleted.
+  * We run neutron-ovs-cleanup in startup if neutron installed. If
+    nova-compute does not wait for completion it will try to use
+    veth/bridge devices that may be in the process of bring deleted.
  
  [Test Case]
  
-  * Create neutron (ovs) network and boot an instance with this network
- as --nic
+  * Create neutron (ovs) network and boot an instance with this network
+    as --nic
  
-  * Check that creation was successful and network is functional. Also make a
-    note corresponding veth and bridge devices (ip a).
+  * Check that creation was successful and network is functional. Also make
+    a note corresponding veth and bridge devices (ip a).
  
-  * Reboot system, check that expected veth and bridge devices are still
-    there and that nova-compute is happy e.g. try sshing to your instance. Also
-    check /var/log/upstart/nova-compute.log to see if service waited for
-    ovs-cleanup to finish.
+  * Reboot system, check that expected veth and bridge devices are still
+    there and that nova-compute is happy e.g. try sshing to your instance.
+    Also check /var/log/upstart/nova-compute.log to see if service waited
+    for ovs-cleanup to finish.
  
  [Regression Potential]
  
-  * None
- 
+  * None
  
  ---- ---- ---- ----
  
  There is a race when both neutron-ovs-cleanup and nova-compute trying to
  do operations on the qvb*** and qvo*** devices. Below is a scenario I
  recently met,
  
  1. nova-compute was started and creating the veth_pair for VM instances
  running on the host -
  https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1298
  
  2. neutron-ovs-cleanup was kicked off and deleted all the ports.
  
  3. when nova-compute tried to set the MTU at
  https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1280
  , Stderr: u'Cannot find device "qvo***"\n' was reported. Because the
  device that was just created was deleted again by neutron-ovs-cleanup.
  
  As they both operate on the same resources, there needs a way to
  synchronize the operations the two processes do on those resources.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1420572

Title:
  race between neutron-ovs-cleanup and nova-compute

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572/+subscriptions