[Bug 1531963] Re: trusty/icehouse neutron-plugin-openvswitch-agent: lvm.tun_ofports.remove crashes with KeyError

[Corey Bryant]

** Changed in: neutron (Ubuntu)
       Status: Triaged => Fix Committed

** Changed in: neutron (Ubuntu)
     Assignee: (unassigned) => Corey Bryant (corey.bryant)

** Changed in: neutron (Ubuntu)
   Importance: Undecided => High

** Summary changed:

- trusty/icehouse neutron-plugin-openvswitch-agent: lvm.tun_ofports.remove crashes with KeyError
+ [SRU] trusty/icehouse neutron-plugin-openvswitch-agent: lvm.tun_ofports.remove crashes with KeyError

** Description changed:

+ [Impact]
+ Neutron OVS breaks with unhandled exceptions on compute nodes.
+ 
+ [Test Case]
+ For reproduction see the original bug description below.
+ 
+ [Regression Potential]
+ 
+ The backported patch is very straightforward, with a few minor conflicts
+ noted in the patch.
+ 
+ -------------
+ 
+ Original bug description:
+ 
  Filing this on ubuntu/neutron package, as neutron itself is EOL'd for
  Icehouse.
  
  FYI this is a nonHA icehouse/trusty deploy using serverteam's juju
  charms.
  
  On one of our production environments with a rather high rate of API
  calls, (sp for  transient VMs from CI), we frequently get neutron OVS
  breakage on compute nodes¹, which we've been able to more or less
  correlate with the following alike errors at /var/log/neutron
  /openvswitch-agent.log:
  
  2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 399, in _del_fdb_flow
  2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp     lvm.tun_ofports.remove(ofport)
  2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp KeyError: '13'
  
  Detailed log: http://paste.ubuntu.com/14431656/  - note the same time of
  occurrence on the 3 diff compute nodes shown there.
  
  ¹ What we then observe are missing are missing tun_ids from
-   ovs-ofctl dump-flows br-tun
+   ovs-ofctl dump-flows br-tun
  ie provider:segmentation_id not present at the compute node for a VM with a neutron network that has it.
  
  Afaics this had been fixed upstream at ( lp#1421105 ):
  https://git.openstack.org/cgit/openstack/neutron/commit/?id=841b2f58f375df53b380cf5796bb31c82cd09260
  , please consider backporting it to Icehouse, it's a pretty trivial fix.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1531963

Title:
  [SRU] trusty/icehouse neutron-plugin-openvswitch-agent:
  lvm.tun_ofports.remove crashes with KeyError

Status in neutron package in Ubuntu:
  Fix Committed

Bug description:
  [Impact]
  Neutron OVS breaks with unhandled exceptions on compute nodes.

  [Test Case]
  For reproduction see the original bug description below.

  [Regression Potential]

  The backported patch is very straightforward, with a few minor
  conflicts noted in the patch.

  -------------

  Original bug description:

  Filing this on ubuntu/neutron package, as neutron itself is EOL'd for
  Icehouse.

  FYI this is a nonHA icehouse/trusty deploy using serverteam's juju
  charms.

  On one of our production environments with a rather high rate of API
  calls, (sp for  transient VMs from CI), we frequently get neutron OVS
  breakage on compute nodes¹, which we've been able to more or less
  correlate with the following alike errors at /var/log/neutron
  /openvswitch-agent.log:

  2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 399, in _del_fdb_flow
  2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp     lvm.tun_ofports.remove(ofport)
  2016-01-07 06:33:48.917 18357 TRACE neutron.openstack.common.rpc.amqp KeyError: '13'

  Detailed log: http://paste.ubuntu.com/14431656/  - note the same time
  of occurrence on the 3 diff compute nodes shown there.

  ¹ What we then observe are missing are missing tun_ids from
    ovs-ofctl dump-flows br-tun
  ie provider:segmentation_id not present at the compute node for a VM with a neutron network that has it.

  Afaics this had been fixed upstream at ( lp#1421105 ):
  https://git.openstack.org/cgit/openstack/neutron/commit/?id=841b2f58f375df53b380cf5796bb31c82cd09260
  , please consider backporting it to Icehouse, it's a pretty trivial fix.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1531963/+subscriptions