[Bug 1573073] Re: When router has no ports _process_updated_router fails because the namespace does not exist

Corey Bryant corey.bryant at canonical.com
Thu Nov 10 21:22:44 UTC 2016 

Hi Savario,  Thanks again for getting your patch upstream.  The upstream
patches have landed for master, newton, and mitaka, so I've picked those
patches into our packages and uploaded to zesty, yakkety, and xenial.
yakkety and xenial are awaiting SRU team review before they'll be
accepted into the -proposed pockets for testing.  Once those get
accepted we can get those backported to corresponding cloud archives,
and start the backport to the liberty cloud archive.

** Summary changed:

- When router has no ports _process_updated_router fails because the namespace does not exist
+ [SRU] When router has no ports _process_updated_router fails because the namespace does not exist

** Description changed:

+ [Description]
+ [Testcase]
  Happens in Kilo. Cannot test on other releases.
  
  Steps to reproduce:
  
  1) create a router and set at least a port, also the gateway is fine
  2) check that the namespace exists with
-    ip netns show | grep qrouter-<uuid>
+    ip netns show | grep qrouter-<uuid>
  3) check the ports are there
-    ip netns exec qrouter-<uuid> ip addr show
+    ip netns exec qrouter-<uuid> ip addr show
  4) delete all ports from the router
  5) check that only loopback interface is present
-    ip netns exec qrouter-<uuid> ip addr show
+    ip netns exec qrouter-<uuid> ip addr show
  6) run the cronjob task that is installed in the file
-    /etc/cron.d/neutron-l3-agent-netns-cleanup
-     so basically run this command:
-    /usr/bin/neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini
+    /etc/cron.d/neutron-l3-agent-netns-cleanup
+     so basically run this command:
+    /usr/bin/neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini
  7) the namespace should be gone:
-    ip netns show | grep qrouter-<uuid>
+    ip netns show | grep qrouter-<uuid>
  8) delete the neutron router.
  9) check log file /var/log/neutron/vpn-agent.log
  
- 
- When the router has no ports the namespace is deleted from the network node by the cronjob. However this brakes the router updates and the file vpn-agent.log is flooded with this traces:
+ When the router has no ports the namespace is deleted from the network
+ node by the cronjob. However this brakes the router updates and the file
+ vpn-agent.log is flooded with this traces:
  
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     return func(*args, **kwargs)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     self._process_internal_ports()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     raise RuntimeError(m)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info RuntimeError:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Exit code: 1
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdin:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdout:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.774 23382 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '8fc0f640-35bb-4d0b-bbbd-80c22be0e762'
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 467, in _process_router_update
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 414, in _process_router_if_compatible
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_updated_router(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 428, in _process_updated_router
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ri.process(self)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 346, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self.logger(e)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     return func(*args, **kwargs)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_internal_ports()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     raise RuntimeError(m)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent RuntimeError:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Exit code: 1
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdin:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdout:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
+ 
+ [Regression Potential]
+ The patches that we're picking up in our packages have already landed upstream in master (zesty), stable/newton (yakkety), and stable/mitaka (xenial).

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1573073

Title:
  [SRU] When router has no ports _process_updated_router fails because
  the namespace does not exist

Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive liberty series:
  Triaged
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in Ubuntu Cloud Archive newton series:
  Triaged
Status in neutron:
  In Progress
Status in neutron package in Ubuntu:
  Triaged
Status in neutron source package in Xenial:
  Triaged
Status in neutron source package in Yakkety:
  Triaged
Status in neutron source package in Zesty:
  Triaged

Bug description:
  [Description]
  [Testcase]
  Happens in Kilo. Cannot test on other releases.

  Steps to reproduce:

  1) create a router and set at least a port, also the gateway is fine
  2) check that the namespace exists with
     ip netns show | grep qrouter-<uuid>
  3) check the ports are there
     ip netns exec qrouter-<uuid> ip addr show
  4) delete all ports from the router
  5) check that only loopback interface is present
     ip netns exec qrouter-<uuid> ip addr show
  6) run the cronjob task that is installed in the file
     /etc/cron.d/neutron-l3-agent-netns-cleanup
      so basically run this command:
     /usr/bin/neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini
  7) the namespace should be gone:
     ip netns show | grep qrouter-<uuid>
  8) delete the neutron router.
  9) check log file /var/log/neutron/vpn-agent.log

  When the router has no ports the namespace is deleted from the network
  node by the cronjob. However this brakes the router updates and the
  file vpn-agent.log is flooded with this traces:

  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     return func(*args, **kwargs)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     self._process_internal_ports()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     raise RuntimeError(m)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info RuntimeError:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Exit code: 1
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdin:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdout:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.774 23382 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '8fc0f640-35bb-4d0b-bbbd-80c22be0e762'
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 467, in _process_router_update
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 414, in _process_router_if_compatible
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_updated_router(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 428, in _process_updated_router
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ri.process(self)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 346, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self.logger(e)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     return func(*args, **kwargs)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_internal_ports()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     raise RuntimeError(m)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent RuntimeError:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Exit code: 1
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdin:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdout:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent

  [Regression Potential]
  The patches that we're picking up in our packages have already landed upstream in master (zesty), stable/newton (yakkety), and stable/mitaka (xenial).

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1573073/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list