[Bug 1850779] Re: [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart

[Arjun Baindur]

I think this fix causes problems. We have multiple nodes that are
DVR_SNAT mode. Snat namespace is scheduled to 1 of them.

When l3-agent is restarted on the othre nodes, now, initialize() is
invoked always for DvrEdgeRouter which creates the SNAT namespace
prematurely. This in turn causes external_gateway_added() to later
detect that this host is NOT hosting snat router, but the namespace
exists, so it removes it by triggerring
external_gateway_removed(dvr_edge_router --> dvr_local_router)

Problem is that the dvr_local_router code for external_gateway_removed()
ends up DELETING the rfp/fpr pair and severs the qrouter connection to
fip namespace (and deletes all the FIP routes in fip namespace as a
result).


Prior to this bug fix, _create_snat_namespace for DvrEdgeRouter was only invoked in _create_dvr_gateway(), which was only invoked when the node was actually hosting SNAT for the router.


Even without the breaking issue of deleting the rtr_2_fip link, this fix uneccesarily creates SNAT namespace on every host, only for it to be deleted.

FYI this is for non-HA routers

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1850779

Title:
  [L3] snat-ns will be initialized twice for DVR+HA routers during agent
  restart

Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Triaged
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released
Status in neutron source package in Hirsute:
  Fix Released

Bug description:
  If the  DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart.
  And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent.

  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  SRU:

  [Impact]
  Longer l3-agent initialization time during restarts due to creation of snat namespace and setting corresponding sysctl twice.
  With this fix, the initialization phase is triggered only once.

  [Test Case]
  * deploy Openstack on bionic queens (with neutron dvr l3 ha settings and debug mode on for neutron ) and create a router
    (If stsstack-bundles are used, here are the commands
     ./generate-bundle.sh -s bionic -n bionicqueens --dvr-snat-l3ha --create-model --run
     ./configure
     # Configure creates a router with external gateway attached
    )
  * Restart neutron-l3-agent on one of the node
    systemctl restart neutron-l3-agent.service

  * Check /var/log/neutron/neutron-l3-agent.log and wait for the logs to be settled with all initialization steps
    During initialization steps, following sysctl's are configured [1] [2]. 
    Verify if the debug logs show sysctl execution statements are displayed twice after restart for snat namespace.
    (If the fix is applied they should be displayed only once)
     
    grep -inr snat-<router-id> /var/log/neutron/neutron-l3-agent.log | grep sysctl

    Example log:
    2718:2021-04-14 05:17:20.114 10868 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'snat-f64dded1-ef73-47b4-bcee-bb25840e9a02', 'sysctl', '-w', 'net.ipv4.ip_forward=1'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:87

  [Where problems could occur]
  This patch reduces the l3 agent initialization time during restart and is not expected to introduce new regressions.

  [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1850779/+subscriptions