[Bug 1850779] Re: [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart

[Robie Basak]

Hello LIU, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu7 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
bionic to verification-done-bionic. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-bionic. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: neutron (Ubuntu Bionic)
       Status: Triaged => Fix Committed

** Tags added: verification-needed verification-needed-bionic

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1850779

Title:
  [L3] snat-ns will be initialized twice for DVR+HA routers during agent
  restart

Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Committed
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released
Status in neutron source package in Hirsute:
  Fix Released

Bug description:
  If the  DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart.
  And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent.

  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  SRU:

  [Impact]
  Longer l3-agent initialization time during restarts due to creation of snat namespace and setting corresponding sysctl twice.
  With this fix, the initialization phase is triggered only once.

  [Test Case]
  * deploy Openstack on bionic queens (with neutron dvr l3 ha settings and debug mode on for neutron ) and create a router
    (If stsstack-bundles are used, here are the commands
     ./generate-bundle.sh -s bionic -n bionicqueens --dvr-snat-l3ha --create-model --run
     ./configure
     # Configure creates a router with external gateway attached
    )
  * Restart neutron-l3-agent on one of the node
    systemctl restart neutron-l3-agent.service

  * Check /var/log/neutron/neutron-l3-agent.log and wait for the logs to be settled with all initialization steps
    During initialization steps, following sysctl's are configured [1] [2].
    Verify if the debug logs show sysctl execution statements are displayed twice after restart for snat namespace.
    (If the fix is applied they should be displayed only once)

    grep -inr snat-<router-id> /var/log/neutron/neutron-l3-agent.log |
  grep sysctl

    Example log:
    2718:2021-04-14 05:17:20.114 10868 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'snat-f64dded1-ef73-47b4-bcee-bb25840e9a02', 'sysctl', '-w', 'net.ipv4.ip_forward=1'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:87

  [Where problems could occur]
  no regression is expected, but if one occurs it would likely result in longer init time and/or failure to correctly init the snat-namespace

  [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1850779/+subscriptions