[Bug 2074504] Please test proposed package

[Timo Aaltonen]

Hello Goutham, or anyone else affected,

Accepted manila into jammy-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/manila/1:14.1.1-0ubuntu1.2 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
jammy to verification-done-jammy. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-jammy. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to manila in Ubuntu.
https://bugs.launchpad.net/bugs/2074504

Title:
  [SRU] Manila's NeutronNetworkPlugin with external networks doesn't
  work with OVN

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive antelope series:
  Won't Fix
Status in Ubuntu Cloud Archive bobcat series:
  Won't Fix
Status in Ubuntu Cloud Archive caracal series:
  In Progress
Status in Ubuntu Cloud Archive dalmatian series:
  Fix Released
Status in Ubuntu Cloud Archive epoxy series:
  Fix Released
Status in Ubuntu Cloud Archive yoga series:
  In Progress
Status in Ubuntu Cloud Archive zed series:
  Won't Fix
Status in OpenStack Shared File Systems Service (Manila):
  Fix Released
Status in manila package in Ubuntu:
  Fix Released
Status in manila source package in Jammy:
  Fix Committed
Status in manila source package in Noble:
  Fix Committed
Status in manila source package in Oracular:
  Fix Released

Bug description:
  *********** SRU TEMPLATE AT THE BOTTOM ***********

  Description
  ===========

  When using NeutronNetworkPlugin with DHSS=True, manila requests
  neutron ports for creating network connections on share servers on the
  user provided Share Network.

  Deployers and users have the flexibility to use external networks
  (i.e., a "provider networks" in neutron parlance) as their Share
  Networks. When they do this, they expect to use Neutron to merely
  perform IPAM. Neutron does create ports to reserve IP addresses;
  however, we don't expect these ports to work or respond to ARP
  requests. This worked even when OVN was used as the ML2 plugin in the
  deployment; however, OVN had a change in its default behavior [1].
  This change makes OVN setup flows for DOWN ports; when ARP responses
  are received from OVN ports, traffic is effectively misrouted/dropped.
  This means that end users cannot reach their share export paths from
  their eventual VMs/containers/bare metal hosts. OVN has a
  configuration option to turn this behavior off ("ignore_lsp_down"). By
  default, OpenStack Neutron sets this "ignore_lsp_down" option to False
  [2] - meaning OVN is not supposed to setup flow table entries for any
  ports that are DOWN.

  However, this behavior isn't working as one would expect.

  Steps to reproduce
  ==================

  A chronological list of steps which will help reproduce the issue you hit:
  * Create a provider network on OpenStack
  * Configure manila with a DHSS=True driver that can use an "external" storage system (example, NetApp)
  * Create a share network mapped to the provider network
  * Create a share with the share network
  * Create a tenant VM
  * Create appropriate access rule/s in manila
  * Attempt to mount the share in the VM

  Expected result
  ===============
  Share is reachable/mountable

  Actual result
  =============
  Share failed to be mounted. Cannot ping the export IPs either because the provider network is unreachable. When you debug this further, you'll notice packets are dropped, citing a MAC address mismatch.

  Environment
  ===========
  1. Version of OpenStack Manila: OpenStack Wallaby

  2. Which storage backend did you use: NetApp (although this should be
  a problem with any non-generic DHSS=true backend)

  3. Which networking type did you use? OVN

  [1] https://www.mail-archive.com/ovs-dev@openvswitch.org/msg60064.html
  [2] https://review.opendev.org/c/openstack/neutron/+/896545

  ===============
  SRU DESCRIPTION
  ===============

  [Impact]

  This issue blocks connectivity of users to external storage backends
  when using OVN, therefore the users cannot access their shares.

  [Test case]

  We cannot reproduce the issue in Canonical lab as we don't have any
  external storage with DHSS=True mode. I tried to reproduce it with the
  generic driver in DHSS=True mode but couldn't. The included unit tests
  should provide coverage. Additionally we have already provided test
  PPAs to customers affected with the issue and they confirmed the issue
  was addressed. Some upstream users have also validated the fix.

  UPDATE: After discussing with Heitor, we agreed that we would do a
  smoke test using the generic driver in DHSS=True mode because it goes
  through the same code that is modified by the fix. The difference of
  why the generic driver is not affected but other drivers are is
  because the issue only manifests with real hardware that have physical
  ports, while the generic driver only has virtual ports, so the OVN
  issue doesn't happen, despite the code used to set things up being the
  same. Therefore, a smoke test creating a share and accessing it in the
  generic driver using DHSS=True will go through the modified code and
  confirm that the code did not break with the update.

  Steps:

  1) Deploy manila with generic driver in DHSS=True mode

  2) create a share network, a share, add access rules, and mount it
  successfully, write a dummy file in it

  3) Install updated package

  4) create another share network, another share, add access rules,
  mount it successfully, write a dummy file in it. Because it is a
  different share network, no IPs or ports from step (2) should be re-
  used, but it is best to confirm just to be sure that the resources are
  different.

  
  [Where problems could occur]

  The code only affects the neutron plugin for manila, which is used
  only for DHSS=True mode. However, the code is shared between both OVN
  and OVS modes, so attempting to fix the issue for OVN could
  potentially break all the current users using OVS. The breakage
  wouldn't be immediate upon installing the package, as the code is
  executed only when new share servers are created. So as long as only
  shares are being created while the existing share servers and networks
  were already functional prior to the upgrade, the damage can be
  mitigated. On the other hand, OVN users are broken without
  connectivity so the benefits generally outweight the risks here. In
  case of regression, the previous funcionality can be restored through
  package downgrade.

  [Other Info]

  For Jammy/Yoga, bugfix LP#2049507 will also be included.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2074504/+subscriptions