[Bug 2074504] Re: [SRU] Manila's NeutronNetworkPlugin with external networks doesn't work with OVN
Tiago Pasqualini da Silva
2074504 at bugs.launchpad.net
Wed Aug 20 15:28:23 UTC 2025
Hi all, sorry for the delay on this. Just performed Yoga SRU
verification:
ubuntu at juju-e0ef3b-sru-15:~$ dpkg -l | grep manila
ii manila-api 1:14.1.1-0ubuntu1.2 all OpenStack shared file system as a service - API server
ii manila-common 1:14.1.1-0ubuntu1.2 all OpenStack shared file system as a service - common files
ii manila-data 1:14.1.1-0ubuntu1.2 all Manila storage service - Data service
ii manila-scheduler 1:14.1.1-0ubuntu1.2 all OpenStack shared file system as a service - Scheduler server
ii manila-share 1:14.1.1-0ubuntu1.2 all OpenStack shared file system as a service - Share server
ii python3-manila 1:14.1.1-0ubuntu1.2 all OpenStack shared file system as a service - Python 3 libs
ubuntu at juju-e0ef3b-sru-15:~$
logout
Connection to 10.149.3.144 closed.
ubuntu at stg-reproducer-tiago-pasqualini-project-bastion:~$ openstack share show s1
+---------------------------------------+----------------------------------------------------------------------+
| Field | Value |
+---------------------------------------+----------------------------------------------------------------------+
| access_rules_status | active |
| availability_zone | nova |
| create_share_from_snapshot_support | False |
| created_at | 2025-08-20T15:09:02.488740 |
| description | None |
| export_locations | |
| | id = f8df74a9-219a-44fd-b81d-add771320431 |
| | path = 10.254.0.4:/shares/share-48bd8ff7-bb94-4060-9e58-ce7a346ac097 |
| | preferred = False |
| | share_instance_id = 48bd8ff7-bb94-4060-9e58-ce7a346ac097 |
| | is_admin_only = False |
| | id = f3a05474-7712-40b4-990c-c24cb6db8bae |
| | path = 10.254.0.4:/shares/share-48bd8ff7-bb94-4060-9e58-ce7a346ac097 |
| | preferred = False |
| | share_instance_id = 48bd8ff7-bb94-4060-9e58-ce7a346ac097 |
| | is_admin_only = True |
| has_replicas | False |
| host | juju-e0ef3b-sru-15 at generic#generic |
| id | ff9139fc-075c-454d-8dac-6c90660b9003 |
| is_public | False |
| is_soft_deleted | False |
| mount_snapshot_support | False |
| name | s1 |
| progress | 100% |
| project_id | 5360f5b04067480781eed751bf4d657d |
| properties | |
| replication_type | None |
| revert_to_snapshot_support | False |
| scheduled_to_be_deleted_at | None |
| share_group_id | None |
| share_network_id | d7024dcf-779d-407f-a2de-65be3092aed6 |
| share_proto | NFS |
| share_server_id | b99d144e-cc94-48e3-9958-1b6a478f6933 |
| share_type | 81ab2a73-e8e0-4a80-84a9-875df31abe5f |
| share_type_name | default |
| size | 1 |
| snapshot_id | None |
| snapshot_support | False |
| source_share_group_snapshot_member_id | None |
| status | available |
| task_state | None |
| user_id | cef7a508f7f644ee9b75bdea889df095 |
| volume_type | default |
+---------------------------------------+----------------------------------------------------------------------+
ubuntu at stg-reproducer-tiago-pasqualini-project-bastion:~$ ssh -i ~/testkey.priv ubuntu at 10.149.3.120
Warning: Permanently added '10.149.3.120' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-143-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Wed Aug 20 15:22:38 UTC 2025
System load: 0.42 Processes: 93
Usage of /: 8.8% of 19.20GB Users logged in: 0
Memory usage: 9% IPv4 address for ens2: 192.168.21.95
Swap usage: 0%
Expanded Security Maintenance for Applications is not enabled.
43 updates can be applied immediately.
38 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
New release '24.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Aug 20 15:22:39 2025 from 10.149.3.152
ubuntu at jammy:~$ cd /media/
ubuntu at jammy:/media$ sudo mount -t nfs 10.254.0.4:/shares/share-48bd8ff7-bb94-4060-9e58-ce7a346ac097 nfs
ubuntu at jammy:/media$ cd nfs
ubuntu at jammy:/media/nfs$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=985648k,nr_inodes=246412,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=201100k,mode=755,inode64)
/dev/vda1 on / type ext4 (rw,relatime,discard,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15162)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/lxd_31333.snap on /snap/lxd/31333 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_24718.snap on /snap/snapd/24718 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_2599.snap on /snap/core20/2599 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/vda15 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=201100k,mode=755,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=201096k,nr_inodes=50274,mode=700,uid=1000,gid=1000,inode64)
10.254.0.4:/shares/share-48bd8ff7-bb94-4060-9e58-ce7a346ac097 on /media/nfs type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.21.95,local_lock=none,addr=10.254.0.4)
ubuntu at jammy:/media/nfs$ dd if=/dev/urandom of=a.iso bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.185325 s, 56.6 MB/s
ubuntu at jammy:/media/nfs$
** Tags removed: verification-needed verification-needed-jammy verification-needed-noble verification-yoga-needed
** Tags added: verification-done verification-done-jammy verification-done-noble verification-yoga-done
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2074504
Title:
[SRU] Manila's NeutronNetworkPlugin with external networks doesn't
work with OVN
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
Won't Fix
Status in Ubuntu Cloud Archive bobcat series:
Won't Fix
Status in Ubuntu Cloud Archive caracal series:
Fix Committed
Status in Ubuntu Cloud Archive dalmatian series:
Fix Released
Status in Ubuntu Cloud Archive epoxy series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
Fix Committed
Status in Ubuntu Cloud Archive zed series:
Won't Fix
Status in OpenStack Shared File Systems Service (Manila):
Fix Released
Status in manila package in Ubuntu:
Fix Released
Status in manila source package in Jammy:
Fix Committed
Status in manila source package in Noble:
Fix Committed
Status in manila source package in Oracular:
Fix Released
Bug description:
*********** SRU TEMPLATE AT THE BOTTOM ***********
Description
===========
When using NeutronNetworkPlugin with DHSS=True, manila requests
neutron ports for creating network connections on share servers on the
user provided Share Network.
Deployers and users have the flexibility to use external networks
(i.e., a "provider networks" in neutron parlance) as their Share
Networks. When they do this, they expect to use Neutron to merely
perform IPAM. Neutron does create ports to reserve IP addresses;
however, we don't expect these ports to work or respond to ARP
requests. This worked even when OVN was used as the ML2 plugin in the
deployment; however, OVN had a change in its default behavior [1].
This change makes OVN setup flows for DOWN ports; when ARP responses
are received from OVN ports, traffic is effectively misrouted/dropped.
This means that end users cannot reach their share export paths from
their eventual VMs/containers/bare metal hosts. OVN has a
configuration option to turn this behavior off ("ignore_lsp_down"). By
default, OpenStack Neutron sets this "ignore_lsp_down" option to False
[2] - meaning OVN is not supposed to setup flow table entries for any
ports that are DOWN.
However, this behavior isn't working as one would expect.
Steps to reproduce
==================
A chronological list of steps which will help reproduce the issue you hit:
* Create a provider network on OpenStack
* Configure manila with a DHSS=True driver that can use an "external" storage system (example, NetApp)
* Create a share network mapped to the provider network
* Create a share with the share network
* Create a tenant VM
* Create appropriate access rule/s in manila
* Attempt to mount the share in the VM
Expected result
===============
Share is reachable/mountable
Actual result
=============
Share failed to be mounted. Cannot ping the export IPs either because the provider network is unreachable. When you debug this further, you'll notice packets are dropped, citing a MAC address mismatch.
Environment
===========
1. Version of OpenStack Manila: OpenStack Wallaby
2. Which storage backend did you use: NetApp (although this should be
a problem with any non-generic DHSS=true backend)
3. Which networking type did you use? OVN
[1] https://www.mail-archive.com/ovs-dev@openvswitch.org/msg60064.html
[2] https://review.opendev.org/c/openstack/neutron/+/896545
===============
SRU DESCRIPTION
===============
[Impact]
This issue blocks connectivity of users to external storage backends
when using OVN, therefore the users cannot access their shares.
[Test case]
We cannot reproduce the issue in Canonical lab as we don't have any
external storage with DHSS=True mode. I tried to reproduce it with the
generic driver in DHSS=True mode but couldn't. The included unit tests
should provide coverage. Additionally we have already provided test
PPAs to customers affected with the issue and they confirmed the issue
was addressed. Some upstream users have also validated the fix.
UPDATE: After discussing with Heitor, we agreed that we would do a
smoke test using the generic driver in DHSS=True mode because it goes
through the same code that is modified by the fix. The difference of
why the generic driver is not affected but other drivers are is
because the issue only manifests with real hardware that have physical
ports, while the generic driver only has virtual ports, so the OVN
issue doesn't happen, despite the code used to set things up being the
same. Therefore, a smoke test creating a share and accessing it in the
generic driver using DHSS=True will go through the modified code and
confirm that the code did not break with the update.
Steps:
1) Deploy manila with generic driver in DHSS=True mode
2) create a share network, a share, add access rules, and mount it
successfully, write a dummy file in it
3) Install updated package
4) create another share network, another share, add access rules,
mount it successfully, write a dummy file in it. Because it is a
different share network, no IPs or ports from step (2) should be re-
used, but it is best to confirm just to be sure that the resources are
different.
[Where problems could occur]
The code only affects the neutron plugin for manila, which is used
only for DHSS=True mode. However, the code is shared between both OVN
and OVS modes, so attempting to fix the issue for OVN could
potentially break all the current users using OVS. The breakage
wouldn't be immediate upon installing the package, as the code is
executed only when new share servers are created. So as long as only
shares are being created while the existing share servers and networks
were already functional prior to the upgrade, the damage can be
mitigated. On the other hand, OVN users are broken without
connectivity so the benefits generally outweight the risks here. In
case of regression, the previous funcionality can be restored through
package downgrade.
[Other Info]
For Jammy/Yoga, bugfix LP#2049507 will also be included.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2074504/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list