[Bug 1892132] Re: Failure to get the correct UpLink Representor
Frode Nordahl
1892132 at bugs.launchpad.net
Fri Sep 10 16:13:27 UTC 2021
os-vif bionic/ussuri:
$ uname -a
Linux node-laveran 5.4.0-84-generic #94~18.04.1-Ubuntu SMP Thu Aug 26 23:17:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ dpkg -l | grep os-vif
ii python3-os-vif 2.0.0-0ubuntu2~cloud0 all Integration library between network and compute - Python 3.x
$ sudo grep -A6 hostdev /etc/libvirt/qemu/instance-00000019.xml
<interface type='hostdev' managed='yes'>
<mac address='fa:16:3e:a8:b0:d7'/>
<source>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x4'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
$ sudo ovs-vsctl find interface name=eth2
_uuid : c1b7651a-ca84-481a-bbb3-8531abddd7d4
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
error : []
external_ids : {attached-mac="fa:16:3e:a8:b0:d7", iface-id="f58fbe72-698c-4aa7-b38a-427c68ea660d", iface-status=active, vm-uuid="850b1c28-1fd4-4c83-b3b4-cf1727253bb7"}
ifindex : 77
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 0
link_speed : []
link_state : up
lldp : {}
mac : []
mac_in_use : "0e:93:79:d4:cb:21"
mtu : 8942
mtu_request : []
name : eth2
ofport : 6
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=26846, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_missed_errors=0, rx_over_err=0, rx_packets=273, tx_bytes=15409, tx_dropped=0, tx_errors=0, tx_packets=101}
status : {driver_name=mlx5e_rep, driver_version="5.4.0-84-generic", firmware_version="16.31.1014 (MT_0000000183)"}
type : ""
$ sudo ls -l /sys/class/net/eth2/device
ls: cannot access '/sys/class/net/eth2/device': No such file or directory
$ sudo ls -l /sys/class/net/enp129s0f0/device/net/
total 0
drwxr-xr-x 6 root root 0 Sep 10 15:26 enp129s0f0
<INSTALL OFED DRIVERS AND REBOOT>
$ sudo ovs-vsctl find interface name=enp3s0f0_2
_uuid : 05cf3156-83fa-4c08-af91-45e510d32367
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : full
error : []
external_ids : {attached-mac="fa:16:3e:a8:b0:d7", iface-id="f58fbe72-698c-4aa7-b38a-427c68ea660d", iface-status=active, vm-uuid="850b1c28-1fd4-4c83-b3b4-cf1727253bb7"}
ifindex : 76
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : 10000000000
link_state : up
lldp : {}
mac : []
mac_in_use : "0e:0f:d1:84:b8:10"
mtu : 8942
mtu_request : []
name : enp3s0f0_2
ofport : 7
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=24822, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_missed_errors=0, rx_over_err=0, rx_packets=284, tx_bytes=1503, tx_dropped=0, tx_errors=0, tx_packets=11}
status : {driver_name=mlx5e_rep, driver_version="5.4.0-84-generic", firmware_version="16.31.1014 (MT_0000000183)"}
type : ""
$ sudo ls -l /sys/class/net/enp3s0f0_2/device
lrwxrwxrwx 1 root root 0 Sep 10 15:57 /sys/class/net/enp3s0f0_2/device -> ../../../0000:03:00.0
$ ls -l /sys/class/net/enp129s0f0/device/net/
total 0
drwxr-xr-x 13 root root 0 Sep 10 15:55 enp129s0f0
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_0
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_1
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_10
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_11
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_12
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_13
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_14
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_15
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_16
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_17
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_18
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_19
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_2
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_20
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_21
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_22
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_23
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_24
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_25
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_26
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_27
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_28
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_29
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_3
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_30
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_31
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_4
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_5
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_6
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_7
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_8
drwxr-xr-x 5 root root 0 Sep 10 15:57 enp3s0f0_9
** Tags removed: verification-needed verification-ussuri-needed
** Tags added: verification-done verification-ussuri-done
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1892132
Title:
Failure to get the correct UpLink Representor
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive ussuri series:
Fix Committed
Status in Ubuntu Cloud Archive victoria series:
Fix Committed
Status in os-vif:
Fix Released
Status in os-vif victoria series:
Fix Committed
Status in libvirt package in Ubuntu:
Fix Released
Status in python-os-vif package in Ubuntu:
Fix Released
Status in libvirt source package in Focal:
Fix Released
Status in python-os-vif source package in Focal:
Fix Committed
Status in libvirt source package in Groovy:
Won't Fix
Status in python-os-vif source package in Groovy:
Won't Fix
Status in libvirt source package in Hirsute:
Fix Released
Status in python-os-vif source package in Hirsute:
Fix Released
Status in libvirt source package in Impish:
Fix Released
Status in python-os-vif source package in Impish:
Fix Released
Bug description:
[Impact]
An update to the mlx5_core driver [1] which will be made available to users of stable releases both through HWE kernels and DKMS packages provided by NVIDIA/Mellanox [2] makes some assumptions about sysfs layout made by OS-VIF and Libvirt apparent.
To allow users with this hardware to continue to enjoy their existing
systems with the most recent drivers updates are required to os-vif
and libvirt.
Without this update these systems will stop functioning when upgrading
to the new mlx5_core driver.
[Test Plan]
Note: Hardware making use of the mlx5_core driver with support for HWOL is required to test these changes.
1. Deploy OpenStack on machines with HWOL enabled using kernel without [1]
2. Create an instance using an HWOL port
3. Confirm the instance can start and that it has connectivity
4. Upgrade to kernel with [1] and re-confirm
[Regression Potential]
For OS-VIF the changes are made to code paths used exclusively by consumers of this type of hardware and HWOL enabled. They are also made in a backward compatible way so that it works both with the old and new driver.
For Libvirt the change is made in such a way that it will behave as
before when used to look up hardware that populates net/phys_port_id.
When used with hardware that do not populate net/phys_port_id but use
net_phys_port_name instead, which is typical for the hardware in
question, the new behavior is used.
[Original Bug Description]
Due to new kernel patch here [1], the PF and VF representors are linked to their parent PCI device.
Old Structure:
The structure of VF's PCI Address/physfn/net contains only the PF of that VF
$ ls /sys/bus/pci/devices/<vf-pci-addre>/physfn/net/
enp2s0f0
$ ls -l /sys/class/net
...
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_0 -> ../../devices/virtual/net/enp2s0f0_0
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_1 -> ../../devices/virtual/net/enp2s0f0_1
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_2 -> ../../devices/virtual/net/enp2s0f0_2
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_3 -> ../../devices/virtual/net/enp2s0f0_3
...
New Structure:
The structure of VF's PCI Address/physfn/net contains the PF of that VF and the VF representors
$ ls /sys/bus/pci/devices/<vf-pci-addre>/physfn/net/
enp3s0f0 enp3s0f0_0 enp3s0f0_1 enp3s0f0_2 enp3s0f0_3
$ ls -l /sys/class/net
...
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_0 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_0
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_1 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_1
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_2 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_2
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_3 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_3
...
[1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1
[2] https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1892132/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list