[Bug 1988018] Re: [SRU][mlx5] Intermittent VF-LAG activation failure

Lukas Märdian 1988018 at bugs.launchpad.net
Thu Feb 27 15:16:22 UTC 2025 

After a reboot with the configuration from the test case on Mellanox
CX6, the system comes up in a half broken state (not all VFs created,
"legacy" eswitch mode, LAG disabled):

ubuntu at romano:~$ sudo lshw -c network -businfo
Bus info          Device          Class          Description
============================================================
pci at 0000:21:00.0  ens13f0np0      network        BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
pci at 0000:21:00.1  ens13f1np1      network        BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
pci at 0000:61:00.0  ens7f0          network        MT2892 Family [ConnectX-6 Dx]
pci at 0000:61:00.1  ens7f1          network        MT2892 Family [ConnectX-6 Dx]
pci at 0000:61:00.2  ens7f0v0        network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.3  ens7f0v1        network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.4  ens7f0v2        network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.5  ens7f0v3        network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.6  ens7f0v4        network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.7  ens7f0v5        network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.0  ens7f0v6        network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.1  ens7f0v7        network        ConnectX Family mlx5Gen Virtual Function
ubuntu at romano:~$ sudo devlink dev eswitch show pci/0000:61:00.0
pci/0000:61:00.0: mode legacy inline-mode none encap-mode basic
ubuntu at romano:~$ sudo devlink dev eswitch show pci/0000:61:00.1
pci/0000:61:00.1: mode legacy inline-mode none encap-mode basic
ubuntu at romano:~$ sudo cat /sys/kernel/debug/mlx5/0000:61:00.0/lag/state
disabled
ubuntu at romano:~$ sudo cat /sys/kernel/debug/mlx5/0000:61:00.1/lag/state
disabled


Interestingly, after running "netplan apply" after the reboot, the system seems to transition into the desired state:
ubuntu at romano:~$ sudo netplan apply
[]
Cannot find unique matching interface for ens7f0
[]
Cannot find unique matching interface for ens7f1
ubuntu at romano:~$ sudo lshw -c network -businfo
Bus info          Device          Class          Description
============================================================
pci at 0000:21:00.0  ens13f0np0      network        BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
pci at 0000:21:00.1  ens13f1np1      network        BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
pci at 0000:61:00.0  ens7f0          network        MT2892 Family [ConnectX-6 Dx]
pci at 0000:61:00.1  ens7f1          network        MT2892 Family [ConnectX-6 Dx]
pci at 0000:61:00.2                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.3                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.4                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.5                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.6                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.7                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.0                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.1                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.2                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.3                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.4                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.5                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.6                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:01.7                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:02.0                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:02.1                  network        ConnectX Family mlx5Gen Virtual Function
pci at 0000:61:00.0  ens7f0npf0vf0   network        Ethernet interface
pci at 0000:61:00.0  ens7f0npf0vf1   network        Ethernet interface
pci at 0000:61:00.0  ens7f0npf0vf2   network        Ethernet interface
pci at 0000:61:00.0  ens7f0npf0vf3   network        Ethernet interface
pci at 0000:61:00.0  ens7f0npf0vf4   network        Ethernet interface
pci at 0000:61:00.0  ens7f0npf0vf5   network        Ethernet interface
pci at 0000:61:00.0  ens7f0npf0vf6   network        Ethernet interface
pci at 0000:61:00.0  ens7f0npf0vf7   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf0   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf1   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf2   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf3   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf4   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf5   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf6   network        Ethernet interface
pci at 0000:61:00.1  ens7f1npf1vf7   network        Ethernet interface
ubuntu at romano:~$ sudo devlink dev eswitch show pci/0000:61:00.1
pci/0000:61:00.1: mode switchdev inline-mode none encap-mode basic
ubuntu at romano:~$ sudo devlink dev eswitch show pci/0000:61:00.0
pci/0000:61:00.0: mode switchdev inline-mode none encap-mode basic
ubuntu at romano:~$ sudo cat /sys/kernel/debug/mlx5/0000:61:00.1/lag/state
active
ubuntu at romano:~$ sudo cat /sys/kernel/debug/mlx5/0000:61:00.0/lag/state
active


This isn't quite as it should be, but might be an easy workaround for Mellanox CX6..

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to a duplicate bug report
(1977851).
https://bugs.launchpad.net/bugs/1988018

Title:
  [SRU][mlx5] Intermittent VF-LAG activation failure

Status in linux package in Ubuntu:
  Fix Committed
Status in netplan.io package in Ubuntu:
  Fix Released
Status in linux source package in Jammy:
  Confirmed
Status in netplan.io source package in Jammy:
  Fix Committed
Status in linux source package in Kinetic:
  Won't Fix
Status in netplan.io source package in Kinetic:
  Won't Fix
Status in linux source package in Mantic:
  Won't Fix
Status in netplan.io source package in Mantic:
  Won't Fix
Status in linux source package in Noble:
  Fix Committed
Status in netplan.io source package in Noble:
  Fix Released

Bug description:
  [ Impact ]

  Due to limitations in how Netplan handles SR-IOV devices, the VF-LAG
  feature found on Mellanox NICs couldn't be used. Certain configuration steps
  must happen in a very specific order and Netplan fails to perform the set up correctly.

  Netplan must wait until the backend finishes adding interfaces to the Bond
  and the Mellanox driver reports the VF-LAG feature as "active" before binding VFs to
  the driver.

  See also https://bugs.launchpad.net/netplan/+bug/2083008

  This problem is fixed by introducing a proper ordering in the configuration process
  and monitoring the driver state until it reports as ready (or times out).

  This fix is available on Ubuntu 24.04.

  [ Test Plan ]

  To reproduce the problem addressed by this SRU one needs to
  have access to specialized hardware (SR-IOV-capable Mellanox NICs).

  The fix for the problem described above was already verified on Ubuntu 22.04 and
  solved the problem (more details https://bugs.launchpad.net/netplan/+bug/2083008).

  We will work with Canonical's Openstack team to do the fix
  verification.

   * detailed instructions how to reproduce the bug

  A configuration file that looks like the one below can be used
  to test the fix.

  After booting the system with this configuration, the Mellanox driver
  should report the LAG state as "active" for all the devices.
  It can be checked in the debugfs file: /sys/kernel/debug/mlx5/{pci_addr}/lag/state

  network:
    version: 2
    ethernets:
      ens4f0np0:
        virtual-function-count: 16
        embedded-switch-mode: switchdev
        delay-virtual-functions-rebind: true

      ens4f1np1:
        virtual-function-count: 16
        embedded-switch-mode: switchdev
        delay-virtual-functions-rebind: true

    bonds:
      bond0:
        interfaces:
          - ens4f0np0
          - ens4f1np1
        parameters:
          mode: active-backup

  [ Where problems could occur ]

  These changes should affect only SR-IOV related scenarios.
  Undetected problems could cause Netplan to fail to configure the device
  and Virtual Functions wouldn't be created anymore.

  [ Other Info ]

  Related work:

  https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/1988018
  https://github.com/canonical/netplan/pull/439

  A PPA for Ubuntu 22.04 can be found here
  https://launchpad.net/~danilogondolfo/+archive/ubuntu/netplan-sru

  ---- Original bug description ----

  During system initialization there is a specific sequence that must be
  followed to enable the use of hardware offload and VF-LAG.

  Intermittently one may see that VF-LAG initialization fails:
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: lag map port 1:1 port 2:2 shared_fdb:1
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_cmd_check:782:(pid 9): CREATE_LAG(0x840) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x7d49cb)
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_create_lag:248:(pid 9): Failed to create LAG (-22)
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_activate_lag:288:(pid 9): Failed to activate VF LAG
                             Make sure all VFs are unbound prior to VF LAG activation or deactivation

  This is caused by rebinding the driver prior to the VF lag being
  ready.

  A sysfs knob has recently been added to the driver [0] and we should
  monitor it before attempting to rebind the driver:

      $ cat /sys/kernel/debug/mlx5/0000\:08\:00.0/lag/state

  The kernel feature is available in the upcoming Kinetic 5.19 kernel
  and we should probably backport it to the Jammy 5.15 kernel.

  0:
  https://github.com/torvalds/linux/commit/7f46a0b7327ae261f9981888708dbca22c283900

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1988018/+subscriptions

More information about the foundations-bugs mailing list