[Bug 1988018] Re: [SRU][mlx5] Intermittent VF-LAG activation failure

Robert Malz 1988018 at bugs.launchpad.net
Wed Apr 8 11:13:28 UTC 2026 

I tested following scenario on 0.107.1-3ubuntu0.22.04.3:
ubuntu at romano:~$ cat /etc/netplan/99-bond.yaml 
network:
  version: 2
  ethernets:
    ens7f0:
      virtual-function-count: 16
      embedded-switch-mode: switchdev
      delay-virtual-functions-rebind: true

    ens7f1:
      virtual-function-count: 16
      embedded-switch-mode: switchdev
      delay-virtual-functions-rebind: true

  bonds:
    bond0:
      interfaces:
        - ens7f0
        - ens7f1
      parameters:
        mode: active-backup

bond created as expected:
ubuntu at romano:~$ cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v5.15.0-174-generic

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: ens7f1
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

Slave Interface: ens7f1
MII Status: up
Speed: 100000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b8:3f:d2:2d:68:7f
Slave queue ID: 0

Slave Interface: ens7f0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: b8:3f:d2:2d:68:7e
Slave queue ID: 0

Both interfaces switch to eswitch mode:
ubuntu at romano:~$ sudo devlink dev eswitch show pci/0000:61:00.0
pci/0000:61:00.0: mode switchdev inline-mode none encap-mode basic
ubuntu at romano:~$ sudo devlink dev eswitch show pci/0000:61:00.1
pci/0000:61:00.1: mode switchdev inline-mode none encap-mode basic

Both PFs created 16 VFs:
root at romano:/home/ubuntu# ip l show  ens7f0
5: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 5e:52:b2:1b:0e:07 brd ff:ff:ff:ff:ff:ff permaddr b8:3f:d2:2d:68:7e
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 8     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 9     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 10     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 11     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 12     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 13     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 14     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 15     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    altname enp97s0f0np0
root at romano:/home/ubuntu# ip l show  ens7f1
6: ens7f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 5e:52:b2:1b:0e:07 brd ff:ff:ff:ff:ff:ff permaddr b8:3f:d2:2d:68:7f
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 8     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 9     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 10     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 11     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 12     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 13     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 14     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 15     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    altname enp97s0f1np1

netplan-sriov-apply.service finished properly
root at romano:/home/ubuntu# systemctl status netplan-sriov-apply.service
○ netplan-sriov-apply.service - Apply SR-IOV configuration
     Loaded: loaded (/run/systemd/system/netplan-sriov-apply.service; enabled-runtime; vendor preset: enabled)
     Active: inactive (dead) since Wed 2026-04-08 11:04:13 UTC; 8min ago
    Process: 1193 ExecStartPre=udevadm control --reload (code=exited, status=0/SUCCESS)
    Process: 1276 ExecStartPre=udevadm trigger --action=add --subsystem-match=net (code=exited, status=0/SUCCESS)
    Process: 1279 ExecStartPre=udevadm settle (code=exited, status=0/SUCCESS)
    Process: 1298 ExecStart=/usr/sbin/netplan apply --sriov-only (code=exited, status=0/SUCCESS)
   Main PID: 1298 (code=exited, status=0/SUCCESS)
        CPU: 1.789s

Apr 08 11:03:09 romano netplan[1298]: `gateway4` has been deprecated, use default routes instead.
                                      See the 'Default routes' section of the documentation for more details.
Apr 08 11:03:09 romano netplan[1298]: Permissions for /etc/netplan/99-bond.yaml are too open. Netplan configuration should NOT be accessible by others.
Apr 08 11:03:09 romano netplan[1298]: `gateway4` has been deprecated, use default routes instead.
                                      See the 'Default routes' section of the documentation for more details.
Apr 08 11:03:09 romano netplan[1298]: Permissions for /etc/netplan/99-bond.yaml are too open. Netplan configuration should NOT be accessible by others.
Apr 08 11:04:13 romano systemd[1]: netplan-sriov-apply.service: Deactivated successfully.
Apr 08 11:04:13 romano systemd[1]: Finished Apply SR-IOV configuration.
Apr 08 11:04:13 romano systemd[1]: netplan-sriov-apply.service: Consumed 1.789s CPU time.


** Tags removed: block-proposed-jammy verification-needed-jammy
** Tags added: verification-done-jammy

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to a duplicate bug report
(1977851).
https://bugs.launchpad.net/bugs/1988018

Title:
  [SRU][mlx5] Intermittent VF-LAG activation failure

Status in linux package in Ubuntu:
  Fix Committed
Status in netplan.io package in Ubuntu:
  Fix Released
Status in linux source package in Jammy:
  Confirmed
Status in netplan.io source package in Jammy:
  Fix Committed
Status in linux source package in Kinetic:
  Won't Fix
Status in netplan.io source package in Kinetic:
  Won't Fix
Status in linux source package in Mantic:
  Won't Fix
Status in netplan.io source package in Mantic:
  Won't Fix
Status in linux source package in Noble:
  Fix Committed
Status in netplan.io source package in Noble:
  Fix Released

Bug description:
  [ Impact ]

  Due to limitations in how Netplan handles SR-IOV devices, the VF-LAG
  feature found on Mellanox NICs couldn't be used. Certain configuration steps
  must happen in a very specific order and Netplan fails to perform the set up correctly.

  Netplan must wait until the backend finishes adding interfaces to the Bond
  and the Mellanox driver reports the VF-LAG feature as "active" before binding VFs to
  the driver.

  See also https://bugs.launchpad.net/netplan/+bug/2083008

  This problem is fixed by introducing a proper ordering in the configuration process
  and monitoring the driver state until it reports as ready (or times out).

  This fix is available on Ubuntu 24.04.

  [ Test Plan ]

  To reproduce the problem addressed by this SRU one needs to
  have access to specialized hardware (SR-IOV-capable Mellanox NICs).

  The fix for the problem described above was already verified on Ubuntu 22.04 and
  solved the problem (more details https://bugs.launchpad.net/netplan/+bug/2083008).

  We will work with Canonical's Openstack team to do the fix
  verification.

   * detailed instructions how to reproduce the bug

  A configuration file that looks like the one below can be used
  to test the fix.

  After booting the system with this configuration, the Mellanox driver
  should report the LAG state as "active" for all the devices.
  It can be checked in the debugfs file: /sys/kernel/debug/mlx5/{pci_addr}/lag/state

  network:
    version: 2
    ethernets:
      ens4f0np0:
        virtual-function-count: 16
        embedded-switch-mode: switchdev
        delay-virtual-functions-rebind: true

      ens4f1np1:
        virtual-function-count: 16
        embedded-switch-mode: switchdev
        delay-virtual-functions-rebind: true

    bonds:
      bond0:
        interfaces:
          - ens4f0np0
          - ens4f1np1
        parameters:
          mode: active-backup

  [ Where problems could occur ]

  These changes should affect only SR-IOV related scenarios.
  Undetected problems could cause Netplan to fail to configure the device
  and Virtual Functions wouldn't be created anymore.

  [ Other Info ]

  Related work:

  https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/1988018
  https://github.com/canonical/netplan/pull/439

  A PPA for Ubuntu 22.04 can be found here
  https://launchpad.net/~danilogondolfo/+archive/ubuntu/netplan-sru

  ---- Original bug description ----

  During system initialization there is a specific sequence that must be
  followed to enable the use of hardware offload and VF-LAG.

  Intermittently one may see that VF-LAG initialization fails:
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: lag map port 1:1 port 2:2 shared_fdb:1
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_cmd_check:782:(pid 9): CREATE_LAG(0x840) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x7d49cb)
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_create_lag:248:(pid 9): Failed to create LAG (-22)
  [Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_activate_lag:288:(pid 9): Failed to activate VF LAG
                             Make sure all VFs are unbound prior to VF LAG activation or deactivation

  This is caused by rebinding the driver prior to the VF lag being
  ready.

  A sysfs knob has recently been added to the driver [0] and we should
  monitor it before attempting to rebind the driver:

      $ cat /sys/kernel/debug/mlx5/0000\:08\:00.0/lag/state

  The kernel feature is available in the upcoming Kinetic 5.19 kernel
  and we should probably backport it to the Jammy 5.15 kernel.

  0:
  https://github.com/torvalds/linux/commit/7f46a0b7327ae261f9981888708dbca22c283900

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1988018/+subscriptions

More information about the foundations-bugs mailing list