[Bug 1989190] [NEW] Bionic networking failures after NIC reordering
Chris Patterson
1989190 at bugs.launchpad.net
Fri Sep 9 12:38:02 UTC 2022
Public bug reported:
Documented across https://bugs.launchpad.net/bugs/1958280 and
https://canonical.force.com/ua/s/case/5004K00000E96qlQAB/vf-nic-not-
getting-renamed-properly-for-ubuntu-2004.
Creating this bug to focus on Bionic, because it's different than 20.04+
and last week's failure
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119 helped me
identify part of the root cause.
When NICs are renamed on boot, networkd tends to fail to configure them.
####################################
# WITHOUT THE PROPOSED SYSTEMD PATCH
####################################
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether n/a unmanaged
4 eth2 ether routable configured
5 eth3 ether routable configured
6 eth4 ether routable configured
7 eth5 ether off unmanaged
8 eth6 ether off unmanaged
9 eth7 ether off unmanaged
### As expected, we can see the properties are missing.
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ sudo udevadm info /sys/class/net/eth7
P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022481f-69aa-0022-481f-69aa0022481f/net/eth7
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022481f-69aa-0022-481f-69aa0022481f/net/rename9
E: ID_NET_NAME_MAC=enx0022481f69aa
E: ID_OUI_FROM_DATABASE=Microsoft Corporation
E: ID_PATH=acpi-VMBUS:01
E: ID_PATH_TAG=acpi-VMBUS_01
E: IFINDEX=9
E: INTERFACE=eth1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/rename9 /sys/subsystem/net/devices/eth1 /sys/subsystem/net/devices/cirename0 /sys/subsystem/net/devices/eth7
E: TAGS=:systemd:
E: USEC_INITIALIZED=11203606
### As expected, restarting networkd does not fix the issue.
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ sudo systemctl restart systemd-networkd
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether off unmanaged
4 eth2 ether routable configured
5 eth3 ether routable configured
6 eth4 ether routable configured
7 eth5 ether off unmanaged
8 eth6 ether off unmanaged
9 eth7 ether off unmanaged
9 links listed.
####################################
# WITH THE PROPOSED SYSTEMD PATCH
####################################
I built systemd with the proposed patches in
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119. With
these patches, networking still comes up broken, but restarting networkd
does fix things.
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether n/a unmanaged
4 eth2 ether n/a unmanaged
5 eth3 ether n/a unmanaged
6 eth4 ether routable configured
7 eth5 ether n/a unmanaged
8 eth6 ether n/a unmanaged
9 eth7 ether n/a unmanaged
9 links listed.
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ sudo udevadm info /sys/class/net/eth1
P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022482b-f769-0022-482b-f7690022482b/net/eth1
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022482b-f769-0022-482b-f7690022482b/net/rename3
E: ID_NET_DRIVER=hv_netvsc
E: ID_NET_LINK_FILE=/run/systemd/network/10-netplan-eth7.link
E: ID_NET_NAME=eth1
E: ID_NET_NAME_MAC=enx0022482bf769
E: ID_OUI_FROM_DATABASE=Microsoft Corporation
E: ID_PATH=acpi-VMBUS:01
E: ID_PATH_TAG=acpi-VMBUS_01
E: IFINDEX=3
E: INTERFACE=eth7
E: NM_UNMANAGED=1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/rename3 /sys/subsystem/net/devices/eth7 /sys/subsystem/net/devices/eth1
E: TAGS=:systemd:
E: USEC_INITIALIZED=10280176
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ sudo systemctl restart systemd-networkd
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether routable configured
4 eth2 ether routable configured
5 eth3 ether routable configured
6 eth4 ether routable configured
7 eth5 ether routable configured
8 eth6 ether routable configured
9 eth7 ether routable configured
9 links listed.
I've attached the journal logs for repro with the proposed systemd
(237-3ubuntu10.55).
** Affects: systemd (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "boot log for reproducing with proposed systemd patch"
https://bugs.launchpad.net/bugs/1989190/+attachment/5614804/+files/journal-systemd-55.log
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1989190
Title:
Bionic networking failures after NIC reordering
Status in systemd package in Ubuntu:
New
Bug description:
Documented across https://bugs.launchpad.net/bugs/1958280 and
https://canonical.force.com/ua/s/case/5004K00000E96qlQAB/vf-nic-not-
getting-renamed-properly-for-ubuntu-2004.
Creating this bug to focus on Bionic, because it's different than
20.04+ and last week's failure
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119 helped
me identify part of the root cause.
When NICs are renamed on boot, networkd tends to fail to configure
them.
####################################
# WITHOUT THE PROPOSED SYSTEMD PATCH
####################################
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether n/a unmanaged
4 eth2 ether routable configured
5 eth3 ether routable configured
6 eth4 ether routable configured
7 eth5 ether off unmanaged
8 eth6 ether off unmanaged
9 eth7 ether off unmanaged
### As expected, we can see the properties are missing.
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ sudo udevadm info /sys/class/net/eth7
P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022481f-69aa-0022-481f-69aa0022481f/net/eth7
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022481f-69aa-0022-481f-69aa0022481f/net/rename9
E: ID_NET_NAME_MAC=enx0022481f69aa
E: ID_OUI_FROM_DATABASE=Microsoft Corporation
E: ID_PATH=acpi-VMBUS:01
E: ID_PATH_TAG=acpi-VMBUS_01
E: IFINDEX=9
E: INTERFACE=eth1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/rename9 /sys/subsystem/net/devices/eth1 /sys/subsystem/net/devices/cirename0 /sys/subsystem/net/devices/eth7
E: TAGS=:systemd:
E: USEC_INITIALIZED=11203606
### As expected, restarting networkd does not fix the issue.
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ sudo systemctl restart systemd-networkd
cpatterson at test-ubu1804-nicrenamerepro-x1:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether off unmanaged
4 eth2 ether routable configured
5 eth3 ether routable configured
6 eth4 ether routable configured
7 eth5 ether off unmanaged
8 eth6 ether off unmanaged
9 eth7 ether off unmanaged
9 links listed.
####################################
# WITH THE PROPOSED SYSTEMD PATCH
####################################
I built systemd with the proposed patches in
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119. With
these patches, networking still comes up broken, but restarting
networkd does fix things.
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether n/a unmanaged
4 eth2 ether n/a unmanaged
5 eth3 ether n/a unmanaged
6 eth4 ether routable configured
7 eth5 ether n/a unmanaged
8 eth6 ether n/a unmanaged
9 eth7 ether n/a unmanaged
9 links listed.
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ sudo udevadm info /sys/class/net/eth1
P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022482b-f769-0022-482b-f7690022482b/net/eth1
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022482b-f769-0022-482b-f7690022482b/net/rename3
E: ID_NET_DRIVER=hv_netvsc
E: ID_NET_LINK_FILE=/run/systemd/network/10-netplan-eth7.link
E: ID_NET_NAME=eth1
E: ID_NET_NAME_MAC=enx0022482bf769
E: ID_OUI_FROM_DATABASE=Microsoft Corporation
E: ID_PATH=acpi-VMBUS:01
E: ID_PATH_TAG=acpi-VMBUS_01
E: IFINDEX=3
E: INTERFACE=eth7
E: NM_UNMANAGED=1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/rename3 /sys/subsystem/net/devices/eth7 /sys/subsystem/net/devices/eth1
E: TAGS=:systemd:
E: USEC_INITIALIZED=10280176
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ sudo systemctl restart systemd-networkd
cpatterson at test-ubu1804-nicrenamerepro-systemd55-x2:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether routable configured
4 eth2 ether routable configured
5 eth3 ether routable configured
6 eth4 ether routable configured
7 eth5 ether routable configured
8 eth6 ether routable configured
9 eth7 ether routable configured
9 links listed.
I've attached the journal logs for repro with the proposed systemd
(237-3ubuntu10.55).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1989190/+subscriptions
More information about the foundations-bugs
mailing list