[Bug 2002445] Re: udev NIC renaming race with mlx5_core driver
Launchpad Bug Tracker
2002445 at bugs.launchpad.net
Tue Mar 7 00:24:57 UTC 2023
This bug was fixed in the package systemd - 252.5-2ubuntu1
---------------
systemd (252.5-2ubuntu1) lunar; urgency=medium
* Merge 252.5-2 from Debian unstable
- Drop test-handle-Debian-s-etc-default-locale-in-testsuite-74.f.patch.
Applied upstream: https://github.com/systemd/systemd/commit/9b42646b22
File: debian/patches/test-handle-Debian-s-etc-default-locale-in-testsuite-74.f.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1b0789416172ec60d8086fe2b458b5396bb7e857
- Drop test-make-sure-mount-point-exists-in-testsuite-64.sh.patch.
Applied upstream: https://github.com/systemd/systemd/commit/07e4787106
File: debian/patches/test-make-sure-mount-point-exists-in-testsuite-64.sh.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f97b2d5ae1a1f35668c4648f1c7fc715a588de50
- Drop test-remove-no-longer-needed-quirk-for-set-locale-on-Debi.patch.
Fixed upstream: https://github.com/systemd/systemd-stable/commit/1c325f6d7f
File: debian/patches/test-remove-no-longer-needed-quirk-for-set-locale-on-Debi.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=5f85226d61393c08d7ea51c2f28db7fd4c79bcc6
* udev: avoid NIC renaming race with kernel (LP: #2002445)
Files:
- debian/patches/lp2002445-sd-netlink-add-a-test-for-rtnl_set_link_name.patch
- debian/patches/lp2002445-sd-netlink-do-not-swap-old-name-and-alternative-name.patch
- debian/patches/lp2002445-sd-netlink-restore-altname-on-error-in-rtnl_set_link_name.patch
- debian/patches/lp2002445-test-network-add-a-test-for-renaming-device-to-current-al.patch
- debian/patches/lp2002445-udev-attempt-device-rename-even-if-interface-is-up.patch
- debian/patches/lp2002445-udev-net-allow-new-link-name-as-an-altname-before-renamin.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=58d29c2b376f03c44ed5a719877c95b332018cdc
* Deny-list TEST-74-AUX-UTILS on s390x.
Since this currently is only known to fail on the autopkgtest
infrastructure, we believe this is a temporary issue.
File: debian/patches/Deny-list-TEST-74-AUX-UTILS-on-s390x.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a3a059d86e2fe3a104419ae2afcab557171f9809
systemd (252.5-2) unstable; urgency=medium
* Fix boot-and-services autopkgtest.
systemd (252.5-1) unstable; urgency=medium
[ Nick Rosbrook ]
* debian/tests: remove systemd-fsckd autopkgtest. This test never runs
in Debian autopkgtest because of missing machine isolation
requirements, and it nevers runs in Ubuntu because: SKIP: root file
system is being checked by initramfs already Since the test is not
providing any good feedback, and generally has not been maintained,
let's just remove it.
[ Luca Boccassi ]
* New upstream version 252.5
* Drop patches merged in v252.5
* Refresh patches
* Set default status format to 'combined': show both unit name and
description in logs/boot messages
systemd (252.4-2) unstable; urgency=medium
[ Michael Biebl ]
* Refresh patches
* Tweak description of systemd and systemd-sysv package.
Remove redundancy and de-emphasize sysvinit.
* autopkgtest: add psmsic to upstream suite.
Needed for the killall binary.
See https://github.com/systemd/systemd/pull/24569
* autopkgtest: add xkb-data, locales and locales-all to upstream suite.
Use locales-all so all necessary locales can be installed into the test
image without having to generate them on-the-fly.
See https://github.com/systemd/systemd/pull/23709
* autopkgtest: prefer knot-dnssecutils over knot-dnsutils for upstream
suite.
The kzonecheck utility required by TEST-75-RESOLVED was split out from
knot-dnsutils into knot-dnssecutils so update the test dependencies
accordingly. Keep knot-dnsutils as alternative dependency to make
backports easier.
* Cherry-pick upstream fixes for TEST-74-AUX-UTILS
* Cherry-pick upstream fix for TEST-73-LOCALE
* Skip firstboot --prompt-keymap check in TEST-74-AUX-UTILS.
This test requires compatible keymaps from kbd which are not available
in Debian.
[ Luca Boccassi ]
* autopkgtest: add netlabel-tools to networkd-test.py suite.
The netlabelctl tool is needed to test the NetLabel integration.
See https://github.com/systemd/systemd/pull/23888
* autopkgtest: add bsdutils to upstream suite.
The logger utility is now used in TEST-04-JOURNAL.
See https://github.com/systemd/systemd/pull/23086
* autopkgtest: add knot, knot-dnsutils, bind9-dnsutils, bind9-host to
upstream suite.
Needed by TEST-75-RESOLVED.
See https://github.com/systemd/systemd/pull/23104
* autopkgtest: add jq to upstream suite.
Needed by TEST-58-REPART.
See https://github.com/systemd/systemd/pull/24572
* autopkgtest: add mtools to upstream suite.
Needed by TEST-58-REPART.
See https://github.com/systemd/systemd/pull/24944
* autopkgtest: add erofs-utils to upstream suite.
Needed by TEST-58-REPART.
See https://github.com/systemd/systemd/pull/25686
-- Nick Rosbrook <nick.rosbrook at canonical.com> Tue, 14 Feb 2023
11:52:31 -0500
** Changed in: systemd (Ubuntu Lunar)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2002445
Title:
udev NIC renaming race with mlx5_core driver
Status in systemd package in Ubuntu:
Fix Released
Status in systemd source package in Focal:
Triaged
Status in systemd source package in Jammy:
Triaged
Status in systemd source package in Kinetic:
New
Status in systemd source package in Lunar:
Fix Released
Bug description:
[Impact]
On systems with mellanox NICs, udev's NIC renaming races with the mlx5_core driver's own configuration of subordinate interfaces. When the kernel wins this race, the device cannot be renamed as udev has attempted, and this causes systemd-network-online.target to timeout waiting for links to be configured. This ultimately results in boot being delayed by about 2 minutes.
[Test Plan]
Repeated launches of Standard_D8ds_v5 instance types will generally hit this race around 1 in 10 runs. Create a vm snapshot with updated systemd from ppa:enr0n/systemd-245. Launch 100 Standard_D8ds_v5 instances with updated systemd. Assert not failure in cloud-init status and no 2 minute delay in network-online.target.
To check for failure symptom:
- Assert that network-online.target isn't the longest pole from systemd-analyze blame.
To assert success condition during net rename busy race:
- assert when "eth1" is still the primary device name, that two altnames are listed (preserving the altname due to the primary NIC rename being hit.
Sample script uses pycloudlib to create modified base image for test
and launches 100 VMs of type Standard_D8ds_v5, counting both successes
and any failures seen.
#!/usr/bin/env python3
# This file is part of pycloudlib. See LICENSE file for license information.
"""Basic examples of various lifecycle with an Azure instance."""
import logging
import json
import pycloudlib
LOG = logging.getLogger()
base_cfg = """#cloud-config
ssh-import-id: [chad.smith, enr0n]
"""
apt_cfg = """
# Add developer PPA
apt:
sources:
systemd-testing:
source: "deb [allow-insecure=yes] https://ppa.launchpadcontent.net/enr0n/systemd-245/ubuntu focal main"
# upgrade systemd after cloud-init is nearly done
runcmd:
- apt install systemd udev -y --allow-unauthenticated
"""
debug_systemd_cfg = """
# Create systemd-udev debug override.conf in base image
write_files:
- path: /etc/systemd/system/systemd-networkd.service.d/override.conf
owner: root:root
defer: {defer}
content: |
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
- path: /etc/systemd/system/systemd-udevd.service.d/override.conf
owner: root:root
defer: {defer}
content: |
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
LogRateLimitIntervalSec=0
"""
cloud_config = base_cfg + apt_cfg + debug_systemd_cfg
cloud_config2 = base_cfg + debug_systemd_cfg
def debug_systemd_image_launch_overlake_v5_with_snapshot():
"""Test overlake v5 timeouts
test procedure:
- Launch base focal image
- enable ppa:enr0n/systemd-245 and systemd/udev debugging
- cloud-init clean --logs && deconfigure waalinux agent before shutdown
- snapshot a base image
- launch v5 system from snapshot
- check systemd-analyze for expected timeout
"""
client = pycloudlib.Azure(tag="azure")
image_id = client.daily_image(release="focal")
pub_path = "/home/ubuntu/.ssh/id_rsa.pub"
priv_path = "/home/ubuntu/.ssh/id_rsa"
client.use_key(pub_path, priv_path)
base_instance = client.launch(
image_id=image_id,
instance_type="Standard_DS1_v2",
user_data=cloud_config.format(defer="true"),
)
LOG.info(f"base instance: ssh ubuntu@{base_instance.ip}")
base_instance.wait()
LOG.info(base_instance.execute("apt cache policy systemd"))
snapshotted_image_id = client.snapshot(base_instance)
reproducer = False
tries = 0
success_count_with_race = 0
success_count_no_race = 0
failure_count_network_delay = 0
failure_count_no_altnames = 0
TEST_SUMMARY_TMPL = """
----- Test run complete: {tries} attempted -----
Successes without rename race: {success_count_no_race}
Successes with rename race and preserved altname: {success_count_with_race}
Failures due to network delay: {failure_count_network_delay}
Failures due to no altnames persisted: {failure_count_no_altnames}
===================================
"""
while tries < 100 and not reproducer:
tries += 1
new_instance = client.launch(
image_id=snapshotted_image_id,
instance_type="Standard_D8ds_v5",
user_data=cloud_config.format(defer="false"),
)
# breadcrumb for us pycloudlib/Azure will reuse available IPs
new_instance.wait()
blame = new_instance.execute("systemd-analyze blame").splitlines()
LOG.info(f"--- Attempt {tries} ssh ubuntu@{new_instance.ip} Blame: {blame[0]}")
ip_addr = json.loads(new_instance.execute("ip -j addr").stdout)
rename_race_present = False # set true when we see eth1 not renamed
for d in ip_addr:
if d['ifname'] == "eth1":
rename_race_present = True
if len(d.get('altnames', [])) > 1:
LOG.info(
f"--- SUCCESS persisting altnames {d['altnames']} due to rename race on resource busy on {d['ifname']}"
)
else:
failure_count_no_altnames += 1
LOG.error(f"FAILURE: to preserve altnames for {d['ifname']}. Only preserved {d.get('altnames', [])}")
LOG.info(new_instance.execute("journalctl -u systemd-udevd.service -b 0 --no-pager"))
LOG.info('\n'.join([f'{d["ifname"]}: {d.get("altnames")}' for d in ip_addr]))
if "systemd-networkd-wait-online.service" not in blame[0]:
if rename_race_present:
success_count_with_race += 1
else:
success_count_no_race += 1
LOG.info(f"Destroying instance, normal boot seen: {blame[0]}")
new_instance.delete()
else:
failure_count_network_delay += 1
LOG.info(f"--- Attempt {tries} found delayed instance boot: {blame[0]}")
r = new_instance.execute("journalctl -u systemd-udevd.service -b 0 --no-pager")
LOG.info(r)
if "Failure to rename" in str(r):
LOG.info(f"Found rename refusal!: {r[0]}")
new_instance.delete()
LOG.info(
TEST_SUMMARY_TMPL.format(
success_count_with_race=success_count_with_race,
success_count_no_race=success_count_no_race,
failure_count_network_delay=failure_count_network_delay,
failure_count_no_altnames=failure_count_no_altnames, tries=tries
)
)
base_instance.delete(wait=False)
if __name__ == "__main__":
# Avoid polluting the log with azure info
logging.getLogger("paramiko").setLevel(logging.WARNING)
logging.getLogger("pycloudlib").setLevel(logging.WARNING)
logging.getLogger("adal-python").setLevel(logging.WARNING)
logging.getLogger("cli.azure.cli.core").setLevel(logging.WARNING)
logging.basicConfig(filename="sru-systemd.log", level=logging.INFO)
debug_systemd_image_launch_overlake_v5_with_snapshot()
[Where problems could occur]
The patches effectively make it so that if a interface cannot be renamed from udev, then the new name is left as an alternative name as a fallback. If problems occur, it would be related to device renaming, and particularly related to the devices alternative names.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2002445/+subscriptions
More information about the foundations-bugs
mailing list