[Bug 2037667] Re: Regression on Jammy's kernel 5.15 when creating ip6gre and vti6 tunnels

Nick Rosbrook 2037667 at bugs.launchpad.net
Tue Nov 19 14:44:39 UTC 2024 

I verified the fix using systemd 249.11-0ubuntu3.13 in jammy-proposed.

First, I confirmed that my systemd and kernel combination is affected by
this bug:

$ uname -a
Linux clean-jammy-amd64 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ apt policy systemd
systemd:
  Installed: 249.11-0ubuntu3.12
  Candidate: 249.11-0ubuntu3.12
  Version table:
 *** 249.11-0ubuntu3.12 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
$ cat > /etc/netplan/10-tun.yaml <<EOF
network:
  renderer: networkd
  version: 2
  tunnels:
    tun0:
      mode: ip6gre
      local: fe80::1
      remote: 2001:dead:beef::2
    tun1:
      mode: vti6
      local: fe80::2
      remote: 2001:dead:beef::3
EOF
$ netplan apply

** (generate:2017): WARNING **: 09:34:31.572: Permissions for
/etc/netplan/01-network-manager-all.yaml are too open. Netplan
configuration should NOT be accessible by others.

** (generate:2017): WARNING **: 09:34:31.573: Permissions for /etc/netplan/10-tun.yaml are too open. Netplan configuration should NOT be accessible by others.
WARNING:root:Cannot call Open vSwitch: ovsdb-server.service is not running.

** (process:2015): WARNING **: 09:34:32.082: Permissions for
/etc/netplan/01-network-manager-all.yaml are too open. Netplan
configuration should NOT be accessible by others.

** (process:2015): WARNING **: 09:34:32.083: Permissions for
/etc/netplan/10-tun.yaml are too open. Netplan configuration should NOT
be accessible by others.

** (process:2015): WARNING **: 09:34:32.302: Permissions for
/etc/netplan/01-network-manager-all.yaml are too open. Netplan
configuration should NOT be accessible by others.

** (process:2015): WARNING **: 09:34:32.302: Permissions for
/etc/netplan/10-tun.yaml are too open. Netplan configuration should NOT
be accessible by others.

** (process:2015): WARNING **: 09:34:32.302: Permissions for
/etc/netplan/01-network-manager-all.yaml are too open. Netplan
configuration should NOT be accessible by others.

** (process:2015): WARNING **: 09:34:32.302: Permissions for /etc/netplan/10-tun.yaml are too open. Netplan configuration should NOT be accessible by others.
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:c6:8b:40 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ip6tnl0 at NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd :: permaddr 5a74:cc76:65cf::
4: ip6gre0 at NONE: <NOARP> mtu 1448 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre6 :: brd :: permaddr dafd:9a91:813c::
5: ip6_vti0 at NONE: <NOARP> mtu 1332 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd :: permaddr c6ed:bf78:136c::
$ journalctl -u systemd-networkd -b --since "1 min ago" --no-pager
Nov 19 09:34:32 clean-jammy-amd64 systemd-networkd[1874]: tun0: netdev could not be created: Invalid argument
Nov 19 09:34:32 clean-jammy-amd64 systemd-networkd[1874]: tun1: netdev could not be created: Invalid argument

Then, I installed systemd 249.11-0ubuntu3.13 from jammy-proposed, and
confirmed the bug was resolved:

$ cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
$ apt update
[ ... ]
$ apt install -y systemd
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libflashrom1 libftdi1-2 libllvm13 linux-headers-5.15.0-25 linux-headers-5.15.0-25-generic linux-headers-5.15.0-27 linux-headers-5.15.0-27-generic linux-image-5.15.0-25-generic
  linux-image-5.15.0-27-generic linux-modules-5.15.0-25-generic linux-modules-5.15.0-27-generic linux-modules-extra-5.15.0-25-generic linux-modules-extra-5.15.0-27-generic
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libnss-systemd libpam-systemd libsystemd0 systemd-oomd systemd-sysv systemd-timesyncd
Suggested packages:
  systemd-container libtss2-rc0
The following packages will be upgraded:
  libnss-systemd libpam-systemd libsystemd0 systemd systemd-oomd systemd-sysv systemd-timesyncd
7 upgraded, 0 newly installed, 0 to remove and 177 not upgraded.
Need to get 5,312 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 libnss-systemd amd64 249.11-0ubuntu3.13 [133 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 systemd-timesyncd amd64 249.11-0ubuntu3.13 [31.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 systemd-sysv amd64 249.11-0ubuntu3.13 [10.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 systemd-oomd amd64 249.11-0ubuntu3.13 [34.8 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 libpam-systemd amd64 249.11-0ubuntu3.13 [203 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 systemd amd64 249.11-0ubuntu3.13 [4,581 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 libsystemd0 amd64 249.11-0ubuntu3.13 [318 kB]
Fetched 5,312 kB in 1s (3,947 kB/s)
(Reading database ... 275907 files and directories currently installed.)
Preparing to unpack .../0-libnss-systemd_249.11-0ubuntu3.13_amd64.deb ...
Unpacking libnss-systemd:amd64 (249.11-0ubuntu3.13) over (249.11-0ubuntu3.12) ...
Preparing to unpack .../1-systemd-timesyncd_249.11-0ubuntu3.13_amd64.deb ...
Unpacking systemd-timesyncd (249.11-0ubuntu3.13) over (249.11-0ubuntu3.12) ...
Preparing to unpack .../2-systemd-sysv_249.11-0ubuntu3.13_amd64.deb ...
Unpacking systemd-sysv (249.11-0ubuntu3.13) over (249.11-0ubuntu3.12) ...
Preparing to unpack .../3-systemd-oomd_249.11-0ubuntu3.13_amd64.deb ...
Unpacking systemd-oomd (249.11-0ubuntu3.13) over (249.11-0ubuntu3.12) ...
Preparing to unpack .../4-libpam-systemd_249.11-0ubuntu3.13_amd64.deb ...
Unpacking libpam-systemd:amd64 (249.11-0ubuntu3.13) over (249.11-0ubuntu3.12) ...
Preparing to unpack .../5-systemd_249.11-0ubuntu3.13_amd64.deb ...
Unpacking systemd (249.11-0ubuntu3.13) over (249.11-0ubuntu3.12) ...
Preparing to unpack .../6-libsystemd0_249.11-0ubuntu3.13_amd64.deb ...
Unpacking libsystemd0:amd64 (249.11-0ubuntu3.13) over (249.11-0ubuntu3.12) ...
Setting up libsystemd0:amd64 (249.11-0ubuntu3.13) ...
Setting up systemd (249.11-0ubuntu3.13) ...
Setting up systemd-timesyncd (249.11-0ubuntu3.13) ...
Setting up systemd-oomd (249.11-0ubuntu3.13) ...
Setting up systemd-sysv (249.11-0ubuntu3.13) ...
Setting up libnss-systemd:amd64 (249.11-0ubuntu3.13) ...
Setting up libpam-systemd:amd64 (249.11-0ubuntu3.13) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
$ apt policy systemd
systemd:
  Installed: 249.11-0ubuntu3.13
  Candidate: 249.11-0ubuntu3.13
  Version table:
 *** 249.11-0ubuntu3.13 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.12 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
$ ip link 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:c6:8b:40 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ip6tnl0 at NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd :: permaddr 5a74:cc76:65cf::
4: ip6gre0 at NONE: <NOARP> mtu 1448 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre6 :: brd :: permaddr dafd:9a91:813c::
5: ip6_vti0 at NONE: <NOARP> mtu 1332 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd :: permaddr c6ed:bf78:136c::
6: tun0 at NONE: <NOARP,UP,LOWER_UP> mtu 1448 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/gre6 fe80::1 brd 2001:dead:beef::2 permaddr eeb8:3eff:9c04::
7: tun1 at NONE: <NOARP,UP,LOWER_UP> mtu 1332 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/tunnel6 fe80::2 brd 2001:dead:beef::3 permaddr 8e47:2223:4643::
$ journalctl -u systemd-networkd -b --since "1 min ago" --no-pager
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun0: Re-configuring with /run/systemd/network/10-netplan-tun0.network
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun0: DHCPv6 lease lost
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun1: Re-configuring with /run/systemd/network/10-netplan-tun1.network
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun1: DHCPv6 lease lost
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun0: Re-configuring with /run/systemd/network/10-netplan-tun0.network
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun0: DHCPv6 lease lost
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun1: Re-configuring with /run/systemd/network/10-netplan-tun1.network
Nov 19 09:41:36 clean-jammy-amd64 systemd-networkd[3014]: tun1: DHCPv6 lease lost


** Tags removed: verification-needed verification-needed-jammy
** Tags added: verification-done verification-done-jammy

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2037667

Title:
  Regression on Jammy's kernel 5.15 when creating ip6gre and vti6
  tunnels

Status in linux package in Ubuntu:
  Confirmed
Status in systemd package in Ubuntu:
  Invalid
Status in linux source package in Jammy:
  Triaged
Status in systemd source package in Jammy:
  Fix Committed

Bug description:
  [Impact]

  We noticed that some of Netplan's integration tests started to fail on
  Jammy. These tests will try to create ip6gre and vti6 virtual
  interfaces and systemd-networkd is failing to create them starting on
  kernel 5.15.0-83.92. As far as I can tell, kernel 5.15.0-82.91 is the
  last revision where it works. So, some change between 5.15.0-82.91 and
  5.15.0-83.92 is causing this regression.

  [Test Plan]

  This test can be done on any Jammy machine. It is best to test in a
  VM.

  # Create a netplan file that creates 2 tunnels:

  $ cat > /etc/netplan/10-tun.yaml <<EOF
  network:
    renderer: networkd
    version: 2
    tunnels:
      tun0:
        mode: ip6gre
        local: fe80::1
        remote: 2001:dead:beef::2
      tun1:
        mode: vti6
        local: fe80::2
        remote: 2001:dead:beef::3
  EOF
  $ netplan apply

  # Check with "ip link" that both tun0 and tun1 *were not created* and
  check networkd for errors:

  $ ip link
  $ journalctl -u systemd-networkd

  On an affected system, we would see logs like:

  Sep 28 17:04:40 jammy systemd-networkd[360]: tun0: netdev could not be created: Invalid argument
  Sep 28 17:04:40 jammy systemd-networkd[360]: tun1: netdev could not be created: Invalid argument

  On a patched system, we should see the tun0 and tun1 devices, and the
  above logs should not be present.

  [Where problems could occur]

  The proposed patch is a minimized version of an upstream patch. The
  result is a patch that disables the "generate_mac" flag for non-ether
  devices. Hence, any problems related to this patch would be related to
  systemd-networkd creating certain types of net devices.

  [Original Description]

  We noticed that some of Netplan's integration tests started to fail on
  Jammy. These tests will try to create ip6gre and vti6 virtual
  interfaces and systemd-networkd is failing to create them starting on
  kernel 5.15.0-83.92. As far as I can tell, kernel 5.15.0-82.91 is the
  last revision where it works. So, some change between 5.15.0-82.91 and
  5.15.0-83.92 is causing this regression.

  How to reproduce the issue:

  # Launch a jammy cloud VM:

  lxc launch images:ubuntu/jammy/cloud jammy --vm
  lxc shell jammy

  # Create a netplan file that creates 2 tunnels:

  cat > /etc/netplan/10-tun.yaml <<EOF
  network:
    renderer: networkd
    version: 2
    tunnels:
      tun0:
        mode: ip6gre
        local: fe80::1
        remote: 2001:dead:beef::2
      tun1:
        mode: vti6
        local: fe80::2
        remote: 2001:dead:beef::3
  EOF

  # Apply the configuration

  netplan apply

  # Check with "ip link" that both tun0 and tun1 *were not created* and
  check networkd for errors:

  journalctl -u systemd-networkd

  Sep 28 17:04:40 jammy systemd-networkd[360]: tun0: netdev could not be created: Invalid argument
  Sep 28 17:04:40 jammy systemd-networkd[360]: tun1: netdev could not be created: Invalid argument

  # Download, install and boot on kernel 5.15.0-82.91

  wget http://ie.archive.ubuntu.com/ubuntu/pool/main/l/linux-
  signed/linux-image-5.15.0-82-generic_5.15.0-82.91_amd64.deb
  http://ie.archive.ubuntu.com/ubuntu/pool/main/l/linux/linux-
  modules-5.15.0-82-generic_5.15.0-82.91_amd64.deb

  dpkg -i *.deb

  grub-reboot '1>2' && reboot

  # Check with "ip link" again that both tun0 and tun1 were created

  # Reboot again to go back to the most recent kernel and check with "ip link" that both tun0 and tun1 were not created.
  ---
  ProblemType: Bug
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Sep 29 12:52 seq
   crw-rw---- 1 root audio 116, 33 Sep 29 12:52 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu82.5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: [Errno 2] No such file or directory: 'fuser'
  CRDA: N/A
  CasperMD5CheckResult: unknown
  CloudArchitecture: x86_64
  CloudID: lxd
  CloudName: lxd
  CloudPlatform: lxd
  CloudSubPlatform: LXD socket API v. 1.0 (/dev/lxd/sock)
  DistroRelease: Ubuntu 22.04
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lspci: Error: [Errno 2] No such file or directory: 'lspci'
  Lspci-vt: Error: [Errno 2] No such file or directory: 'lspci'
  Lsusb: Error: [Errno 2] No such file or directory: 'lsusb'
  Lsusb-t: Error: [Errno 2] No such file or directory: 'lsusb'
  Lsusb-v: Error: [Errno 2] No such file or directory: 'lsusb'
  MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
  Package: linux (not installed)
  PciMultimedia:

  ProcEnviron:
   TERM=screen-256color
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB: 0 virtio_gpudrmfb
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-84-generic root=/dev/sda2 ro quiet splash console=tty1 console=ttyS0 vt.handoff=7
  ProcVersionSignature: Ubuntu 5.15.0-84.93-generic 5.15.116
  RelatedPackageVersions:
   linux-restricted-modules-5.15.0-84-generic N/A
   linux-backports-modules-5.15.0-84-generic  N/A
   linux-firmware                             N/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  Tags:  jammy uec-images
  Uname: Linux 5.15.0-84-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: N/A
  _MarkForUpload: True
  dmi.bios.date: 2/2/2022
  dmi.bios.release: 0.0
  dmi.bios.vendor: EDK II
  dmi.bios.version: unknown
  dmi.board.name: LXD
  dmi.board.vendor: Canonical Ltd.
  dmi.board.version: pc-q35-8.0
  dmi.chassis.type: 1
  dmi.chassis.vendor: QEMU
  dmi.chassis.version: pc-q35-8.0
  dmi.modalias: dmi:bvnEDKII:bvrunknown:bd2/2/2022:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-8.0:rvnCanonicalLtd.:rnLXD:rvrpc-q35-8.0:cvnQEMU:ct1:cvrpc-q35-8.0:sku:
  dmi.product.name: Standard PC (Q35 + ICH9, 2009)
  dmi.product.version: pc-q35-8.0
  dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2037667/+subscriptions

More information about the foundations-bugs mailing list