[Bug 1863873] Re: Systemd fails to configure bridged network in LXC container
Dan Streetman
ddstreet at canonical.com
Fri Feb 28 14:21:56 UTC 2020
> lxc.net.0.ipv4.gateway = 192.168.252.1
> lxc.net.0.ipv4.address = 192.168.252.171/32
> 3. inside the container, deactivate dhcp (dhcp4: false )in
`/etc/netplan/10-lxc.yaml`,
ok, this is your problem; you're misconfiguring the instance.
Unfortunately, you were just 'lucky' that it worked before, but this is
absolutely intended behavior of systemd: when you tell it to manage an
interface, it expects to control 100% of the interface's addresses and
routing (with the minor exception that newer systemd has added
'KeepConfiguration' config for specific situations where externally-
configured networking should be kept on the interface, such as some HA
setups).
Your problem here is that you're configuring the interface networking
*outside* networkd, but then also telling networkd it should manage the
interface. It's fine to use lxc to externally configure the interface
networking, but then inside the container you must make sure networkd
does not think it's managing eth0, which means you must completely
remove the 'eth0' section inside the netplan config yaml - if you leave
any 'eth0' config there at all, it will create a network file telling
networkd that it should manage eth0, even though it's not assigning any
ipv4 addresses (it does setup ipv6 link-local by default, however). The
file that netplan creates for the eth0 interface will be
/run/systemd/network/10-netplan-eth0.network.
You can see in the output of networkctl:
root at bionic:/# networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
29 eth0 ether degraded configured
since in your use case you do not use (or want!) networkd to manage
eth0, you want the 'SETUP' column to show 'unmanaged', not 'configured'.
What you need to do is, instead of simply changing the netplan eth0
dhcp4 configuration to false, you need to remove the eth0 section
completely. If that's the only section you have, you'll need to remove
the entire 'ethernets' section, or just remove the yaml file entirely
(or rename it to anything that doesn't end in '.yaml').
Then, when you 'netplan apply' or reboot, you should notice that the
/run/systemd/network/ file for eth0 is no longer created, and networkd
no longer thinks it manages eth0, and your externally-configured eth0
networking should work again.
I hope this works for you, and very sorry for the disruption of your
production environment.
** Changed in: systemd (Ubuntu)
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1863873
Title:
Systemd fails to configure bridged network in LXC container
Status in systemd package in Ubuntu:
Invalid
Bug description:
In all our LXC containers running Bionic Beaver, Eoan Ermine or Focal
Fossa, installing the latest systemd package results in losing network
configuration.
It is still possible to configure the network "by hand" with
/usr/sbin/ip, but of course, the configuration is lost at reboot.
An example is provided, followed by a complete procedure to reproduce
the issue.
Affected container distributions
================================
Xenial Xerus systemd 229-4ubuntu21.27: OK, not affected
Bionic Beaver systemd 237-3ubuntu10.38: OK, not affected
Bionic Beaver systemd 237-3ubuntu10.39: BUGGY
Disco Dingo systemd 240-6ubuntu5.8: OK, not affected
Eoan Ermine systemd 242-7ubuntu3.6: OK, not affected
Eoan Ermine systemd 242-7ubuntu3.7: BUGGY
Focal Fossa systemd 244.2-1ubuntu1: BUGGY
Affected hosts
==============
Debian Buster with default 4.19.0-6-amd64, custom 5.3.9, 5.4.8 or 5.4.13 kernel
Ubuntu 16.04 lxc 2.0.8-0ubuntu1~16.04.2 (https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1863873/comments/7)
Example
=======
Example host bridge configuration
---------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether 00:25:90:2b:f1:60 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br1 state DOWN group default qlen 1000
link/ether 00:25:90:2b:f1:61 brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:25:90:2b:f1:60 brd ff:ff:ff:ff:ff:ff
inet 192.168.252.24/24 brd 192.168.252.255 scope global br0
valid_lft forever preferred_lft forever
inet 192.168.193.203/24 brd 192.168.193.255 scope global br0:1
valid_lft forever preferred_lft forever
inet6 fe80::225:90ff:fe2b:f160/64 scope link
valid_lft forever preferred_lft forever
Example container network configuration
---------------------------------------
lxc.net.0.type = veth
lxc.net.0.veth.pair = vps525389
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.hwaddr = 02:00:00:52:53:89
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 192.168.252.1
lxc.net.0.ipv4.address = 192.168.252.177/32
Example steps to reproduce, inside the container
------------------------------------------------
root at vps525389:~# lsb_release -rd
Description: Ubuntu 18.04.4 LTS
Release: 18.04
root at vps525389:~# apt-cache policy systemd
systemd:
Installed: 237-3ubuntu10.38
Candidate: 237-3ubuntu10.39
Version table:
237-3ubuntu10.39 500
500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
*** 237-3ubuntu10.38 500
500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
100 /var/lib/dpkg/status
237-3ubuntu10 500
500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
root at vps525389:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
1958: eth0 at if1959: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:00:00:52:53:89 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.252.177/32 brd 255.255.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 xxxx:xxxx:x:xx::x:xxxx/128 scope global
valid_lft forever preferred_lft forever
inet6 xxxx::xx:xxxx:xxxx/64 scope link
valid_lft forever preferred_lft forever
root at vps525389:~# apt install systemd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libnss-systemd libpam-systemd libsystemd0
Suggested packages:
systemd-container policykit-1
The following packages will be upgraded:
libnss-systemd libpam-systemd libsystemd0 systemd
4 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 3330 kB of archives.
After this operation, 7168 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnss-systemd amd64 237-3ubuntu10.39 [104 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-systemd amd64 237-3ubuntu10.39 [107 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd amd64 237-3ubuntu10.39 [2912 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd0 amd64 237-3ubuntu10.39 [206 kB]
Fetched 3330 kB in 3s (1274 kB/s)
(Reading database ... 18195 files and directories currently installed.)
Preparing to unpack .../libnss-systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking libnss-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../libpam-systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking libpam-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking systemd (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../libsystemd0_237-3ubuntu10.39_amd64.deb ...
Unpacking libsystemd0:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Setting up libsystemd0:amd64 (237-3ubuntu10.39) ...
Setting up systemd (237-3ubuntu10.39) ...
Setting up libnss-systemd:amd64 (237-3ubuntu10.39) ...
Setting up libpam-systemd:amd64 (237-3ubuntu10.39) ...
Processing triggers for dbus (1.12.2-1ubuntu1.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
root at vps525389:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
1958: eth0 at if1959: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:00:00:52:53:89 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::ff:fe52:5389/64 scope link
valid_lft forever preferred_lft forever
Complete procedure to reproduce the issue
=========================================
It is here assumed that there is a DHCP server available elsewhere on
the network.
Set-up
------
1. Install an amd64 Debian Buster (default network install),
2. create a bridge on the host with a static IP and deactivate DHCP, in `/etc/network/interfaces`,
```
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
## The primary network interface
#allow-hotplug ens18
#iface ens18 inet dhcp
## This is an autoconfigured IPv6 interface
#iface ens18 inet6 auto
iface ens18 inet manual
auto br0
iface br0 inet static
address 192.168.1.168
netmask 255.255.255.0
gateway 192.168.1.220
bridge_ports ens18
```
3. reboot the host,
```bash
reboot
```
4. install lxc and create a bionic amd64 container,
```bash
apt install lxc
lxc-create -t download -n bionic
```
5. on the host, modify the network configuration of the container to use the bridge with a static IP in `/var/lib/lxc/bionic/config`,
```
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# Template script checksum (SHA-1): 273c51343604eb85f7e294c8da0a5eb769d648f3
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
# For Ubuntu 14.04
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.arch = linux64
# Container specific configuration
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
lxc.rootfs.path = dir:/var/lib/lxc/bionic/rootfs
lxc.uts.name = bionic
## Network configuration
#lxc.net.0.type = empty
# Network configuration
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 192.168.1.220
lxc.net.0.ipv4.address = 192.168.1.169/32
```
6. inside the container, install the systemd packages without the bug, and deactivate DHCP in `/etc/netplan/10-lxc.yaml`,
```bash
lxc-start -n bionic
lxc-attach -n bionic
apt install systemd=237-3ubuntu10.38 libsystemd0=237-3ubuntu10.38 libnss-systemd=237-3ubuntu10.38 libpam-systemd=237-3ubuntu10.38
sed -i 's/true/false/' /etc/netplan/10-lxc.yaml
exit
```
7. stop the container.
```bash
lxc-stop -n bionic
```
Let’s do it
-----------
1. Start the container and check the IP config, which should be ok,
```bash
lxc-start -n bionic
lxc-attach -n bionic
ip a
```
2. upgrade the system and check the IP config, the static IP is gone.
```bash
apt upgrade
ip a
exit
```
If systemd is downgraded again to 237-3ubuntu10.38, the IP is back at
the next reboot of the container.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1863873/+subscriptions
More information about the foundations-bugs
mailing list