[Bug 1891215] Re: systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for every IPv6 RA received

[Dan Streetman]

> that was the atime

yep, really no way around updating the atime, as systemd of course needs
to access the file to check if the content is the same or not

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1891215

Title:
  systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for
  every IPv6 RA received

Status in systemd:
  Unknown
Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Bionic:
  New
Status in systemd source package in Focal:
  Fix Committed
Status in systemd source package in Groovy:
  Fix Committed
Status in systemd source package in Hirsute:
  Fix Committed

Bug description:
  [impact]

  networking changes, like RA events, can cause systemd-resolved to re-
  write the resolv.conf file, even if the contents didn't change,
  resulting in unnecessary increased amount of inotify events

  [test case]

  see original description for ipv6ra-related reproducer, or simple
  reproducer here:

  configure networkd with some config for (e.g.) eth0, but not a config
  that would result in /etc/resolv.conf changing when the interface goes
  up/down - for example, use static config with no DNS search domains.
  Then bring eth0 up/down while observing the md5sum (file content) does
  not change but the mtime does change.

  root at lp1891215-h:~# ip l set down dev eth0
  root at lp1891215-h:~# md5sum /etc/resolv.conf
  db23e80078515192c312e5f321ff0340  /etc/resolv.conf
  root at lp1891215-h:~# stat -t -L /etc/resolv.conf
  /etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238218 1625238216 1625238216 0 4096
  root at lp1891215-h:~# ip l set up dev eth0
  root at lp1891215-h:~# md5sum /etc/resolv.conf
  db23e80078515192c312e5f321ff0340  /etc/resolv.conf
  root at lp1891215-h:~# stat -t -L /etc/resolv.conf
  /etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238227 1625238226 1625238226 0 4096

  [regression potential]

  regressions would result in incorrect or missing data in the
  resolv.conf file, possibly resulting in dns failures or errors

  [scope]

  this is needed for h and eralier

  this is (potentially) fixed upstream by
  f3e1f00d03445911ee73729219cea88c8a70c612 which in first included in
  v248, so this is needed in hirsute and earlier

  [original description]

  # Issue description:

  On 2 Linode VMs that are used as lxd hosts, we noticed that
  /run/systemd/resolve/*resolv.conf were re-created quite frequently (~
  once per second). We noticed because of the log noise from lxd's
  dnsmasq instance using inotify to watch the target of /etc/resolv.conf
  (which points to the stub-resolv.conf in our case). This was (wrongly)
  reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until
  it became apparent it was more likely to be a problem with
  systemd(-resolved)?.

  The log noise is the observable problem that would be nice to see
  addressed:

    root at lxd02:~# uptime
     17:55:48 up  9:52,  1 user,  load average: 0.18, 0.11, 0.05
    root at lxd02:~# journalctl -b0 | grep -cF dnsmasq
    158609

  Upon further investigation, it seems that systemd-resolved re-creates
  the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is
  received.

  1) One can observe that by setting systemd-resolved's service in debug
  mode:

  $ sudo systemctl edit systemd-resolved

  and in the editor that is opened, add and save this content:

  [Service]
  Environment=SYSTEMD_LOG_LEVEL=debug

  then restart systemd-resolved and watch the logs scroll by with:

  $ journalctl -fu systemd-resolved

  3) In another terminal, watch the files be recreated with:

  watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf

  3) In yet another terminal, run a packet capture and watch "ICMP6,
  router advertisement" messages come by:

  sudo tcpdump -ni eth0 icmp6

  You will see that every time a RA packet comes in, resolved's journal
  will log this:

    Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message
  type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1
  interface=org.freedesktop.DBus.Properties member=PropertiesChanged
  cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-
  message=n/a

  And the stat monitoring terminal will blink to highlight the new inode
  and timestamps of the freshly replaced stub-resolv.conf file.

  # Additional information:

  root at lxd02:~# lsb_release -rd
  Description:	Ubuntu 20.04.1 LTS
  Release:	20.04

  root at lxd02:~# apt-cache policy systemd
  systemd:
    Installed: 245.4-4ubuntu3.2
    Candidate: 245.4-4ubuntu3.2
    Version table:
   *** 245.4-4ubuntu3.2 500
          500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       245.4-4ubuntu3 500
          500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

  root at lxd02:~# uname -a
  Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1891215/+subscriptions