[Bug 1902960] [NEW] Upgrade from 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to break DNS resolution in some cases

[David Lawson]

Public bug reported:

The systemd upgrade 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to have
broken DNS resolution across much of our Azure fleet earlier today.  We
ended up mitigating this by forcing reboots on the associated instances,
no combination of networkctl reload, reconfigure, systemctl daemon-
reexec, systemctl daemon-reload, netplan generate, netplan apply would
get resolvectl to have a DNS server again.  The main symptom appears to
have been systemd-networkd believing it wasn't managing the eth0
interfaces:


ubuntu at machine-1:~$ sudo networkctl                                                                                                                                                                             
IDX LINK TYPE     OPERATIONAL SETUP                                                                                                                                                                                
  1 lo   loopback carrier     unmanaged                                                                                                                                                                            
  2 eth0 ether    routable    unmanaged                                                                                                                                                                            
                                                                                                                                                                                                                   2 links listed.

Which eventually made them lose their DNS resolvers:

ubuntu at machine-1:~$ sudo resolvectl dns
Global:
Link 2 (eth0):

After rebooting, we see this behaving properly:

ubuntu at machine-1:~$ sudo networkctl list
IDX LINK TYPE     OPERATIONAL SETUP     
  1 lo   loopback carrier     unmanaged 
  2 eth0 ether    routable    configured

2 links listed.
ubuntu at machine-1:~$ sudo resolvectl dns
Global:
Link 2 (eth0): 168.63.129.16


This appears to be specifically linked to the upgrade, i.e. we were able to provoke the issue by upgrading the systemd package, so I suspect it's part of the packaging in the upgrade process.

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1902960

Title:
  Upgrade from 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to break DNS
  resolution in some cases

Status in systemd package in Ubuntu:
  New

Bug description:
  The systemd upgrade 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to
  have broken DNS resolution across much of our Azure fleet earlier
  today.  We ended up mitigating this by forcing reboots on the
  associated instances, no combination of networkctl reload,
  reconfigure, systemctl daemon-reexec, systemctl daemon-reload, netplan
  generate, netplan apply would get resolvectl to have a DNS server
  again.  The main symptom appears to have been systemd-networkd
  believing it wasn't managing the eth0 interfaces:

  
  ubuntu at machine-1:~$ sudo networkctl                                                                                                                                                                             
  IDX LINK TYPE     OPERATIONAL SETUP                                                                                                                                                                                
    1 lo   loopback carrier     unmanaged                                                                                                                                                                            
    2 eth0 ether    routable    unmanaged                                                                                                                                                                            
                                                                                                                                                                                                                     2 links listed.

  Which eventually made them lose their DNS resolvers:

  ubuntu at machine-1:~$ sudo resolvectl dns
  Global:
  Link 2 (eth0):

  After rebooting, we see this behaving properly:

  ubuntu at machine-1:~$ sudo networkctl list
  IDX LINK TYPE     OPERATIONAL SETUP     
    1 lo   loopback carrier     unmanaged 
    2 eth0 ether    routable    configured

  2 links listed.
  ubuntu at machine-1:~$ sudo resolvectl dns
  Global:
  Link 2 (eth0): 168.63.129.16

  
  This appears to be specifically linked to the upgrade, i.e. we were able to provoke the issue by upgrading the systemd package, so I suspect it's part of the packaging in the upgrade process.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1902960/+subscriptions