Ubuntu 18.04: systemd-resolved -- crashing or failing to start properly...

Tue Jun 4 20:29:05 UTC 2019

At Mon,  3 Jun 2019 13:22:38 -0400 (EDT) "Ubuntu user technical support,  not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:

> 
> At Sat, 1 Jun 2019 12:31:11 +0200 "Ubuntu user technical support,  not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:
> 
> > 
> > On Sat, Jun 1, 2019 at 2:29 AM Robert Heller <heller at deepsoft.com> wrote:
> > >
> > > I have an *intermittent* problem with systemd-resolved. It is
> > > either crashing or fails to start properly. The sympton is that
> > > when I go to do an update (eg apt-get update), I get name resolver
> > > errors to the *local* (LAN only) mirror host. Doing a "systemctl
> > > restart systemd-resolved" cures the problem. I am not sure exactly
> > > what is going on, but it appears that for some reason
> > > systemd-resolved is not seeing (or using) the *local* LAN DNS
> > > server (bind9 running on a CentOS 6 server). For *all* of the
> > > Ubuntu 18.04 machines, *I* would *rather* not have systemd-resolved
> > > running at all and just have /etc/resolve.conf reference the CentOS
> > > 6 server and bind9 running there. (I have configured things on the
> > > DRBL server to do just that.) It is already a cacheing DNS server
> > > for the whole LAN and provides local DNS for the LAN, including
> > > intranet services (NFS server, LDAP server, LAN-only web services,
> > > etc.). The problem is an issue on the laptops and one workstation,
> > > and only really affects LAN-only web services (which includes the
> > > local repo mirror) -- the NFS mounts and LDAP configuration use the
> > > hard IP address of the server(s) in question.
> > 
> > [ It's "/etc/resolv.conf" not "/etc/resolve.conf" but I assume that
> > this is an email typo not an on-filesystem one ]
> 
> Yes...
> 
> > 
> > Are you feeding "systemd-resolved" the LAN DNS server? What's the
> > output of "resolvectl"?
> > 
> 
> systemd-resolved should be get parameters via DHCP from the DHCP server:
> 
> (From the dhcpd.conf file on the CentOS 6 machine:
> 
>   option domain-name "wendellfreelibrary.org";
>   option domain-name-servers 192.168.1.251, 8.8.8.8, 8.8.4.4;
>   option broadcast-address 192.168.1.255;
>   option routers 192.168.1.251;
>         
> )
> 
> There is no resolvectl program.  It does not appear to be installed on my 
> Ubuntu 18.05 system.

OK, what *exactly* does this mean:

ub180464% sudo systemctl status systemd-resolved
[sudo] password for heller: 
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vend
   Active: active (running) since Mon 2019-06-03 13:18:59 EDT; 1 day 2h ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-c
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-
 Main PID: 574 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4658)
   CGroup: /system.slice/systemd-resolved.service
           └─574 /lib/systemd/systemd-resolved

Jun 03 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 03 16:00:02 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 13:24:22 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
Jun 04 16:16:44 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
ub180464% dig -x 192.168.1.251

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20218
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;251.1.168.192.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
251.1.168.192.in-addr.arpa. 0   IN      PTR     _gateway.

;; Query time: 17 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Jun 04 16:17:12 EDT 2019
;; MSG SIZE  rcvd: 77

Note:

ub180464% dig -x 192.168.1.251 @192.168.1.251

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251 @192.168.1.251
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29665
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;251.1.168.192.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
251.1.168.192.in-addr.arpa. 86400 IN    PTR     newserver.wendellfreelibrary.org.

;; AUTHORITY SECTION:
1.168.192.in-addr.arpa. 86400   IN      NS      newserver.wendellfreelibrary.org.

;; ADDITIONAL SECTION:
newserver.wendellfreelibrary.org. 86400 IN A    192.168.1.251

;; Query time: 0 msec
;; SERVER: 192.168.1.251#53(192.168.1.251)
;; WHEN: Tue Jun 04 16:19:12 EDT 2019
;; MSG SIZE  rcvd: 131

And:

ub180464% less -X /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=192.168.1.251
#FallbackDNS=
Domains=wendellfreelibrary.org
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

Restarting systemd-resolved solves the problem:

ub180464% sudo systemctl restart systemd-resolved
ub180464% dig -x 192.168.1.251

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12328
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;251.1.168.192.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
251.1.168.192.in-addr.arpa. 86400 IN    PTR     newserver.wendellfreelibrary.org.

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Jun 04 16:20:53 EDT 2019
;; MSG SIZE  rcvd: 101

Question: Do I really need to set up a cronjob to periotically restart
systemd-resolved? The reason this matters is because I am getting messages
from AMANDA that this silly Ubuntu VM can't ack the AMANDA client check
because there is no entry for "_gateway", which is because systemd-resolved is
going south randomly and not doing its job. Oh, this also affects updates
since I use a local repo mirror and that depends on sane DNS lookups (which
systemd-resolved is not reliably giving me).

I'm of more than 1/2 a mind to nuke systemd-resolved and hard-wire 
/etc/resolv.conf to just use my stable, working, no-nonsense Bind9 DNS server 
running on my dead-reliable CentOS 6 system.

> 
> 

-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services