[Bug 1918141] Re: nfs-server.service needs name resolution and network online

Thu Mar 25 13:58:24 UTC 2021

> This isn't a Ubuntu decision - upstream of the package/functionality
goes this way, so if challenged one should challenge it there.

I disagree. This is entirely a question of how we integrate between
services. The distribution is where that integration happens, and users
expect consistency of integration from the distribution. This is the
point of packaging. Where there is a conflict, consistency here should
be across different packages in the distribution, rather than between
individual packages and their upstreams. So we should figure out what
best practice should be across our package archive first.

> I remember a few of the old incarnations of this discussion, but it
already has become a (use)case-by-case decision throughout various
packages and services.

Even more reason to seek to develop a best practice then, rather than
continue to exacerbate a problem that might already exist.

If it's decided that network-online.target should normally be provided
by default by any package that provides a service that (might be
configured/always) depend on a network service, then that's fine, the
change in Hirsute will be correct, and we can continue to change
packages when this issue is inevitably brought up again. But that should
be a deliberate choice we make, not one made on a piecemeal basis only
considering one use case at a time. Because consistency across the
distribution matters for user expectations and a good user experience.

To be clear, I'm not presupposing the answer here; I just want this
properly considered and a consistent decision made that considers the
distribution as a whole.

> For this I honestly see no risks of regressions for nfs-server.

As Christian noted, there's a difference here in changing this for the
future, and changing this in existing stable releases. There's a
relatively low downside if we don't change this in existing stable
releases. Users aren't blocked: they just need to configure their
dependencies according to their local requirements. This is relatively
easy and already explained here. The downsides in not making the change
is: Users configuring a service for the first time will have to find an
issue and discover the bug. The downside in making the change is: We're
changing behaviour for users who've already configured the service,
which might break them depending on their specific local setup.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1918141

Title:
  nfs-server.service needs name resolution and network online

Status in nfs-utils package in Ubuntu:
  Fix Released
Status in nfs-utils source package in Bionic:
  Confirmed
Status in nfs-utils source package in Focal:
  Confirmed
Status in nfs-utils source package in Groovy:
  Confirmed
Status in nfs-utils source package in Hirsute:
  Fix Released

Bug description:
  [ Impact ]

  nfs-server.service (part of the nfs-utils package) has insufficient
  dependencies to start correctly in a setting where the NFS exports
  list contains DNS hostnames (not in local hosts file) or netgroups
  served via network (for example sssd).

  This issue can negatively impact users that, for example, cannot
  easily resort to using IP addresses directly in the /etc/exports file,
  and need to use hostnames instead in order to be able to mount NFS
  shares over the network.

  [ Test Case ]

  You can reproduce the issue by following the instructions below.
  Adjust the Ubuntu release on "images:ubuntu/hirsute" accordingly.

  $ lxc launch images:ubuntu/hirsute nfs-utils-bug1918141 --vm
  $ lxc shell nfs-utils-bug1918141
  # apt update
  # apt install -y nfs-kernel-server
  # mkdir /testshare
  # cat >> /etc/exports << EOF
  /testshare ubuntu.com(ro,sync,no_subtree_check)
  EOF
  # systemctl edit --full systemd-networkd.service

  You will have to:

  - Comment out the "Wants=" line.
  - Remove "network.target" from the "Before=" line.
  - Add a new "After=fakenet.service" line.
  - Add a new "Wants=fakenet.service" line.
  - Add the following line in the "[Service]" section:

    ExecStartPre=/bin/sleep 10

  # systemctl edit --force --full fakenet.service

  This new service file will contain:

  [Unit]
  Description=Fake network.target
  DefaultDependencies=no
  After=systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  [Service]
  Type=oneshot
  ExecStart=/bin/true

  [Install]
  WantedBy=multi-user.target

  # systemctl enable fakenet.service
  # systemctl disable systemd-resolved
  # systemctl mask systemd-resolved
  # rm /etc/resolv.conf
  # cat > /etc/resolv.conf << EOF
  nameserver 8.8.8.8
  EOF
  # reboot

  After a few seconds, you can enter the VM again:

  $ lxc shell nfs-utils-bug1918141
  # sleep 20 && systemctl status nfs-server.service
  ● nfs-server.service - NFS server and services
       Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
       Active: active (exited) since Tue 2021-03-16 20:51:42 UTC; 39s ago
      Process: 310 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
      Process: 311 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
     Main PID: 311 (code=exited, status=0/SUCCESS)

  Mar 16 20:51:41 nfs-utils-bug1918141 systemd[1]: Starting NFS server and services...
  Mar 16 20:51:41 nfs-utils-bug1918141 exportfs[310]: exportfs: Failed to resolve ubuntu.com
  Mar 16 20:51:41 nfs-utils-bug1918141 exportfs[310]: exportfs: Failed to resolve ubuntu.com
  Mar 16 20:51:42 nfs-utils-bug1918141 systemd[1]: Finished NFS server and services.

  As you can see, the service was started but exportfs failed to resolve
  the hostname.

  [Where problems could occur]

  This fix has been applied upstream for quite a while now, and is even
  part of a release, so it has been extensively tested by users and
  other distributions that have more recent nfs-utils.  I cannot easily
  envision problems with the proposed changes but:

  * If the user has manually modified the nfs-server.service file
  without taking proper precautions (i.e., editin the file directly on
  /lib/systemd/system/ instead of using "systemctl edit"), then he might
  experience a conflict when installing a new version of the file, and
  his modifications will be lost.  However, this falls under the "local
  configuration issue", in my opinion.

  [ Original Description ]

  nfs-server.service has insufficient dependencies to start correctly in
  a setting where the nfs exports list contains DNS host names (not in
  local hosts file) or netgroups served via network (for example sssd).

  Typical failures listed by systemctl status nfs-server.service are:

  Mar 08 14:16:52 server.example.com exportfs[844]: exportfs: Failed to resolve client1.example.com
  Mar 08 14:16:52 server.example.com exportfs[844]: exportfs: Failed to resolve client2.example.com

  Our workaround is to add the appropriate dependecies in
  /etc/systemd/system/nfs-server.service.d/dependencies.conf like so:

  ----------------------8<----------------------------
  [Unit]
  # nfs-server.service runs exportfs on startup, thus we need to be able to
  # do host and netgroup lookups which requires network to be online.
  After=network-online.target nss-lookup.target nss-user-lookup.target
  ----------------------8<----------------------------

  While nfs-server.service do depend on network.target, that only means
  that the network has been configured. On physical hardware it can take
  significantly longer for the network to come online (8+ seconds for
  our 10G NICs). Also note that we configure static IP:s via systemd-
  networkd, things might behave differently when using DHCP, network-
  manager etc. In any case, depending on network.target is almost always
  wrong, and network-online.target is usually the right one.

  nss-lookup.target is needed to ensure that DNS resolution works, and
  nss-user-lookup.target is the best approximation to ensure that
  netgroup resolution via sssd or equivalent works. Usually things "just
  works" even without these dependencies, but to ensure correct startup
  they should be present.

  It should be noted that this seems to once have been fixed in Ubuntu,
  but has been lost along the way for quite some time. When googling I
  find for example https://code.launchpad.net/~ubuntu-
  branches/ubuntu/wily/nfs-utils/wily-201507271018/+merge/265946 that
  fixes the network.target vs network-online.target dependency, but it
  has since been lost in the wind it seems.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1918141/+subscriptions