[Bug 1590799] Re: nfs-kernel-server does not start because of dependency failure
Rafael David Tinoco
rafael.tinoco at canonical.com
Thu Mar 2 02:36:57 UTC 2017
** Description changed:
+ [Impact]
+
+ * nfs-mountd doesn't get started because of a race condition happening when rpcbind.socket is not specified as a needed service for it to start.
+ * nfs-server using rpcbind.target instead of using rpcbind.socket. Target should not be used (Comment #24)
+
+ [Test Case]
+
+ * Install nfs-kernel-server inside a xenial lxc guest and restart it until nfs-mountd doesn't start complaining on rpc error.
+ * Comment #25
+
+ [Regression Potential]
+
+ * Cons: Systemd dependencies could brake for nfs-server and nfs-mountd.
+ * Pros: Patches have been accepted upstream (and tested).
+
+ [Other Info]
+
+ # Original Bug Description
+
Immediately after boot:
root at feynmann:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
- Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
- Active: inactive (dead)
+ Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
+ Active: inactive (dead)
Jun 09 14:35:47 feynmann systemd[1]: Dependency failed for NFS server and services.
Jun 09 14:35:47 feynmann systemd[1]: nfs-server.service: Job nfs-server.service/start failed
-
- root at feynmann:~# systemctl status nfs-mountd.service
+
+ root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
- Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
- Active: failed (Result: exit-code) since Thu 2016-06-09 14:35:47 BST; 7min ago
- Process: 1321 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)
+ Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
+ Active: failed (Result: exit-code) since Thu 2016-06-09 14:35:47 BST; 7min ago
+ Process: 1321 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)
Jun 09 14:35:47 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 14:35:47 feynmann rpc.mountd[1321]: mountd: could not create listeners
- Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Control process exited, code=exited
+ Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Control process exited, code=exited
Jun 09 14:35:47 feynmann systemd[1]: Failed to start NFS Mount Daemon.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Unit entered failed state.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.
root at feynmann:~# systemctl list-dependencies nfs-kernel-server
nfs-kernel-server.service
● ├─auth-rpcgss-module.service
● ├─nfs-config.service
● ├─nfs-idmapd.service
● ├─nfs-mountd.service
● ├─proc-fs-nfsd.mount
● ├─rpc-svcgssd.service
● ├─system.slice
● ├─network.target
● └─rpcbind.target
● └─rpcbind.service
- root at feynmann:~# systemctl list-dependencies nfs-mountd.service
+ root at feynmann:~# systemctl list-dependencies nfs-mountd.service
nfs-mountd.service
● ├─nfs-config.service
● ├─nfs-server.service
● ├─proc-fs-nfsd.mount
● └─system.slice
- root at feynmann:~#
+ root at feynmann:~#
root at feynmann:~# lsb_release -rd
Description: Ubuntu 16.04 LTS
Release: 16.04
root at feynmann:~# apt-cache policy nfs-kernel-server
nfs-kernel-server:
- Installed: 1:1.2.8-9ubuntu12
- Candidate: 1:1.2.8-9ubuntu12
- Version table:
- *** 1:1.2.8-9ubuntu12 500
- 500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
- 100 /var/lib/dpkg/status
+ Installed: 1:1.2.8-9ubuntu12
+ Candidate: 1:1.2.8-9ubuntu12
+ Version table:
+ *** 1:1.2.8-9ubuntu12 500
+ 500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
+ 100 /var/lib/dpkg/status
Additional comments:
- 1. There seems to be a circular dependency between nfs-mountd and nfs-kernel-server
- 2. I can get it working by changing the AFter,Requires in /lib/ssystemd/system/nfs-{mountd|server}.service files. I have managed to get nfs-kernel-server to start but not nfs-mountd.
- 3. /usr/lib/systemd/scripts/nfs-utils_env.sh references /etc/sysconfig/nfs which is Centos/RedHat location of this file. Also /etc/default/nfs does not exist. (possibly unrelated to this bug)
- 4. A file "/lib/systemd/system/-.slice" exists. this file prevents execution of 'ls *' or 'grep xxx *' commands in that directory. I am unsure whether this is intended by the systemd developers but it is unfriendly when investigating this bug.
+ 1. There seems to be a circular dependency between nfs-mountd and nfs-kernel-server
+ 2. I can get it working by changing the AFter,Requires in /lib/ssystemd/system/nfs-{mountd|server}.service files. I have managed to get nfs-kernel-server to start but not nfs-mountd.
+ 3. /usr/lib/systemd/scripts/nfs-utils_env.sh references /etc/sysconfig/nfs which is Centos/RedHat location of this file. Also /etc/default/nfs does not exist. (possibly unrelated to this bug)
+ 4. A file "/lib/systemd/system/-.slice" exists. this file prevents execution of 'ls *' or 'grep xxx *' commands in that directory. I am unsure whether this is intended by the systemd developers but it is unfriendly when investigating this bug.
Attempted solution:
- 1. Edit /lib/systemd/system/nfs-server.service (original lines are
+ 1. Edit /lib/systemd/system/nfs-server.service (original lines are
commented out:
[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires=network.target proc-fs-nfsd.mount rpcbind.target
# Requires=nfs-mountd.service
Wants=nfs-idmapd.service
-
+
After=local-fs.target
#After=network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
- After=network.target proc-fs-nfsd.mount rpcbind.target
+ After=network.target proc-fs-nfsd.mount rpcbind.target
After=nfs-idmapd.service rpc-statd.service
#Before=rpc-statd-notify.service
Before=nfs-mountd.service rpc-statd-notify.service
...
followed by a systemctl daemon-reload and a reboot.
This results in nfs-kernel-server starting correctly but nfs-mountd not
so. However starting nfs-mountd manually after reboot is successful:
- root at feynmann:~# systemctl status nfs-kernel-server.service
+ root at feynmann:~# systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
- Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
- Active: active (exited) since Thu 2016-06-09 15:07:23 BST; 1min 25s ago
- Process: 1391 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
- Process: 1384 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
- Main PID: 1391 (code=exited, status=0/SUCCESS)
- Tasks: 0
- Memory: 0B
- CPU: 0
- CGroup: /system.slice/nfs-server.service
+ Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
+ Active: active (exited) since Thu 2016-06-09 15:07:23 BST; 1min 25s ago
+ Process: 1391 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
+ Process: 1384 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
+ Main PID: 1391 (code=exited, status=0/SUCCESS)
+ Tasks: 0
+ Memory: 0B
+ CPU: 0
+ CGroup: /system.slice/nfs-server.service
Jun 09 15:07:23 feynmann systemd[1]: Starting NFS server and services...
Jun 09 15:07:23 feynmann systemd[1]: Started NFS server and services.
- root at feynmann:~# systemctl status nfs-mountd.service
+ root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
- Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
- Active: inactive (dead)
+ Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
+ Active: inactive (dead)
root at feynmann:~# systemctl start nfs-mountd.service
-
- root at feynmann:~# systemctl status nfs-mountd.service
+
+ root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
- Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
- Active: active (running) since Thu 2016-06-09 15:09:02 BST; 3s ago
- Process: 2044 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
- Main PID: 2046 (rpc.mountd)
- Tasks: 1
- Memory: 904.0K
- CPU: 12ms
- CGroup: /system.slice/nfs-mountd.service
- └─2046 /usr/sbin/rpc.mountd --manage-gids
+ Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
+ Active: active (running) since Thu 2016-06-09 15:09:02 BST; 3s ago
+ Process: 2044 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
+ Main PID: 2046 (rpc.mountd)
+ Tasks: 1
+ Memory: 904.0K
+ CPU: 12ms
+ CGroup: /system.slice/nfs-mountd.service
+ └─2046 /usr/sbin/rpc.mountd --manage-gids
Jun 09 15:09:02 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 15:09:02 feynmann rpc.mountd[2046]: Version 1.2.8 starting
Jun 09 15:09:02 feynmann systemd[1]: Started NFS Mount Daemon.
Enabling nfs-mountd.service (systemctl enable nfs-mountd.service) has no
effect in this case.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nfs-kernel-server 1:1.2.8-9ubuntu12 [modified: lib/systemd/system/nfs-server.service]
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Thu Jun 9 14:38:58 2016
InstallationDate: Installed on 2016-06-08 (1 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
ProcEnviron:
- SHELL=/bin/bash
- TERM=linux
- PATH=(custom, no user)
- LANG=en_GB.UTF-8
- LANGUAGE=en_GB:en
+ SHELL=/bin/bash
+ TERM=linux
+ PATH=(custom, no user)
+ LANG=en_GB.UTF-8
+ LANGUAGE=en_GB:en
SourcePackage: nfs-utils
UpgradeStatus: No upgrade log present (probably fresh install)
** Changed in: nfs-utils (Ubuntu)
Importance: High => Medium
** Changed in: nfs-utils (Ubuntu Xenial)
Importance: High => Medium
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1590799
Title:
nfs-kernel-server does not start because of dependency failure
Status in nfs-utils package in Ubuntu:
In Progress
Status in nfs-utils source package in Xenial:
In Progress
Bug description:
[Impact]
* nfs-mountd doesn't get started because of a race condition happening when rpcbind.socket is not specified as a needed service for it to start.
* nfs-server using rpcbind.target instead of using rpcbind.socket. Target should not be used (Comment #24)
[Test Case]
* Install nfs-kernel-server inside a xenial lxc guest and restart it until nfs-mountd doesn't start complaining on rpc error.
* Comment #25
[Regression Potential]
* Cons: Systemd dependencies could brake for nfs-server and nfs-mountd.
* Pros: Patches have been accepted upstream (and tested).
[Other Info]
# Original Bug Description
Immediately after boot:
root at feynmann:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Jun 09 14:35:47 feynmann systemd[1]: Dependency failed for NFS server and services.
Jun 09 14:35:47 feynmann systemd[1]: nfs-server.service: Job nfs-server.service/start failed
root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2016-06-09 14:35:47 BST; 7min ago
Process: 1321 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)
Jun 09 14:35:47 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 14:35:47 feynmann rpc.mountd[1321]: mountd: could not create listeners
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Control process exited, code=exited
Jun 09 14:35:47 feynmann systemd[1]: Failed to start NFS Mount Daemon.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Unit entered failed state.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.
root at feynmann:~# systemctl list-dependencies nfs-kernel-server
nfs-kernel-server.service
● ├─auth-rpcgss-module.service
● ├─nfs-config.service
● ├─nfs-idmapd.service
● ├─nfs-mountd.service
● ├─proc-fs-nfsd.mount
● ├─rpc-svcgssd.service
● ├─system.slice
● ├─network.target
● └─rpcbind.target
● └─rpcbind.service
root at feynmann:~# systemctl list-dependencies nfs-mountd.service
nfs-mountd.service
● ├─nfs-config.service
● ├─nfs-server.service
● ├─proc-fs-nfsd.mount
● └─system.slice
root at feynmann:~#
root at feynmann:~# lsb_release -rd
Description: Ubuntu 16.04 LTS
Release: 16.04
root at feynmann:~# apt-cache policy nfs-kernel-server
nfs-kernel-server:
Installed: 1:1.2.8-9ubuntu12
Candidate: 1:1.2.8-9ubuntu12
Version table:
*** 1:1.2.8-9ubuntu12 500
500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
Additional comments:
1. There seems to be a circular dependency between nfs-mountd and nfs-kernel-server
2. I can get it working by changing the AFter,Requires in /lib/ssystemd/system/nfs-{mountd|server}.service files. I have managed to get nfs-kernel-server to start but not nfs-mountd.
3. /usr/lib/systemd/scripts/nfs-utils_env.sh references /etc/sysconfig/nfs which is Centos/RedHat location of this file. Also /etc/default/nfs does not exist. (possibly unrelated to this bug)
4. A file "/lib/systemd/system/-.slice" exists. this file prevents execution of 'ls *' or 'grep xxx *' commands in that directory. I am unsure whether this is intended by the systemd developers but it is unfriendly when investigating this bug.
Attempted solution:
1. Edit /lib/systemd/system/nfs-server.service (original lines are
commented out:
[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires=network.target proc-fs-nfsd.mount rpcbind.target
# Requires=nfs-mountd.service
Wants=nfs-idmapd.service
After=local-fs.target
#After=network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
After=network.target proc-fs-nfsd.mount rpcbind.target
After=nfs-idmapd.service rpc-statd.service
#Before=rpc-statd-notify.service
Before=nfs-mountd.service rpc-statd-notify.service
...
followed by a systemctl daemon-reload and a reboot.
This results in nfs-kernel-server starting correctly but nfs-mountd
not so. However starting nfs-mountd manually after reboot is
successful:
root at feynmann:~# systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2016-06-09 15:07:23 BST; 1min 25s ago
Process: 1391 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 1384 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 1391 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CPU: 0
CGroup: /system.slice/nfs-server.service
Jun 09 15:07:23 feynmann systemd[1]: Starting NFS server and services...
Jun 09 15:07:23 feynmann systemd[1]: Started NFS server and services.
root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
Active: inactive (dead)
root at feynmann:~# systemctl start nfs-mountd.service
root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
Active: active (running) since Thu 2016-06-09 15:09:02 BST; 3s ago
Process: 2044 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
Main PID: 2046 (rpc.mountd)
Tasks: 1
Memory: 904.0K
CPU: 12ms
CGroup: /system.slice/nfs-mountd.service
└─2046 /usr/sbin/rpc.mountd --manage-gids
Jun 09 15:09:02 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 15:09:02 feynmann rpc.mountd[2046]: Version 1.2.8 starting
Jun 09 15:09:02 feynmann systemd[1]: Started NFS Mount Daemon.
Enabling nfs-mountd.service (systemctl enable nfs-mountd.service) has
no effect in this case.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nfs-kernel-server 1:1.2.8-9ubuntu12 [modified: lib/systemd/system/nfs-server.service]
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Thu Jun 9 14:38:58 2016
InstallationDate: Installed on 2016-06-08 (1 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
ProcEnviron:
SHELL=/bin/bash
TERM=linux
PATH=(custom, no user)
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
SourcePackage: nfs-utils
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1590799/+subscriptions
More information about the foundations-bugs
mailing list