[Bug 2091737] Re: Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from 1:1.3.4-2.5ubuntu3)
Ram Sun
2091737 at bugs.launchpad.net
Fri Dec 20 22:56:57 UTC 2024
Andreas,
I was able to pick the packages built in
https://launchpad.net/~ahasenack/+archive/ubuntu/nfs-slow-
mounts/+build/29866486 and test out the experiment.
ubuntu at vm-machine-s82gbm-3ojtydo:~/nfs_ppa$ dpkg --list | grep ppa1
ii libnfsidmap-dev 1:2.6.1-1ubuntu1.3~ppa1 amd64 header files and docs for libnfsidmap
ii libnfsidmap1:amd64 1:2.6.1-1ubuntu1.3~ppa1 amd64 NFS idmapping library
ii nfs-common 1:2.6.1-1ubuntu1.3~ppa1 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:2.6.1-1ubuntu1.3~ppa1 amd64 support for NFS kernel server
The package is working as expected and I don't see slow mounts anymore.
With 40 exports, the same command to perform a single mount consistently
takes 1-2s
Mounting /sd/mount/oracle/jammy_patch/test1 to
/tmp/test/jammy_patch/test1
real 0m1.328s
user 0m0.002s
sys 0m0.005s
Mounting /sd/mount/oracle/jammy_patch/test1 to
/tmp/test/jammy_patch/test1
real 0m1.183s
user 0m0.003s
sys 0m0.004s
Even though the custom FUSE filesystem made the issue more evident, we
can still observe the increase in teh number of statx calls and analysis
shared in https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/2091737/comments/6 which can be reproducible on any
filesystem and can be done by anyone.
With regards to the official test plan, would it be possible for the
Ubuntu(jammy) team to check with Neil Brown or nfs-utils maintainers to
confirm since the patch has indeed been committed to upstream nfs-utils
versions and subsequent releases?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/2091737
Title:
Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from
1:1.3.4-2.5ubuntu3)
Status in nfs-utils package in Ubuntu:
Confirmed
Status in nfs-utils source package in Jammy:
In Progress
Bug description:
We are experiencing significant delays (60+ seconds) when mounting NFS
directories after exporting 40 directories. This issue is observed
with nfs-utils 1:2.6.1-1ubuntu1.2 on an Ubuntu Jammy system. We have
40 exports and are attempting to mount via NFS onto /sd/mount paths.
The /sd filesystem is a custom, in-house filesystem built on top of
FUSE. Prior to using nfs-utils 1:2.6.1-1ubuntu1.2, we were using nfs-
utils 1:1.3.4-2.5ubuntu3(Ubuntu 20.04 focal) where mounts were almost
immediate, taking less than 10 seconds.
Applying this patch from upstream nfs-utils resolves the issue:
https://patchwork.kernel.org/project/linux-
nfs/patch/20231023021052.5258-4-neilb at suse.de/
Additionally, the above problem is not reproducible using nfs-utils
1:2.6.4-3ubuntu5(Ubuntu 24.04 Noble Numbat)
System Information:
OS: Ubuntu 22.04 LTS Jammy
nfs-utils Version: 1:2.6.1-1ubuntu1.2
Kernel: 5.15.0-117
Steps to reproduce internally:
Create 40 directories in /sd/mount and export them.
Execute showmount -e to list the exports.
Attempt to mount one of the NFS directories and note the time taken.
Actual Outcome:
Mounting a single directory takes more than 60 seconds consistently. While attempting to perform multiple mounts, some mounts can take more than 300s.
Expected Outcome:
Mounting should be significantly faster.
Logs and Outputs:
Export List:
ubuntu at vm-machine-s82gbm-3ojtydo:~$ showmount -e
Export list for vm-machine-s82gbm-3ojtydo:
/sd/mount/oracle/jammy_patch/test40 *
/sd/mount/oracle/jammy_patch/test39 *
/sd/mount/oracle/jammy_patch/test38 *
/sd/mount/oracle/jammy_patch/test37 *
/sd/mount/oracle/jammy_patch/test36 *
/sd/mount/oracle/jammy_patch/test35 *
/sd/mount/oracle/jammy_patch/test34 *
/sd/mount/oracle/jammy_patch/test33 *
/sd/mount/oracle/jammy_patch/test32 *
/sd/mount/oracle/jammy_patch/test31 *
/sd/mount/oracle/jammy_patch/test30 *
/sd/mount/oracle/jammy_patch/test29 *
/sd/mount/oracle/jammy_patch/test28 *
/sd/mount/oracle/jammy_patch/test27 *
/sd/mount/oracle/jammy_patch/test26 *
/sd/mount/oracle/jammy_patch/test25 *
/sd/mount/oracle/jammy_patch/test24 *
/sd/mount/oracle/jammy_patch/test23 *
/sd/mount/oracle/jammy_patch/test22 *
/sd/mount/oracle/jammy_patch/test21 *
/sd/mount/oracle/jammy_patch/test20 *
/sd/mount/oracle/jammy_patch/test19 *
/sd/mount/oracle/jammy_patch/test18 *
/sd/mount/oracle/jammy_patch/test17 *
/sd/mount/oracle/jammy_patch/test16 *
/sd/mount/oracle/jammy_patch/test15 *
/sd/mount/oracle/jammy_patch/test14 *
/sd/mount/oracle/jammy_patch/test13 *
/sd/mount/oracle/jammy_patch/test12 *
/sd/mount/oracle/jammy_patch/test11 *
/sd/mount/oracle/jammy_patch/test10 *
/sd/mount/oracle/jammy_patch/test9 *
/sd/mount/oracle/jammy_patch/test8 *
/sd/mount/oracle/jammy_patch/test7 *
/sd/mount/oracle/jammy_patch/test6 *
/sd/mount/oracle/jammy_patch/test5 *
/sd/mount/oracle/jammy_patch/test4 *
/sd/mount/oracle/jammy_patch/test3 *
/sd/mount/oracle/jammy_patch/test2 *
/sd/mount/oracle/jammy_patch/test1 *
/sd/mount ::1
Mount command:
time sudo mount -o rw,hard,rsize=1048576,wsize=1048576,vers=3,nointr,tcp,sync ${SERVER_IP}:/sd/mount/oracle/jammy_patch/test1 /tmp/test/test1
Patch Applied:
Fix commit: https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/
diff --git a/utils/exportd/exportd.c b/utils/exportd/exportd.c
index 2dd12cb6015b..6f866445efc2 100644
--- a/utils/exportd/exportd.c
+++ b/utils/exportd/exportd.c
@@ -289,12 +289,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1) fork_workers();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
/* Process incoming upcalls */
diff --git a/utils/mountd/mountd.c b/utils/mountd/mountd.c
index bcf749fabbb3..f9c62cded66c 100644
--- a/utils/mountd/mountd.c
+++ b/utils/mountd/mountd.c
@@ -916,12 +916,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1)
fork_workers();
nfsd_path_init();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
xlog(L_NOTICE, "Version " VERSION " starting");
The above patch seems to fix the following commit which is present on
1:2.6.1-1ubuntu1.2 : https://patchwork.kernel.org/project/linux-
nfs/patch/20190521124701.61849-2-trond.myklebust at hammerspace.com/
Questions:
1. Is the mount issue relevant to the above patch?
2. We are not yet prepared to upgrade to Ubuntu Noble yet which has the patch of interest. Can we expect another upgrade for nfs-utils on Jammy which incorporates this patch?
3. Can the patch https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/ be applied in isolation?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737/+subscriptions
More information about the foundations-bugs
mailing list