[Bug 2091737] Re: Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from 1:1.3.4-2.5ubuntu3)
Ram Sun
2091737 at bugs.launchpad.net
Mon Jan 13 18:25:58 UTC 2025
Andres,
Following up on this again and also have an update based on the
following email discussion with the author that wrote the original diff
and seems like this patch is indeed required to fix the mount
regression. Can you please provide an update if Ubuntu jammy can release
a new package with the patch officially?
On Tue, 07 Jan 2025, Ramakrishnan Sundareswaran wrote:
> Hi Neal,
>
> I would like your inputs on a potential regression in the nfs-utils
> package on Ubuntu 22.04/Jammy.
>
> I've filed the following ticket
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737 for
> the Ubuntu Jammy nfs-utils which seems to have a regression fixed by
> you in this patch
> https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/
> [3/6] export: move cache_open() before workers are forked
>
> However, based on the current discussion in the Ubuntu bug report, we
> still haven't gotten confirmation that this is indeed a regression
> that was fixed in a subsequent commit.
>
> Based on the ticket, there are three key questions that I'm trying to understand
>
> Questions:
> 1. Is the mount issue relevant to the above patch?
The fact that applying the patch makes your problem go away seems fairly
conclusive.
It only affects configurations with more than 1 mountd thread. If you
change back to 1 thread I suspect the slowness will go away. That is
the config that most people test which is probably why most people don't
notice.
If you confirm that:
- without the patch:
- with "--num-threads=1" you don't see slowness
- with "--num-threads=8" you do see slowness
- with the patch
- with --num-threads= 1 or 8 you don't see any slowness
then that would confirm that this patch exactly addresses your problem.
> 2. We are not yet prepared to upgrade to Ubuntu Noble yet which has
> the patch of interest. Can we expect another upgrade for nfs-utils on
> Jammy which incorporates this patch?
> 3. Can the patch
> https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/
> be applied in isolation?
Yes. There are no non-obvious dependencies.
NeilBrown
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/2091737
Title:
Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from
1:1.3.4-2.5ubuntu3)
Status in nfs-utils package in Ubuntu:
Confirmed
Status in nfs-utils source package in Jammy:
In Progress
Bug description:
We are experiencing significant delays (60+ seconds) when mounting NFS
directories after exporting 40 directories. This issue is observed
with nfs-utils 1:2.6.1-1ubuntu1.2 on an Ubuntu Jammy system. We have
40 exports and are attempting to mount via NFS onto /sd/mount paths.
The /sd filesystem is a custom, in-house filesystem built on top of
FUSE. Prior to using nfs-utils 1:2.6.1-1ubuntu1.2, we were using nfs-
utils 1:1.3.4-2.5ubuntu3(Ubuntu 20.04 focal) where mounts were almost
immediate, taking less than 10 seconds.
Applying this patch from upstream nfs-utils resolves the issue:
https://patchwork.kernel.org/project/linux-
nfs/patch/20231023021052.5258-4-neilb at suse.de/
Additionally, the above problem is not reproducible using nfs-utils
1:2.6.4-3ubuntu5(Ubuntu 24.04 Noble Numbat)
System Information:
OS: Ubuntu 22.04 LTS Jammy
nfs-utils Version: 1:2.6.1-1ubuntu1.2
Kernel: 5.15.0-117
Steps to reproduce internally:
Create 40 directories in /sd/mount and export them.
Execute showmount -e to list the exports.
Attempt to mount one of the NFS directories and note the time taken.
Actual Outcome:
Mounting a single directory takes more than 60 seconds consistently. While attempting to perform multiple mounts, some mounts can take more than 300s.
Expected Outcome:
Mounting should be significantly faster.
Logs and Outputs:
Export List:
ubuntu at vm-machine-s82gbm-3ojtydo:~$ showmount -e
Export list for vm-machine-s82gbm-3ojtydo:
/sd/mount/oracle/jammy_patch/test40 *
/sd/mount/oracle/jammy_patch/test39 *
/sd/mount/oracle/jammy_patch/test38 *
/sd/mount/oracle/jammy_patch/test37 *
/sd/mount/oracle/jammy_patch/test36 *
/sd/mount/oracle/jammy_patch/test35 *
/sd/mount/oracle/jammy_patch/test34 *
/sd/mount/oracle/jammy_patch/test33 *
/sd/mount/oracle/jammy_patch/test32 *
/sd/mount/oracle/jammy_patch/test31 *
/sd/mount/oracle/jammy_patch/test30 *
/sd/mount/oracle/jammy_patch/test29 *
/sd/mount/oracle/jammy_patch/test28 *
/sd/mount/oracle/jammy_patch/test27 *
/sd/mount/oracle/jammy_patch/test26 *
/sd/mount/oracle/jammy_patch/test25 *
/sd/mount/oracle/jammy_patch/test24 *
/sd/mount/oracle/jammy_patch/test23 *
/sd/mount/oracle/jammy_patch/test22 *
/sd/mount/oracle/jammy_patch/test21 *
/sd/mount/oracle/jammy_patch/test20 *
/sd/mount/oracle/jammy_patch/test19 *
/sd/mount/oracle/jammy_patch/test18 *
/sd/mount/oracle/jammy_patch/test17 *
/sd/mount/oracle/jammy_patch/test16 *
/sd/mount/oracle/jammy_patch/test15 *
/sd/mount/oracle/jammy_patch/test14 *
/sd/mount/oracle/jammy_patch/test13 *
/sd/mount/oracle/jammy_patch/test12 *
/sd/mount/oracle/jammy_patch/test11 *
/sd/mount/oracle/jammy_patch/test10 *
/sd/mount/oracle/jammy_patch/test9 *
/sd/mount/oracle/jammy_patch/test8 *
/sd/mount/oracle/jammy_patch/test7 *
/sd/mount/oracle/jammy_patch/test6 *
/sd/mount/oracle/jammy_patch/test5 *
/sd/mount/oracle/jammy_patch/test4 *
/sd/mount/oracle/jammy_patch/test3 *
/sd/mount/oracle/jammy_patch/test2 *
/sd/mount/oracle/jammy_patch/test1 *
/sd/mount ::1
Mount command:
time sudo mount -o rw,hard,rsize=1048576,wsize=1048576,vers=3,nointr,tcp,sync ${SERVER_IP}:/sd/mount/oracle/jammy_patch/test1 /tmp/test/test1
Patch Applied:
Fix commit: https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/
diff --git a/utils/exportd/exportd.c b/utils/exportd/exportd.c
index 2dd12cb6015b..6f866445efc2 100644
--- a/utils/exportd/exportd.c
+++ b/utils/exportd/exportd.c
@@ -289,12 +289,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1) fork_workers();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
/* Process incoming upcalls */
diff --git a/utils/mountd/mountd.c b/utils/mountd/mountd.c
index bcf749fabbb3..f9c62cded66c 100644
--- a/utils/mountd/mountd.c
+++ b/utils/mountd/mountd.c
@@ -916,12 +916,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1)
fork_workers();
nfsd_path_init();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
xlog(L_NOTICE, "Version " VERSION " starting");
The above patch seems to fix the following commit which is present on
1:2.6.1-1ubuntu1.2 : https://patchwork.kernel.org/project/linux-
nfs/patch/20190521124701.61849-2-trond.myklebust at hammerspace.com/
Questions:
1. Is the mount issue relevant to the above patch?
2. We are not yet prepared to upgrade to Ubuntu Noble yet which has the patch of interest. Can we expect another upgrade for nfs-utils on Jammy which incorporates this patch?
3. Can the patch https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/ be applied in isolation?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737/+subscriptions
More information about the foundations-bugs
mailing list