[Bug 2091737] Re: Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from 1:1.3.4-2.5ubuntu3)
Ram Sun
2091737 at bugs.launchpad.net
Fri Dec 27 01:00:21 UTC 2024
Andres,
The server OS is Jammy and the client OS is also jammy.
The export options look as follows for a sample export:
/home/ubuntu/test1 *(rw,sync,insecure,no_subtree_check,fsid=1)
For strace, my workflow was something as follows:
Server:
Update /etc/exports file and perform exportfs -ra
Verify exports using showmount -e
Start strace using this command:
sudo strace -tt -T -p $(pgrep -d',' 'rpc.mountd') -o strace_jammy_ppa.log
Client:
Attempt mount using this command:
time sudo mount -o rw,hard,rsize=1048576,wsize=1048576,vers=3,nointr,tcp,sync ${SERVER_IP}:${NFS_DIR} ${LOCAL_DIR}
To repeat the experiment, we don't need to restart the server daemons
but instead, just doing an exportfs -ra should do the job.
As far as the slowness is concerned, I understand that maybe in non
custom filesystems, it might not be that evident but the number of statx
calls iiuc should be reproducible since I was able to do that in
situations where I was using ext4 directly which is what /home/ubuntu is
in my case. However, I have set my config to use 16 mountd threads so
not sure if that might be having an impact. Can you see if increasing
the number of mountd processes helps you observe the difference with
statx?
For a single mount, I saw 42 statx calls with the patch which isn't the case without the patch:
ubuntu at vm-machine-s82gbm-3ojtydo:~$ grep statx mount_ppa_ubuntu_patch_single_trace.log | wc -l
42
I've also attached the entire strace logs for your reference to compare
to ensure that with the ppa builds, we are indeed seeing somewhat
similar calls.
** Attachment added: "strace_logs_statx_ubuntu_ppa.rtf"
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737/+attachment/5847720/+files/strace_logs_statx_ubuntu_ppa.rtf
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/2091737
Title:
Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from
1:1.3.4-2.5ubuntu3)
Status in nfs-utils package in Ubuntu:
Confirmed
Status in nfs-utils source package in Jammy:
In Progress
Bug description:
We are experiencing significant delays (60+ seconds) when mounting NFS
directories after exporting 40 directories. This issue is observed
with nfs-utils 1:2.6.1-1ubuntu1.2 on an Ubuntu Jammy system. We have
40 exports and are attempting to mount via NFS onto /sd/mount paths.
The /sd filesystem is a custom, in-house filesystem built on top of
FUSE. Prior to using nfs-utils 1:2.6.1-1ubuntu1.2, we were using nfs-
utils 1:1.3.4-2.5ubuntu3(Ubuntu 20.04 focal) where mounts were almost
immediate, taking less than 10 seconds.
Applying this patch from upstream nfs-utils resolves the issue:
https://patchwork.kernel.org/project/linux-
nfs/patch/20231023021052.5258-4-neilb at suse.de/
Additionally, the above problem is not reproducible using nfs-utils
1:2.6.4-3ubuntu5(Ubuntu 24.04 Noble Numbat)
System Information:
OS: Ubuntu 22.04 LTS Jammy
nfs-utils Version: 1:2.6.1-1ubuntu1.2
Kernel: 5.15.0-117
Steps to reproduce internally:
Create 40 directories in /sd/mount and export them.
Execute showmount -e to list the exports.
Attempt to mount one of the NFS directories and note the time taken.
Actual Outcome:
Mounting a single directory takes more than 60 seconds consistently. While attempting to perform multiple mounts, some mounts can take more than 300s.
Expected Outcome:
Mounting should be significantly faster.
Logs and Outputs:
Export List:
ubuntu at vm-machine-s82gbm-3ojtydo:~$ showmount -e
Export list for vm-machine-s82gbm-3ojtydo:
/sd/mount/oracle/jammy_patch/test40 *
/sd/mount/oracle/jammy_patch/test39 *
/sd/mount/oracle/jammy_patch/test38 *
/sd/mount/oracle/jammy_patch/test37 *
/sd/mount/oracle/jammy_patch/test36 *
/sd/mount/oracle/jammy_patch/test35 *
/sd/mount/oracle/jammy_patch/test34 *
/sd/mount/oracle/jammy_patch/test33 *
/sd/mount/oracle/jammy_patch/test32 *
/sd/mount/oracle/jammy_patch/test31 *
/sd/mount/oracle/jammy_patch/test30 *
/sd/mount/oracle/jammy_patch/test29 *
/sd/mount/oracle/jammy_patch/test28 *
/sd/mount/oracle/jammy_patch/test27 *
/sd/mount/oracle/jammy_patch/test26 *
/sd/mount/oracle/jammy_patch/test25 *
/sd/mount/oracle/jammy_patch/test24 *
/sd/mount/oracle/jammy_patch/test23 *
/sd/mount/oracle/jammy_patch/test22 *
/sd/mount/oracle/jammy_patch/test21 *
/sd/mount/oracle/jammy_patch/test20 *
/sd/mount/oracle/jammy_patch/test19 *
/sd/mount/oracle/jammy_patch/test18 *
/sd/mount/oracle/jammy_patch/test17 *
/sd/mount/oracle/jammy_patch/test16 *
/sd/mount/oracle/jammy_patch/test15 *
/sd/mount/oracle/jammy_patch/test14 *
/sd/mount/oracle/jammy_patch/test13 *
/sd/mount/oracle/jammy_patch/test12 *
/sd/mount/oracle/jammy_patch/test11 *
/sd/mount/oracle/jammy_patch/test10 *
/sd/mount/oracle/jammy_patch/test9 *
/sd/mount/oracle/jammy_patch/test8 *
/sd/mount/oracle/jammy_patch/test7 *
/sd/mount/oracle/jammy_patch/test6 *
/sd/mount/oracle/jammy_patch/test5 *
/sd/mount/oracle/jammy_patch/test4 *
/sd/mount/oracle/jammy_patch/test3 *
/sd/mount/oracle/jammy_patch/test2 *
/sd/mount/oracle/jammy_patch/test1 *
/sd/mount ::1
Mount command:
time sudo mount -o rw,hard,rsize=1048576,wsize=1048576,vers=3,nointr,tcp,sync ${SERVER_IP}:/sd/mount/oracle/jammy_patch/test1 /tmp/test/test1
Patch Applied:
Fix commit: https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/
diff --git a/utils/exportd/exportd.c b/utils/exportd/exportd.c
index 2dd12cb6015b..6f866445efc2 100644
--- a/utils/exportd/exportd.c
+++ b/utils/exportd/exportd.c
@@ -289,12 +289,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1) fork_workers();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
/* Process incoming upcalls */
diff --git a/utils/mountd/mountd.c b/utils/mountd/mountd.c
index bcf749fabbb3..f9c62cded66c 100644
--- a/utils/mountd/mountd.c
+++ b/utils/mountd/mountd.c
@@ -916,12 +916,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1)
fork_workers();
nfsd_path_init();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
xlog(L_NOTICE, "Version " VERSION " starting");
The above patch seems to fix the following commit which is present on
1:2.6.1-1ubuntu1.2 : https://patchwork.kernel.org/project/linux-
nfs/patch/20190521124701.61849-2-trond.myklebust at hammerspace.com/
Questions:
1. Is the mount issue relevant to the above patch?
2. We are not yet prepared to upgrade to Ubuntu Noble yet which has the patch of interest. Can we expect another upgrade for nfs-utils on Jammy which incorporates this patch?
3. Can the patch https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/ be applied in isolation?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737/+subscriptions
More information about the foundations-bugs
mailing list