[Bug 2091737] Re: Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from 1:1.3.4-2.5ubuntu3)
Ram Sun
2091737 at bugs.launchpad.net
Mon Dec 16 21:36:36 UTC 2024
I have also made a few more observations but one of the key observations that can be easily reproduced on any filesystem is that with the jammy versions, the number of `statx` calls have drastically gone up. Additionally, for each export, the number of statx calls is somewhat proportional to the number of mountd threads which in my case is 16 that I can see as follows by performing an strace on mountd as follows:
sudo strace -tt -T -p $(pgrep -d',' 'rpc.mountd') -o trace.txt
Jammy(1219 statx with 40 exports and attempting 1 mount)
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy.txt | grep statx |wc -l
1219
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy.txt | grep statx | grep "test40" | wc -l
16
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy.txt | grep statx | grep -w "test1" | wc -l
18
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy.txt | grep statx | grep -w "test2" | wc -l
16
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy.txt | grep statx | grep -w "test3" | wc -l
16
Jammy with patch(Similar behavior as Noble/Focal versions) shows only 254 statx calls:
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy_with_patch.txt | grep statx |wc -l
254
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy_with_patch.txt | grep statx | grep -w "test3" | wc -l
5
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy_with_patch.txt | grep statx | grep -w "test1" | wc -l
7
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$ cat statx_jammy_with_patch.txt | grep statx | grep -w "test40" | wc -l
5
AMER-W64TDH9XKR:Desktop Ramakrishnan.Sundareswaran$
How to reproduce?
Export 40 mounts same as earlier and perform a mount on /home/rksupport directory which is on root os.
Mount command:
time sudo mount -o rw,hard,rsize=1048576,wsize=1048576,vers=3,nointr,tcp,sync ${SERVER_IP}:/home/rksupport/test1 /tmp/test/test1
NFS Server is configured with 16 mountd threads and has been verified with ps aux and the following config:
ubuntu at vm-machine-s82gbm-3ojtydo:~/nfs_source/nfs-utils-2.6.1$ cat /etc/nfs.conf.d/local.conf | grep -A3 mountd
[mountd]
manage-gids = 1
port = 32767
threads = 16
ubuntu at vm-machine-s82gbm-3ojtydo:~/nfs_source/nfs-utils-2.6.1$
** Attachment added: "strace for statx"
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737/+attachment/5845621/+files/statx_trace.zip
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/2091737
Title:
Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from
1:1.3.4-2.5ubuntu3)
Status in nfs-utils package in Ubuntu:
Confirmed
Bug description:
We are experiencing significant delays (60+ seconds) when mounting NFS
directories after exporting 40 directories. This issue is observed
with nfs-utils 1:2.6.1-1ubuntu1.2 on an Ubuntu Jammy system. We have
40 exports and are attempting to mount via NFS onto /sd/mount paths.
The /sd filesystem is a custom, in-house filesystem built on top of
FUSE. Prior to using nfs-utils 1:2.6.1-1ubuntu1.2, we were using nfs-
utils 1:1.3.4-2.5ubuntu3(Ubuntu 20.04 focal) where mounts were almost
immediate, taking less than 10 seconds.
Applying this patch from upstream nfs-utils resolves the issue:
https://patchwork.kernel.org/project/linux-
nfs/patch/20231023021052.5258-4-neilb at suse.de/
Additionally, the above problem is not reproducible using nfs-utils
1:2.6.4-3ubuntu5(Ubuntu 24.04 Noble Numbat)
System Information:
OS: Ubuntu 22.04 LTS Jammy
nfs-utils Version: 1:2.6.1-1ubuntu1.2
Kernel: 5.15.0-117
Steps to reproduce internally:
Create 40 directories in /sd/mount and export them.
Execute showmount -e to list the exports.
Attempt to mount one of the NFS directories and note the time taken.
Actual Outcome:
Mounting a single directory takes more than 60 seconds consistently. While attempting to perform multiple mounts, some mounts can take more than 300s.
Expected Outcome:
Mounting should be significantly faster.
Logs and Outputs:
Export List:
ubuntu at vm-machine-s82gbm-3ojtydo:~$ showmount -e
Export list for vm-machine-s82gbm-3ojtydo:
/sd/mount/oracle/jammy_patch/test40 *
/sd/mount/oracle/jammy_patch/test39 *
/sd/mount/oracle/jammy_patch/test38 *
/sd/mount/oracle/jammy_patch/test37 *
/sd/mount/oracle/jammy_patch/test36 *
/sd/mount/oracle/jammy_patch/test35 *
/sd/mount/oracle/jammy_patch/test34 *
/sd/mount/oracle/jammy_patch/test33 *
/sd/mount/oracle/jammy_patch/test32 *
/sd/mount/oracle/jammy_patch/test31 *
/sd/mount/oracle/jammy_patch/test30 *
/sd/mount/oracle/jammy_patch/test29 *
/sd/mount/oracle/jammy_patch/test28 *
/sd/mount/oracle/jammy_patch/test27 *
/sd/mount/oracle/jammy_patch/test26 *
/sd/mount/oracle/jammy_patch/test25 *
/sd/mount/oracle/jammy_patch/test24 *
/sd/mount/oracle/jammy_patch/test23 *
/sd/mount/oracle/jammy_patch/test22 *
/sd/mount/oracle/jammy_patch/test21 *
/sd/mount/oracle/jammy_patch/test20 *
/sd/mount/oracle/jammy_patch/test19 *
/sd/mount/oracle/jammy_patch/test18 *
/sd/mount/oracle/jammy_patch/test17 *
/sd/mount/oracle/jammy_patch/test16 *
/sd/mount/oracle/jammy_patch/test15 *
/sd/mount/oracle/jammy_patch/test14 *
/sd/mount/oracle/jammy_patch/test13 *
/sd/mount/oracle/jammy_patch/test12 *
/sd/mount/oracle/jammy_patch/test11 *
/sd/mount/oracle/jammy_patch/test10 *
/sd/mount/oracle/jammy_patch/test9 *
/sd/mount/oracle/jammy_patch/test8 *
/sd/mount/oracle/jammy_patch/test7 *
/sd/mount/oracle/jammy_patch/test6 *
/sd/mount/oracle/jammy_patch/test5 *
/sd/mount/oracle/jammy_patch/test4 *
/sd/mount/oracle/jammy_patch/test3 *
/sd/mount/oracle/jammy_patch/test2 *
/sd/mount/oracle/jammy_patch/test1 *
/sd/mount ::1
Mount command:
time sudo mount -o rw,hard,rsize=1048576,wsize=1048576,vers=3,nointr,tcp,sync ${SERVER_IP}:/sd/mount/oracle/jammy_patch/test1 /tmp/test/test1
Patch Applied:
Fix commit: https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/
diff --git a/utils/exportd/exportd.c b/utils/exportd/exportd.c
index 2dd12cb6015b..6f866445efc2 100644
--- a/utils/exportd/exportd.c
+++ b/utils/exportd/exportd.c
@@ -289,12 +289,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1) fork_workers();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
/* Process incoming upcalls */
diff --git a/utils/mountd/mountd.c b/utils/mountd/mountd.c
index bcf749fabbb3..f9c62cded66c 100644
--- a/utils/mountd/mountd.c
+++ b/utils/mountd/mountd.c
@@ -916,12 +916,16 @@ main(int argc, char **argv)
else if (num_threads > MAX_THREADS)
num_threads = MAX_THREADS;
/* Open cache channel files BEFORE forking so each upcall is
only handled by one thread. Kernel provides locking for both
read and write.
*/
cache_open();
if (num_threads > 1)
fork_workers();
nfsd_path_init();
/* Open files now to avoid sharing descriptors among forked processes */
cache_open();
v4clients_init();
xlog(L_NOTICE, "Version " VERSION " starting");
The above patch seems to fix the following commit which is present on
1:2.6.1-1ubuntu1.2 : https://patchwork.kernel.org/project/linux-
nfs/patch/20190521124701.61849-2-trond.myklebust at hammerspace.com/
Questions:
1. Is the mount issue relevant to the above patch?
2. We are not yet prepared to upgrade to Ubuntu Noble yet which has the patch of interest. Can we expect another upgrade for nfs-utils on Jammy which incorporates this patch?
3. Can the patch https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/ be applied in isolation?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737/+subscriptions
More information about the foundations-bugs
mailing list