[SRU][J][PATCH 0/3] SUNRPC: System wide grep leads to NULL pointer deference in sysfs reads

[Matthew Ruffell]

BugLink: https://bugs.launchpad.net/bugs/2149767

[Impact]

An unprivileged user doing a simple system wide grep can cause a NULL pointer
dereference and oops in the SUNRPC subsystem, leading to a local Denial Of 
Service.

A user doing a grep such as 

$ grep -R "something" /

will eventually make its way to /sys/kernel/sunrpc/, where it can hit a race
where ->sock in SUNRPC can be set to NULL, like if a network was going down and
up again, or a nfs server was being restarted, leading to the following oops.

BUG: kernel NULL pointer dereference, address: 0000000000000020
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0 
Oops: 0000 SMP NOPTI
CPU: 2 PID: 4933 Comm: grep Not tainted 5.15.0-176-generic #186-Ubuntu
RIP: 0010:kernel_getsockname+0x6/0x20
Code: 00 00 00 00 0f 1f 44 00 00 55 48 8b 47 20 48 8b 40 60 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc cc cc 0f 1f 40 00 0f 1f 44 00 00 55 <48> 8b 47 20 31 d2 48 8b 40 38 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc
Call Trace:
 <TASK>
 ? xs_sock_getport+0x2b/0x70 [sunrpc]
 ? kvmalloc_node+0x28/0xa0
 ? memcg_slab_post_alloc_hook+0x19e/0x210
 get_srcport+0x15/0x20 [sunrpc]
 rpc_sysfs_xprt_info_show+0x110/0x130 [sunrpc]
 kobj_attr_show+0xf/0x30
 sysfs_kf_seq_show+0xa2/0x100
 kernfs_seq_show+0x24/0x30
 seq_read_iter+0x121/0x4b0
 ? _copy_to_user+0x20/0x30
 ? cp_new_stat+0x152/0x180
 kernfs_fop_read_iter+0x30/0x40
 new_sync_read+0x10a/0x190
 vfs_read+0x106/0x1a0
 ksys_read+0x67/0xf0
 __x64_sys_read+0x19/0x20
 x64_sys_call+0x1dba/0x1fa0
 do_syscall_64+0x56/0xb0
 ? do_syscall_64+0x63/0xb0
 ? do_syscall_64+0x63/0xb0
 ? arch_exit_to_user_mode_prepare.constprop.0+0x1e/0xc0
 ? syscall_exit_to_user_mode+0x41/0x80
 ? do_syscall_64+0x63/0xb0
 entry_SYSCALL_64_after_hwframe+0x6c/0xd6

A workaround is to exclude /sys from your grep or find commands.

[Fix]

The fix is to ensure that SUNRPC holds the ->recv_mutex during sysfs reads.
This makes sure that ->sock cannot be modified by an external change, e.g.
a nfs server being restarted.

The fix, and their dependencies and fixes are:

commit 17f09d3f619a7ad2d2b021b4e5246f08225b1b0f
Author: Anna Schumaker <Anna.Schumaker at Netapp.com>
Date: Thu Oct 28 15:17:41 2021 -0400
Subject: SUNRPC: Check if the xprt is connected before handling sysfs reads
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=17f09d3f619a7ad2d2b021b4e5246f08225b1b0f

commit b49ea673e119f59c71645e2f65b3ccad857c90ee
Author: NeilBrown <neil at brown.name>
Date: Mon Jan 17 16:36:53 2022 +1100
Subject: SUNRPC: lock against ->sock changing during sysfs read
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b49ea673e119f59c71645e2f65b3ccad857c90ee

commit 421ab1be43bd015ffe744f4ea25df4f19d1ce6fe
Author: Trond Myklebust <trond.myklebust at hammerspace.com>
Date: Fri Mar 25 10:37:31 2022 -0400
Subject: SUNRPC: Do not dereference non-socket transports in sysfs
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=421ab1be43bd015ffe744f4ea25df4f19d1ce6fe

These landed during 5.16-rc1 and 5.18-rc1.

[Testcase]

Create a fresh jammy VM.

Create a NFS share:

$ sudo apt install nfs-kernel-server
$ sudo mkdir -p /mnt/nfs_share
$ sudo chown nobody:nogroup /mnt/nfs_share
$ sudo chmod 777 /mnt/nfs_share
$ sudo vim /etc/exports
/mnt/nfs_share  192.168.1.0/24(rw,sync,no_subtree_check)
$ sudo exportfs -a
$ sudo systemctl restart nfs-kernel-server

Set up a loop where we grep the SUNRPC sysfs interface, causing a read to
happen, and some of these reads will happen when the ->recv_mutex is not held.

$ while true; do
    grep -Rr . /sys/kernel/sunrpc/xprt-switches/;
done

Set up a loop where we mount and unmount the nfs share. This triggers the SUNRPC
transport to disconnect or change states.

$ sudo -s
# while true; do
    mount -t nfs 192.168.122.126:/mnt/nfs_share /mnt/test
    umount /mnt/test
done

Wait a few seconds and the kernel will oops with a null pointer dereference.

There is a test kernel available in the following PPA:

https://launchpad.net/~mruffell/+archive/ubuntu/sf434838-test

If you install the test kernel, you can keep running the loops forever without
any kernel oops.

[Where problems can occur]

We are changing how SUNRPC protects sysfs reads, ensuring we take a mutex to
protect the socket transport from changing due to external factors. Taking the
mutex might take time, and slow down sysfs read operations, or cause deadlocks
in other places in SUNRPC if not done correctly.

The patch "SUNRPC: Do not dereference non-socket transports in sysfs" is quite a
refactor, but the risk to RDMA users not having the patch is higher than
carrying the patch.

If a regression were to occur, users could likely work around the issue by not
using system wide grep or find commands that parse sysfs entries.

[Other info]

This is known as CVE-2022-48816.

https://ubuntu.com/security/CVE-2022-48816
https://nvd.nist.gov/vuln/detail/cve-2022-48816

Anna Schumaker (1):
  SUNRPC: Check if the xprt is connected before handling sysfs reads

NeilBrown (1):
  SUNRPC: lock against ->sock changing during sysfs read

Trond Myklebust (1):
  SUNRPC: Do not dereference non-socket transports in sysfs

 include/linux/sunrpc/xprt.h     |  3 ++
 include/linux/sunrpc/xprtsock.h |  1 -
 net/sunrpc/sysfs.c              | 62 ++++++++++++++++++---------------
 net/sunrpc/xprtsock.c           | 33 ++++++++++++++++--
 4 files changed, 67 insertions(+), 32 deletions(-)

-- 
2.53.0