[Bug 2015396] [NEW] Performance: excessive paging during mdadm resync
Simon Lambourn
2015396 at bugs.launchpad.net
Wed Apr 5 19:27:20 UTC 2023
Public bug reported:
Periodically, RAID arrays are re-synced even when no faults have been reported.
This naturally causes a large amount of I/O but this I/O seems to use large amounts of buffer space and comes to dominate system memory. This causes excessive swapping and unresponsiveness in other applications and in some cases has caused applications to be terminated, presumably due to severe memory shortage. This can have a serious user impact making the system unusable for many hours while the re-sync completes, especially on large RAID arrays.
I have tried using ionice and renice to reduce the priority of the
resync tasks: the problem is somewhat reduced but sometimes the system
is still unusable and applications are terminated without warning.
This seems unreasonable since the re-sync is doing a serial read of the
whole disk, and being predictable should not need large amounts of
bufferspace.
Desired behaviour: bufferspace for resync should be restricted to a
reasonable limit, permitting resync to happen in a timely manner without
stealing the majority of system memory.
ProblemType: Bug
DistroRelease: Ubuntu 22.10
Package: mdadm 4.2-3ubuntu1
ProcVersionSignature: Ubuntu 5.19.0-38.39-generic 5.19.17
Uname: Linux 5.19.0-38-generic x86_64
ApportVersion: 2.23.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Wed Apr 5 20:12:14 2023
Grub.cfg: Error: [Errno 13] Permission denied: '/boot/grub/grub.cfg'
MDadmExamine.dev.sda: Error: command ['/sbin/mdadm', '-E', '/dev/sda'] failed with exit code 1: mdadm: cannot open /dev/sda: Permission denied
MDadmExamine.dev.sda1: Error: command ['/sbin/mdadm', '-E', '/dev/sda1'] failed with exit code 1: mdadm: cannot open /dev/sda1: Permission denied
MDadmExamine.dev.sda2: Error: command ['/sbin/mdadm', '-E', '/dev/sda2'] failed with exit code 1: mdadm: cannot open /dev/sda2: Permission denied
MDadmExamine.dev.sda3: Error: command ['/sbin/mdadm', '-E', '/dev/sda3'] failed with exit code 1: mdadm: cannot open /dev/sda3: Permission denied
MDadmExamine.dev.sda4: Error: command ['/sbin/mdadm', '-E', '/dev/sda4'] failed with exit code 1: mdadm: cannot open /dev/sda4: Permission denied
MDadmExamine.dev.sda5: Error: command ['/sbin/mdadm', '-E', '/dev/sda5'] failed with exit code 1: mdadm: cannot open /dev/sda5: Permission denied
MDadmExamine.dev.sdb: Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: cannot open /dev/sdb: Permission denied
MDadmExamine.dev.sdb1: Error: command ['/sbin/mdadm', '-E', '/dev/sdb1'] failed with exit code 1: mdadm: cannot open /dev/sdb1: Permission denied
MDadmExamine.dev.sdb2: Error: command ['/sbin/mdadm', '-E', '/dev/sdb2'] failed with exit code 1: mdadm: cannot open /dev/sdb2: Permission denied
MDadmExamine.dev.sdb3: Error: command ['/sbin/mdadm', '-E', '/dev/sdb3'] failed with exit code 1: mdadm: cannot open /dev/sdb3: Permission denied
MDadmExamine.dev.sdb4: Error: command ['/sbin/mdadm', '-E', '/dev/sdb4'] failed with exit code 1: mdadm: cannot open /dev/sdb4: Permission denied
MDadmExamine.dev.sdb5: Error: command ['/sbin/mdadm', '-E', '/dev/sdb5'] failed with exit code 1: mdadm: cannot open /dev/sdb5: Permission denied
MDadmExamine.dev.sdc: Error: command ['/sbin/mdadm', '-E', '/dev/sdc'] failed with exit code 1: mdadm: cannot open /dev/sdc: Permission denied
MDadmExamine.dev.sdc1: Error: command ['/sbin/mdadm', '-E', '/dev/sdc1'] failed with exit code 1: mdadm: cannot open /dev/sdc1: Permission denied
MDadmExamine.dev.sdc2: Error: command ['/sbin/mdadm', '-E', '/dev/sdc2'] failed with exit code 1: mdadm: cannot open /dev/sdc2: Permission denied
MDadmExamine.dev.sdc3: Error: command ['/sbin/mdadm', '-E', '/dev/sdc3'] failed with exit code 1: mdadm: cannot open /dev/sdc3: Permission denied
MDadmExamine.dev.sdc4: Error: command ['/sbin/mdadm', '-E', '/dev/sdc4'] failed with exit code 1: mdadm: cannot open /dev/sdc4: Permission denied
MDadmExamine.dev.sdc5: Error: command ['/sbin/mdadm', '-E', '/dev/sdc5'] failed with exit code 1: mdadm: cannot open /dev/sdc5: Permission denied
MDadmExamine.dev.sdd: Error: command ['/sbin/mdadm', '-E', '/dev/sdd'] failed with exit code 1: mdadm: cannot open /dev/sdd: Permission denied
MDadmExamine.dev.sdd1: Error: command ['/sbin/mdadm', '-E', '/dev/sdd1'] failed with exit code 1: mdadm: cannot open /dev/sdd1: Permission denied
MDadmExamine.dev.sdd2: Error: command ['/sbin/mdadm', '-E', '/dev/sdd2'] failed with exit code 1: mdadm: cannot open /dev/sdd2: Permission denied
MDadmExamine.dev.sdd3: Error: command ['/sbin/mdadm', '-E', '/dev/sdd3'] failed with exit code 1: mdadm: cannot open /dev/sdd3: Permission denied
MDadmExamine.dev.sdd4: Error: command ['/sbin/mdadm', '-E', '/dev/sdd4'] failed with exit code 1: mdadm: cannot open /dev/sdd4: Permission denied
MDadmExamine.dev.sdd5: Error: command ['/sbin/mdadm', '-E', '/dev/sdd5'] failed with exit code 1: mdadm: cannot open /dev/sdd5: Permission denied
MachineType: System manufacturer System Product Name
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.19.0-38-generic root=/dev/mapper/vg1-rootfs ro raid=noautodetect
ProcMounts: Error: [Errno 40] Too many levels of symbolic links: '/proc/mounts'
SourcePackage: mdadm
UpgradeStatus: Upgraded to kinetic on 2022-11-03 (153 days ago)
dmi.bios.date: 09/10/2010
dmi.bios.release: 8.14
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1801
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: M3N78-VM
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev X.0x
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1801:bd09/10/2010:br8.14:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnM3N78-VM:rvrRevX.0x:cvnChassisManufacture:ct3:cvrChassisVersion:skuToBeFilledByO.E.M.:
dmi.product.family: To Be Filled By O.E.M.
dmi.product.name: System Product Name
dmi.product.sku: To Be Filled By O.E.M.
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer
etc.blkid.tab: Error: [Errno 2] No such file or directory: '/etc/blkid.tab'
** Affects: mdadm (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug kinetic wayland-session
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mdadm in Ubuntu.
https://bugs.launchpad.net/bugs/2015396
Title:
Performance: excessive paging during mdadm resync
Status in mdadm package in Ubuntu:
New
Bug description:
Periodically, RAID arrays are re-synced even when no faults have been reported.
This naturally causes a large amount of I/O but this I/O seems to use large amounts of buffer space and comes to dominate system memory. This causes excessive swapping and unresponsiveness in other applications and in some cases has caused applications to be terminated, presumably due to severe memory shortage. This can have a serious user impact making the system unusable for many hours while the re-sync completes, especially on large RAID arrays.
I have tried using ionice and renice to reduce the priority of the
resync tasks: the problem is somewhat reduced but sometimes the system
is still unusable and applications are terminated without warning.
This seems unreasonable since the re-sync is doing a serial read of
the whole disk, and being predictable should not need large amounts of
bufferspace.
Desired behaviour: bufferspace for resync should be restricted to a
reasonable limit, permitting resync to happen in a timely manner
without stealing the majority of system memory.
ProblemType: Bug
DistroRelease: Ubuntu 22.10
Package: mdadm 4.2-3ubuntu1
ProcVersionSignature: Ubuntu 5.19.0-38.39-generic 5.19.17
Uname: Linux 5.19.0-38-generic x86_64
ApportVersion: 2.23.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Wed Apr 5 20:12:14 2023
Grub.cfg: Error: [Errno 13] Permission denied: '/boot/grub/grub.cfg'
MDadmExamine.dev.sda: Error: command ['/sbin/mdadm', '-E', '/dev/sda'] failed with exit code 1: mdadm: cannot open /dev/sda: Permission denied
MDadmExamine.dev.sda1: Error: command ['/sbin/mdadm', '-E', '/dev/sda1'] failed with exit code 1: mdadm: cannot open /dev/sda1: Permission denied
MDadmExamine.dev.sda2: Error: command ['/sbin/mdadm', '-E', '/dev/sda2'] failed with exit code 1: mdadm: cannot open /dev/sda2: Permission denied
MDadmExamine.dev.sda3: Error: command ['/sbin/mdadm', '-E', '/dev/sda3'] failed with exit code 1: mdadm: cannot open /dev/sda3: Permission denied
MDadmExamine.dev.sda4: Error: command ['/sbin/mdadm', '-E', '/dev/sda4'] failed with exit code 1: mdadm: cannot open /dev/sda4: Permission denied
MDadmExamine.dev.sda5: Error: command ['/sbin/mdadm', '-E', '/dev/sda5'] failed with exit code 1: mdadm: cannot open /dev/sda5: Permission denied
MDadmExamine.dev.sdb: Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: cannot open /dev/sdb: Permission denied
MDadmExamine.dev.sdb1: Error: command ['/sbin/mdadm', '-E', '/dev/sdb1'] failed with exit code 1: mdadm: cannot open /dev/sdb1: Permission denied
MDadmExamine.dev.sdb2: Error: command ['/sbin/mdadm', '-E', '/dev/sdb2'] failed with exit code 1: mdadm: cannot open /dev/sdb2: Permission denied
MDadmExamine.dev.sdb3: Error: command ['/sbin/mdadm', '-E', '/dev/sdb3'] failed with exit code 1: mdadm: cannot open /dev/sdb3: Permission denied
MDadmExamine.dev.sdb4: Error: command ['/sbin/mdadm', '-E', '/dev/sdb4'] failed with exit code 1: mdadm: cannot open /dev/sdb4: Permission denied
MDadmExamine.dev.sdb5: Error: command ['/sbin/mdadm', '-E', '/dev/sdb5'] failed with exit code 1: mdadm: cannot open /dev/sdb5: Permission denied
MDadmExamine.dev.sdc: Error: command ['/sbin/mdadm', '-E', '/dev/sdc'] failed with exit code 1: mdadm: cannot open /dev/sdc: Permission denied
MDadmExamine.dev.sdc1: Error: command ['/sbin/mdadm', '-E', '/dev/sdc1'] failed with exit code 1: mdadm: cannot open /dev/sdc1: Permission denied
MDadmExamine.dev.sdc2: Error: command ['/sbin/mdadm', '-E', '/dev/sdc2'] failed with exit code 1: mdadm: cannot open /dev/sdc2: Permission denied
MDadmExamine.dev.sdc3: Error: command ['/sbin/mdadm', '-E', '/dev/sdc3'] failed with exit code 1: mdadm: cannot open /dev/sdc3: Permission denied
MDadmExamine.dev.sdc4: Error: command ['/sbin/mdadm', '-E', '/dev/sdc4'] failed with exit code 1: mdadm: cannot open /dev/sdc4: Permission denied
MDadmExamine.dev.sdc5: Error: command ['/sbin/mdadm', '-E', '/dev/sdc5'] failed with exit code 1: mdadm: cannot open /dev/sdc5: Permission denied
MDadmExamine.dev.sdd: Error: command ['/sbin/mdadm', '-E', '/dev/sdd'] failed with exit code 1: mdadm: cannot open /dev/sdd: Permission denied
MDadmExamine.dev.sdd1: Error: command ['/sbin/mdadm', '-E', '/dev/sdd1'] failed with exit code 1: mdadm: cannot open /dev/sdd1: Permission denied
MDadmExamine.dev.sdd2: Error: command ['/sbin/mdadm', '-E', '/dev/sdd2'] failed with exit code 1: mdadm: cannot open /dev/sdd2: Permission denied
MDadmExamine.dev.sdd3: Error: command ['/sbin/mdadm', '-E', '/dev/sdd3'] failed with exit code 1: mdadm: cannot open /dev/sdd3: Permission denied
MDadmExamine.dev.sdd4: Error: command ['/sbin/mdadm', '-E', '/dev/sdd4'] failed with exit code 1: mdadm: cannot open /dev/sdd4: Permission denied
MDadmExamine.dev.sdd5: Error: command ['/sbin/mdadm', '-E', '/dev/sdd5'] failed with exit code 1: mdadm: cannot open /dev/sdd5: Permission denied
MachineType: System manufacturer System Product Name
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.19.0-38-generic root=/dev/mapper/vg1-rootfs ro raid=noautodetect
ProcMounts: Error: [Errno 40] Too many levels of symbolic links: '/proc/mounts'
SourcePackage: mdadm
UpgradeStatus: Upgraded to kinetic on 2022-11-03 (153 days ago)
dmi.bios.date: 09/10/2010
dmi.bios.release: 8.14
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1801
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: M3N78-VM
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev X.0x
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1801:bd09/10/2010:br8.14:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnM3N78-VM:rvrRevX.0x:cvnChassisManufacture:ct3:cvrChassisVersion:skuToBeFilledByO.E.M.:
dmi.product.family: To Be Filled By O.E.M.
dmi.product.name: System Product Name
dmi.product.sku: To Be Filled By O.E.M.
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer
etc.blkid.tab: Error: [Errno 2] No such file or directory: '/etc/blkid.tab'
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/2015396/+subscriptions
More information about the foundations-bugs
mailing list