[Bug 2040076] [NEW] Something is holding a lock on unmounted partition
Josef Wolf
2040076 at bugs.launchpad.net
Sat Oct 21 21:53:32 UTC 2023
Public bug reported:
Following instructions from https://discourse.maas.io/t/deploying-
servers-with-full-disk-encryption-luks2/3286 to get full disk encryption
with ubuntu-22.04-server
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount -v /boot/efi
umount: /boot/efi unmounted
# umount -v /boot
umount: /boot unmounted
# cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
WARNING!
========
This will overwrite data on /dev/nvme0n1p3 irrevocably.
Are you sure? (Type 'yes' in capital letters); YES
Enter Passphrase for /dev/nvme0n1p3:
Verify Passphrase:
Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
Uh, not what I expected. So i start to investigate:
# grep nvme0n1p3 /proc/mounts
# lsof | grep nvme0n1p3
# lsof nvme0n1p3
# fuser nvme0n1p3
So:
- umount succeeded
- /proc/mounts, lsof, fuser don's show any sign that the partition is in use
BUT:
strace cryptsetup luksFormat --type luks1 /dev/nvme0n1p3 2>&1 |
egrep '(nvme0n1p3|close)'
shows
openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|ODIRECT) = 3
close(3)
openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|O_NONBLOCK|OCLOEXEC) = 3
close(3)
openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDWR|OEXCL) = -1 EBUSY (Device or resource busy)
So there must be something holding a hand on the partition. But there's
no trace of what it might be
=======================================================
Another try (after fresh boot):
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount /boot/efi
# umount /boot
# mount /boot
mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
and again: no sign that the partition is in use:
# grep nvme0n1p3 /proc/mounts
# lsof | grep nvme0n1p3
# lsof nvme0n1p3
# fuser nvme0n1p3
=======================================================
So another try (reboot again):
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount /boot/efi
# mount -o remount,rw /boot # <<<<------- notice this one!
# umount /boot
# mount /boot # <<<<------- Now it succeeds!
Now do the same, but without the "mount -oremount,rw /boot"
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount /boot
# mount /boot
mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
Oooops! Now system is messed up again. No way to "clean up the mess.
Getting closer? Partition originally was mounted read-write. umount
while remounted read-only seems to mess up things and mount looses track
of was is going on..
Thus, another try:
=====================================================
Again freshly booted. Omit the "mount -oremount,ro /boot" this time:
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount -v /boot/efi
umount: /boot/efi unmounted
# umount -v /boot
umount: /boot unmounted
# cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
WARNING!
========
This will overwrite data on /dev/nvme0n1p3 irrevocably.
Are you sure? (Type 'yes' in capital letters); YES
Enter Passphrase for /dev/nvme0n1p3:
Verify Passphrase:
Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
failed again, but
# mount /boot # <<<<------- Now it succeeds!
# umount /boot # <<<<------- Now it succeeds!
# mount /boot # <<<<------- Now it succeeds!
So, how can I track this one down further?
** Affects: util-linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: cryptsetup mount umount
** Description changed:
Following instructions from https://discourse.maas.io/t/deploying-
servers-with-full-disk-encryption-luks2/3286 to get full disk encryption
with ubuntu-22.04-server
- # mount -o remount,ro /boot
- # install -m0600 /dev/null /tmp/boot.tar
- # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
- # umount -v /boot/efi
- umount: /boot/efi unmounted
- # umount -v /boot
- umount: /boot unmounted
- # cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
- WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
+ # mount -o remount,ro /boot
+ # install -m0600 /dev/null /tmp/boot.tar
+ # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
+ # umount -v /boot/efi
+ umount: /boot/efi unmounted
+ # umount -v /boot
+ umount: /boot unmounted
+ # cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
+ WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
- WARNING!
- ========
- This will overwrite dataon /dev/nvme0n1p3 irrevocably.
+ WARNING!
+ ========
+ This will overwrite data on /dev/nvme0n1p3 irrevocably.
- Are you sure? (Type 'yes' in capital letters); YES
- Enter Passphrase for /dev/nvme0n1p3:
- Verify Passphrase:
- Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
+ Are you sure? (Type 'yes' in capital letters); YES
+ Enter Passphrase for /dev/nvme0n1p3:
+ Verify Passphrase:
+ Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
Uh, not what I expected. So i start to investigate:
- # grep nvme0n1p3 /proc/mounts
- # lsof | grep nvme0n1p3
- # lsof nvme0n1p3
- # fuser nvme0n1p3
+ # grep nvme0n1p3 /proc/mounts
+ # lsof | grep nvme0n1p3
+ # lsof nvme0n1p3
+ # fuser nvme0n1p3
So:
- umount succeeded
- /proc/mounts, lsof, fuser don's show any sign that the partition is in use
BUT:
- strace cryptsetup luksFormat --type luks1 /dev/nvme0n1p3 2>&1 |
+ strace cryptsetup luksFormat --type luks1 /dev/nvme0n1p3 2>&1 |
egrep '(nvme0n1p3|close)'
shows
- openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|ODIRECT) = 3
- close(3)
- openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|O_NONBLOCK|OCLOEXEC) = 3
- close(3)
- openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDWR|OEXCL) = -1 EBUSY (Device or resource busy)
+ openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|ODIRECT) = 3
+ close(3)
+ openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|O_NONBLOCK|OCLOEXEC) = 3
+ close(3)
+ openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDWR|OEXCL) = -1 EBUSY (Device or resource busy)
So there must be something holding a hand on the partition. But there's
no trace of what it might be
=======================================================
Another try (after fresh boot):
- # mount -o remount,ro /boot
- # install -m0600 /dev/null /tmp/boot.tar
- # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
- # umount /boot/efi
- # umount /boot
- # mount /boot
- mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
+ # mount -o remount,ro /boot
+ # install -m0600 /dev/null /tmp/boot.tar
+ # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
+ # umount /boot/efi
+ # umount /boot
+ # mount /boot
+ mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
and again: no sign that the partition is in use:
- # grep nvme0n1p3 /proc/mounts
- # lsof | grep nvme0n1p3
- # lsof nvme0n1p3
- # fuser nvme0n1p3
+ # grep nvme0n1p3 /proc/mounts
+ # lsof | grep nvme0n1p3
+ # lsof nvme0n1p3
+ # fuser nvme0n1p3
=======================================================
So another try (reboot again):
- # mount -o remount,ro /boot
- # install -m0600 /dev/null /tmp/boot.tar
- # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
- # umount /boot/efi
- # mount -o remount,rw /boot # <<<<------- notice this one!
- # umount /boot
- # mount /boot # <<<<------- Now it succeeds!
+ # mount -o remount,ro /boot
+ # install -m0600 /dev/null /tmp/boot.tar
+ # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
+ # umount /boot/efi
+ # mount -o remount,rw /boot # <<<<------- notice this one!
+ # umount /boot
+ # mount /boot # <<<<------- Now it succeeds!
Now do the same, but without the "mount -oremount,rw /boot"
- # mount -o remount,ro /boot
- # install -m0600 /dev/null /tmp/boot.tar
- # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
- # umount /boot
- # mount /boot
- mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
+ # mount -o remount,ro /boot
+ # install -m0600 /dev/null /tmp/boot.tar
+ # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
+ # umount /boot
+ # mount /boot
+ mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
Oooops! Now system is messed up again. No way to "clean up the mess.
Getting closer? Partition originally was mounted read-write. umount
while remounted read-only seems to mess up things and mount looses track
of was is going on..
Thus, another try:
=====================================================
Again freshly booted. Omit the "mount -oremount,ro /boot" this time:
- # install -m0600 /dev/null /tmp/boot.tar
- # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
- # umount -v /boot/efi
- umount: /boot/efi unmounted
- # umount -v /boot
- umount: /boot unmounted
- # cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
- WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
+ # install -m0600 /dev/null /tmp/boot.tar
+ # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
+ # umount -v /boot/efi
+ umount: /boot/efi unmounted
+ # umount -v /boot
+ umount: /boot unmounted
+ # cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
+ WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
- WARNING!
- ========
- This will overwrite dataon /dev/nvme0n1p3 irrevocably.
+ WARNING!
+ ========
+ This will overwrite data on /dev/nvme0n1p3 irrevocably.
- Are you sure? (Type 'yes' in capital letters); YES
- Enter Passphrase for /dev/nvme0n1p3:
- Verify Passphrase:
- Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
+ Are you sure? (Type 'yes' in capital letters); YES
+ Enter Passphrase for /dev/nvme0n1p3:
+ Verify Passphrase:
+ Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
failed again, but
- # mount /boot # <<<<------- Now it succeeds!
- # umount /boot # <<<<------- Now it succeeds!
- # mount /boot # <<<<------- Now it succeeds!
-
+ # mount /boot # <<<<------- Now it succeeds!
+ # umount /boot # <<<<------- Now it succeeds!
+ # mount /boot # <<<<------- Now it succeeds!
So, how can I track this one down further?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2040076
Title:
Something is holding a lock on unmounted partition
Status in util-linux package in Ubuntu:
New
Bug description:
Following instructions from https://discourse.maas.io/t/deploying-
servers-with-full-disk-encryption-luks2/3286 to get full disk
encryption with ubuntu-22.04-server
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount -v /boot/efi
umount: /boot/efi unmounted
# umount -v /boot
umount: /boot unmounted
# cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
WARNING!
========
This will overwrite data on /dev/nvme0n1p3 irrevocably.
Are you sure? (Type 'yes' in capital letters); YES
Enter Passphrase for /dev/nvme0n1p3:
Verify Passphrase:
Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
Uh, not what I expected. So i start to investigate:
# grep nvme0n1p3 /proc/mounts
# lsof | grep nvme0n1p3
# lsof nvme0n1p3
# fuser nvme0n1p3
So:
- umount succeeded
- /proc/mounts, lsof, fuser don's show any sign that the partition is in use
BUT:
strace cryptsetup luksFormat --type luks1 /dev/nvme0n1p3 2>&1 |
egrep '(nvme0n1p3|close)'
shows
openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|ODIRECT) = 3
close(3)
openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|O_NONBLOCK|OCLOEXEC) = 3
close(3)
openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDWR|OEXCL) = -1 EBUSY (Device or resource busy)
So there must be something holding a hand on the partition. But
there's no trace of what it might be
=======================================================
Another try (after fresh boot):
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount /boot/efi
# umount /boot
# mount /boot
mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
and again: no sign that the partition is in use:
# grep nvme0n1p3 /proc/mounts
# lsof | grep nvme0n1p3
# lsof nvme0n1p3
# fuser nvme0n1p3
=======================================================
So another try (reboot again):
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount /boot/efi
# mount -o remount,rw /boot # <<<<------- notice this one!
# umount /boot
# mount /boot # <<<<------- Now it succeeds!
Now do the same, but without the "mount -oremount,rw /boot"
# mount -o remount,ro /boot
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount /boot
# mount /boot
mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy
Oooops! Now system is messed up again. No way to "clean up the mess.
Getting closer? Partition originally was mounted read-write. umount
while remounted read-only seems to mess up things and mount looses
track of was is going on..
Thus, another try:
=====================================================
Again freshly booted. Omit the "mount -oremount,ro /boot" this time:
# install -m0600 /dev/null /tmp/boot.tar
# tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
# umount -v /boot/efi
umount: /boot/efi unmounted
# umount -v /boot
umount: /boot unmounted
# cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.
WARNING!
========
This will overwrite data on /dev/nvme0n1p3 irrevocably.
Are you sure? (Type 'yes' in capital letters); YES
Enter Passphrase for /dev/nvme0n1p3:
Verify Passphrase:
Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.
failed again, but
# mount /boot # <<<<------- Now it succeeds!
# umount /boot # <<<<------- Now it succeeds!
# mount /boot # <<<<------- Now it succeeds!
So, how can I track this one down further?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2040076/+subscriptions
More information about the foundations-bugs
mailing list