[Bug 1780332] Re: vaultlocker does not ensure that udev is triggered to create /dev/disk/by-uuid/<uuid-in-luks-header> symlink and fails

Sat Sep 15 15:15:25 UTC 2018

asi,

Thanks for the guidance.

Attached some outputs.

> Look for "dmsetup udevcomplete" call in udev rules.

ubuntu at maas-vhost6:/lib/udev/rules.d$ grep -RiP udevcomplete
55-dm.rules:ENV{DM_COOKIE}=="?*", RUN+="/sbin/dmsetup udevcomplete $env{DM_COOKIE}"

> Sometimes it is hidden by the fact that libdevampper could fallback to device internal nodes creation because it verifies that udev nodes were created (the old way used when you compile it without udev support).

At least I can see that it is not disabled explicitly for xenial in the
build scripts:

https://git.launchpad.net/~usd-import-
team/ubuntu/+source/cryptsetup/tree/configure?h=ubuntu/xenial-
updates#n15870

# Check whether --enable-udev was given.
if test "${enable_udev+set}" = set; then :
  enableval=$enable_udev;
else
  enable_udev=yes
fi

https://git.launchpad.net/~usd-import-
team/ubuntu/+source/cryptsetup/tree/debian/rules?h=ubuntu/xenial-
updates#n43

> This must be the last call in udev chain rules related to device-mapper devices. (Run cryptsetup with --debug and you will see that sync point.)

Do you remember anything specific? I do not see anything above this

# Key length 32, device size 16777216 sectors, header size 2050 sectors.
# Releasing crypt device /dev/sdb context.
# Releasing device-mapper backend.
# Unlocking memory.
Command successful.
14:09:07.630619660

in the attached output that would definitively resemble a sync point
("Releasing context..." could be it but I am not sure without looking at
the code).

journalctl -u systemd-udevd.service -f -o short-precise

# ...
Sep 15 14:09:07.634172 maas-vhost6 systemd-udevd[3793]: IMPORT builtin 'blkid' /lib/udev/rules.d/60-persistent-storage.rules:76
Sep 15 14:09:07.634285 maas-vhost6 systemd-udevd[3793]: probe /dev/sdb raid offset=0
Sep 15 14:09:07.634399 maas-vhost6 systemd-udevd[3793]: LINK 'disk/by-uuid/fcdd1397-8fb7-410c-b343-a7bb1a2f83d0' /lib/udev/rules.d/60-persistent-storage.rules:79

# ...

Sep 15 14:09:07.635434 maas-vhost6 systemd-udevd[3793]: found 'b8:16' claiming '/run/udev/links/\x2fdisk\x2fby-uuid\x2ffcdd1397-8fb7-410c-b343-a7bb1a2f83d0'
Sep 15 14:09:07.635547 maas-vhost6 systemd-udevd[3793]: creating link '/dev/disk/by-uuid/fcdd1397-8fb7-410c-b343-a7bb1a2f83d0' to '/dev/sdb'
Sep 15 14:09:07.635659 maas-vhost6 systemd-udevd[3793]: preserve already existing symlink '/dev/disk/by-uuid/fcdd1397-8fb7-410c-b343-a7bb1a2f83d0' to '../../sdb'

** Attachment added: "udev-dmcrypt-debug-test-xenial-15-09-2018.txt"
   https://bugs.launchpad.net/vaultlocker/+bug/1780332/+attachment/5189138/+files/udev-dmcrypt-debug-test-xenial-15-09-2018.txt

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to cryptsetup in Ubuntu.
https://bugs.launchpad.net/bugs/1780332

Title:
  vaultlocker does not ensure that udev is triggered to create /dev/disk
  /by-uuid/<uuid-in-luks-header> symlink and fails

Status in vaultlocker:
  Fix Released
Status in cryptsetup package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  When an encrypted device is setup up a UUID (osd_fsid) is passed from
  the charm to be used in the cryptsetup command which accepts a UUID to
  place into the LUKS header (shown in cryptsetup luksDump <path-to-
  block-device>).

  https://github.com/openstack/charm-ceph-osd/blob/stable/18.05/lib/ceph/utils.py#L1788-L1804 
  UUID comes from osd_fsid

  https://github.com/openstack-charmers/vaultlocker/blob/8c9cb85dc3ed5dbf18c66a810d189a5230d85c34/vaultlocker/shell.py#L69-L80
  # else statement is used here
       block_uuid = str(uuid.uuid4()) if not args.uuid else args.uuid

       dmcrypt.luks_format(key, block_device, block_uuid) # creates a LUKS header
  # ...
       dmcrypt.luks_open(key, block_uuid) # sets up a device with device mapper decrypting it via dmcrypt

  https://github.com/openstack-
  charmers/vaultlocker/blob/d813233179bdf2eec8ed101c702a8e552a966f44/vaultlocker/dmcrypt.py#L44-L56

  This UUID is visible in blkid output

  /dev/sdc: UUID="<luks-header-uuid>" TYPE="crypto_LUKS"

  and a udev rule exists to create a /dev/disk/by-uuid/<luks-header-
  uuid> symlink (which is normally used for filesystem -> block device
  resolution)

  https://git.launchpad.net/~usd-import-team/ubuntu/+source/lvm2/tree/udev/13-dm-disk.rules.in#n25
  ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"

  Where vaultlocker fails is in luks_open command (right after luks_format) 

   # cryptsetup --batch-mode --key-file - open UUID=<luks-header-uuid>
  crypt-<luks-header-uuid> --type luks

  because it tries to access /dev/disk/by-uuid/<luks-header-uuid> which
  does not exist.

  This happens since udev rules are not re-triggered to create this
  symlink after a LUKS device is created.

  Solution: call the command below after luks_format before luks_open

  udevadm settle --exit-if-exists=/dev/disk/by-uuid/<luks-header-uuid-
  equal-to-osd-fsid>

To manage notifications about this bug go to:
https://bugs.launchpad.net/vaultlocker/+bug/1780332/+subscriptions