[Bug 1780332] Re: vaultlocker does not ensure that udev is triggered to create /dev/disk/by-uuid/<uuid-in-luks-header> symlink and fails

[Dmitrii Shcherbakov]

As a hint on reproducing it, it may be a problem on xenial only (which
we used for that deployment) for clean devices.

On a bionic VM it seems like the symlink is created but it's not clear
if luksFormat returns before that symlink gets created or after - this
is the important part because the automation tries to use that symlink
right after `cryptsetup luksFormat <dev>` exits.

tree /sys/class/block
https://paste.ubuntu.com/p/vhjjzdytH7/

uname -a
Linux maas-vhost6 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

ubuntu at maas-vhost6:~$ tree /dev/disk/by-uuid/
/dev/disk/by-uuid/
└── d26a75c9-15f7-41de-8c0e-20f795ed5729 -> ../../sda1

0 directories, 1 file
ubuntu at maas-vhost6:~$ lsblk 
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda       8:0    0    64G  0 disk 
└─sda1    8:1    0    64G  0 part /
sdb       8:16   0     8G  0 disk 
sdc       8:32   0 102.4M  0 disk 
sdd       8:48   0 102.4M  0 disk 
sde       8:64   0 102.4M  0 disk 
vda     252:0    0 102.4M  0 disk 
vdb     252:16   0 102.4M  0 disk 
nvme0n1 259:0    0    20G  0 disk 
nvme1n1 259:1    0    20G  0 disk 
ubuntu at maas-vhost6:~$ sudo cryptsetup luksFormat /dev/sdb 

WARNING!
========
This will overwrite data on /dev/sdb irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sdb: 
Verify passphrase: 

ubuntu at maas-vhost6:~$ tree /dev/disk/by-uuid/
/dev/disk/by-uuid/
├── a82ddfe0-7de8-4c6c-aaca-a074f000b746 -> ../../sdb
└── d26a75c9-15f7-41de-8c0e-20f795ed5729 -> ../../sda1

0 directories, 2 files


sudo cryptsetup luksFormat /dev/sdb 

WARNING!
========
This will overwrite data on /dev/sdb irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sdb: 
Verify passphrase: 


ubuntu at maas-vhost6:~$ sudo cryptsetup luksDump /dev/sdb | grep UUID
UUID:          	21bacaf0-9eea-4809-9b1c-6f4d7e614f5b


ubuntu at maas-vhost6:~$ tree /dev/disk/by-uuid/
/dev/disk/by-uuid/
├── 21bacaf0-9eea-4809-9b1c-6f4d7e614f5b -> ../../sdb
└── d26a75c9-15f7-41de-8c0e-20f795ed5729 -> ../../sda1

0 directories, 2 files

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to cryptsetup in Ubuntu.
https://bugs.launchpad.net/bugs/1780332

Title:
  vaultlocker does not ensure that udev is triggered to create /dev/disk
  /by-uuid/<uuid-in-luks-header> symlink and fails

Status in vaultlocker:
  Fix Released
Status in cryptsetup package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  When an encrypted device is setup up a UUID (osd_fsid) is passed from
  the charm to be used in the cryptsetup command which accepts a UUID to
  place into the LUKS header (shown in cryptsetup luksDump <path-to-
  block-device>).

  https://github.com/openstack/charm-ceph-osd/blob/stable/18.05/lib/ceph/utils.py#L1788-L1804 
  UUID comes from osd_fsid

  https://github.com/openstack-charmers/vaultlocker/blob/8c9cb85dc3ed5dbf18c66a810d189a5230d85c34/vaultlocker/shell.py#L69-L80
  # else statement is used here
       block_uuid = str(uuid.uuid4()) if not args.uuid else args.uuid

       dmcrypt.luks_format(key, block_device, block_uuid) # creates a LUKS header
  # ...
       dmcrypt.luks_open(key, block_uuid) # sets up a device with device mapper decrypting it via dmcrypt

  https://github.com/openstack-
  charmers/vaultlocker/blob/d813233179bdf2eec8ed101c702a8e552a966f44/vaultlocker/dmcrypt.py#L44-L56

  This UUID is visible in blkid output

  /dev/sdc: UUID="<luks-header-uuid>" TYPE="crypto_LUKS"

  and a udev rule exists to create a /dev/disk/by-uuid/<luks-header-
  uuid> symlink (which is normally used for filesystem -> block device
  resolution)

  https://git.launchpad.net/~usd-import-team/ubuntu/+source/lvm2/tree/udev/13-dm-disk.rules.in#n25
  ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"

  
  Where vaultlocker fails is in luks_open command (right after luks_format) 

   # cryptsetup --batch-mode --key-file - open UUID=<luks-header-uuid>
  crypt-<luks-header-uuid> --type luks

  because it tries to access /dev/disk/by-uuid/<luks-header-uuid> which
  does not exist.

  This happens since udev rules are not re-triggered to create this
  symlink after a LUKS device is created.

  Solution: call the command below after luks_format before luks_open

  udevadm settle --exit-if-exists=/dev/disk/by-uuid/<luks-header-uuid-
  equal-to-osd-fsid>

To manage notifications about this bug go to:
https://bugs.launchpad.net/vaultlocker/+bug/1780332/+subscriptions