[Bug 1828617] Re: Hosts randomly 'losing' disks, breaking ceph-osd service enumeration

[David A. Desrosiers]

This manifests itself as the following, as reported by lsblk(1). Note
the missing Ceph LVM volume on the 6th NVME disk:

$ cat sos_commands/block/lsblk
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                                                                     8:0    0  1.8T  0 disk 
|-sda1                                                                                                  8:1    0  512M  0 part /boot/efi
`-sda2                                                                                                  8:2    0  1.8T  0 part 
  |-foobar--vg-root                                                                              253:0    0  1.8T  0 lvm  /
  `-foobar--vg-swap_1                                                                            253:1    0  976M  0 lvm  [SWAP]
nvme0n1                                                                                               259:0    0  1.8T  0 disk 
`-ceph--c576f63e--dfd4--48f7--9d60--6a7708cbccf6-osd--block--9fdd78b2--0745--47ae--b8d4--04d9803ab448 253:6    0  1.8T  0 lvm  
nvme1n1                                                                                               259:1    0  1.8T  0 disk 
`-ceph--6eb6565f--6392--44a8--9213--833b09f7c0bc-osd--block--a7d3629c--724f--4218--9d15--593ec64781da 253:5    0  1.8T  0 lvm  
nvme2n1                                                                                               259:2    0  1.8T  0 disk 
`-ceph--c14f9ee5--90d0--4306--9b18--99576516f76a-osd--block--bbf5bc79--edea--4e43--8414--b5140b409397 253:4    0  1.8T  0 lvm  
nvme3n1                                                                                               259:3    0  1.8T  0 disk 
`-ceph--a821146b--7674--4bcc--b5e9--0126c4bd5e3b-osd--block--b9371499--ff99--4d3e--ab3f--62ec3cf918c4 253:3    0  1.8T  0 lvm  
nvme4n1                                                                                               259:4    0  1.8T  0 disk 
`-ceph--2e39f75a--5d2a--49ee--beb1--5d0a2991fd6c-osd--block--a1be083e--1fa7--4397--acfa--2ff3d3491572 253:2    0  1.8T  0 lvm  
nvme5n1                                                                                               259:5    0  1.8T  0 disk

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1828617

Title:
  Hosts randomly 'losing' disks, breaking ceph-osd service enumeration

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  Ubuntu 18.04.2 Ceph deployment.

  Ceph OSD devices utilizing LVM volumes pointing to udev-based physical devices.
  LVM module is supposed to create PVs from devices using the links in /dev/disk/by-dname/ folder that are created by udev.
  However on reboot it happens (not always, rather like race condition) that Ceph services cannot start, and pvdisplay doesn't show any volumes created. The folder /dev/disk/by-dname/ however has all necessary device created by the end of boot process.

  The behaviour can be fixed manually by running "#/sbin/lvm pvscan
  --cache --activate ay /dev/nvme0n1" command for re-activating the LVM
  components and then the services can be started.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1828617/+subscriptions