[Bug 1828617] Re: Hosts randomly 'losing' disks, breaking ceph-osd service enumeration

[Corey Bryant]

I chatted with xav in IRC and he showed me a private link to the log
files. The ceph-volume-systemd.log.1 had timestamps of 2019-06-03 which
matches up with the last attempt (see comment #37).

I didn't find any logs from the new code in this log file. That likely
means one of the following: there were no wal/db devices found in lvs
tags (ie. 'sudo lvs -o lv_tags'), the new code isn't working, or the new
code wasn't installed.

I added a few more logs to the patch help understand better what's going
on, and that's rebuilding in the PPA.

I'm attaching all the relevant code to show the log messages to look
for.

** Attachment added: "snippet from src/ceph-volume/ceph_volume/systemd/main.py"
   https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1828617/+attachment/5269330/+files/main.py

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1828617

Title:
  Hosts randomly 'losing' disks, breaking ceph-osd service enumeration

Status in ceph package in Ubuntu:
  Triaged

Bug description:
  Ubuntu 18.04.2 Ceph deployment.

  Ceph OSD devices utilizing LVM volumes pointing to udev-based physical devices.
  LVM module is supposed to create PVs from devices using the links in /dev/disk/by-dname/ folder that are created by udev.
  However on reboot it happens (not always, rather like race condition) that Ceph services cannot start, and pvdisplay doesn't show any volumes created. The folder /dev/disk/by-dname/ however has all necessary device created by the end of boot process.

  The behaviour can be fixed manually by running "#/sbin/lvm pvscan
  --cache --activate ay /dev/nvme0n1" command for re-activating the LVM
  components and then the services can be started.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1828617/+subscriptions