[Bug 1804261] Re: Ceph OSD units requires reboot if they boot before vault (and if not unsealed with 150s)

[dongdong tao]

Hi All,

I can confirm this release fixed the bug, I used below steps to test

1. Deployed a ceph cluster with vault
2. Upgrade all the ceph packages to 12.2.13 at bionic-proposed
3. Add "Environment=CEPH_VOLUME_SYSTEMD_TRIES=2000" at /lib/systemd/system/ceph-volume at .service for some osd node
4. Reboot vault node, then reboot osd node
5. Wait for half an hour
6. Unseal vault
7. All the osd node with the setting "Environment=CEPH_VOLUME_SYSTEMD_TRIES=2000" can come online automatically, while the other osds without this setting can not come online, which is expected.

-Cheers

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1804261

Title:
  Ceph OSD units requires reboot if they boot before vault (and if not
  unsealed with 150s)

Status in OpenStack ceph-osd charm:
  Invalid
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Committed
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Bionic:
  Fix Committed
Status in ceph source package in Disco:
  Won't Fix
Status in ceph source package in Eoan:
  Fix Released
Status in ceph source package in Focal:
  Fix Released

Bug description:
  [Impact]
  Various configuration option values that are read from environment variables are incorrectly parsed as strings rather than ints which means that for certain deployment use-cases, the timeouts for starting the ceph-osd volume units cannot be increased to accommodate dependencies starting first.

  [Test Case]
  Deploy ceph with vault for key management
  set a systemd override for ceph-volume@
  Environment=CEPH_VOLUME_SYSTEMD_TRIES=2000
  Seal vault units (by restarting the vault service)
  Reboot ceph-osd machines - Environment override is ignored as its not correctly parsed.

  [Regression Potential]
  Low - this fix has been accept upstream in later releases.

  
  [Original Bug Report]
  In a scenario where Ceph is encrypted and using Vault as the keymanager, in a scenario where vault and ceph are both stopped, any OSDs on the unit(s) affected will require a further reboot if they try to start before vault is unsealed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-osd/+bug/1804261/+subscriptions