[Bug 1903221] Re: ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-NN/: (11) Resource temporarily unavailable

[Nobuto Murata]

Adding Ubuntu Ceph packaging task here.

30-ceph-osd.conf file is owned by ceph-osd package as follows.

$ dpkg -S /etc/sysctl.d/30-ceph-osd.conf
ceph-osd: /etc/sysctl.d/30-ceph-osd.conf

However, as far as I see in 15.2.8-0ubuntu0.20.04.1/focal there is no
place in /var/lib/dpkg/info/ceph-osd.postinst activating the sysctl file
so it requires a reboot basically or manual intervention is required to
apply the value in the package. I think that's why fs.aio-max-nr wasn't
applied in the first place.

** Also affects: ceph (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1903221

Title:
  ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-
  NN/: (11) Resource temporarily unavailable

Status in OpenStack ceph-osd charm:
  Triaged
Status in ceph package in Ubuntu:
  New

Bug description:
  ussuri-focal, charms revision 20.10.

  ceph-osd fails to initialize 5 out of 36 OSDs on each storage node
  every time I redeploy.

  I have 3 storage nodes. Each node has 36x 4TB disks used as OSDs.
  Bcache for OSDs is also set up. Every time I redeploy the bundle, each
  storage node ends up with only 31 OSDs initialized. During the
  initialization of remaining 5 OSD, the following errors occur:

  unit-ceph-osd-0: 22:03:20 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 94 --monmap /var/lib/ceph/osd/ceph-94/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-94/ --osd-uuid fa372321-a0bc-4995-9936-227d2f663dd2 --setuser ceph --setgroup ceph 
  unit-ceph-osd-0: 22:03:20 WARNING unit.ceph-osd/0.mon-relation-changed  stderr: 2020-11-05T22:03:20.311+0000 7fc1d846dd80 -1 bluestore(/var/lib/ceph/osd/ceph-94/) _read_fsid unparsable uuid 
  unit-ceph-osd-0: 22:03:20 WARNING unit.ceph-osd/0.mon-relation-changed  stderr: 2020-11-05T22:03:20.319+0000 7fc1d846dd80 -1 bdev(0x5556cac96700 /var/lib/ceph/osd/ceph-94//block) _aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr 
  unit-ceph-osd-0: 22:03:20 WARNING unit.ceph-osd/0.mon-relation-changed  stderr: 2020-11-05T22:03:20.319+0000 7fc1d846dd80 -1 bluestore(/var/lib/ceph/osd/ceph-94/) _minimal_open_bluefs add block device(/var/lib/ceph/osd/ceph-94//block) returned: (11) Resource temporarily unavailable 
  unit-ceph-osd-0: 22:03:20 WARNING unit.ceph-osd/0.mon-relation-changed  stderr: 2020-11-05T22:03:20.603+0000 7fc1d846dd80 -1 bluestore(/var/lib/ceph/osd/ceph-94/) mkfs failed, (11) Resource temporarily unavailable 
  unit-ceph-osd-0: 22:03:20 WARNING unit.ceph-osd/0.mon-relation-changed  stderr: 2020-11-05T22:03:20.603+0000 7fc1d846dd80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (11) Resource temporarily unavailable 
  unit-ceph-osd-0: 22:03:20 WARNING unit.ceph-osd/0.mon-relation-changed  stderr: 2020-11-05T22:03:20.603+0000 7fc1d846dd80 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-94/: (11) Resource temporarily unavailable 

  I'm attaching the logs for both failed and successful OSD
  initialization.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-osd/+bug/1903221/+subscriptions