[Bug 1931004] Re: Add support for Pacific to RBD driver

Tue Sep 28 14:03:28 UTC 2021

I've upgraded recently environment to latest stable/victoria build, and I am not able to boot from rbd volume - actually cinder cannot fetch flance images from RBD - it worked on train and ussuri. Ceph is Pacific (16.2.6) - https://paste.opendev.org/raw/809512/
Looks like related to this one https://review.opendev.org/c/openstack/cinder/+/808475 (it is not merged to ussuri yet)
I've tested rbd_store_chunk_size = 8 (the same as in glance). But same issue.

I've used older cinder Victoria build and it worked.

           |__Flow 'volume_create_manager': rbd.PermissionError: [errno 1] error opening image b'7e0beafa-f1a2-403e-8394-f5e6900d0785' at snapshot None
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager Traceback (most recent call last):
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     result = task.execute(**arguments)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1164, in execute
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     **volume_spec)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/utils.py", line 694, in _wrapper
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     return r.call(f, *args, **kwargs)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 356, in iter
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     return fut.result()
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     return self.__get_result()
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     raise self._exception
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     result = fn(*args, **kwargs)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1032, in _create_from_image
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     image_service)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py", line 1584, in clone_image
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     volume_update = self._clone(volume, pool, image, snapshot)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py", line 1034, in _clone
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     stripe_unit = self._get_stripe_unit(src_client.ioctx, src_image)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py", line 1018, in _get_stripe_unit
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager     image = self.rbd.Image(ioctx, volume_name)
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager   File "rbd.pyx", line 2960, in rbd.Image.__init__
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager rbd.PermissionError: [errno 1] error opening image b'7e0beafa-f1a2-403e-8394-f5e6900d0785' at snapshot None
2021-09-22 18:23:19.974 72 ERROR cinder.volume.manager

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to glance in Ubuntu.
https://bugs.launchpad.net/bugs/1931004

Title:
  Add support for Pacific to RBD driver

Status in Cinder:
  Fix Released
Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive wallaby series:
  New
Status in Ubuntu Cloud Archive xena series:
  New
Status in glance package in Ubuntu:
  Confirmed
Status in glance source package in Hirsute:
  Confirmed
Status in glance source package in Impish:
  Confirmed

Bug description:
  When using ceph pacific, volume-from-image operations where both
  glance and cinder are configured to use RBD result in an exception
  when calling clone():

      rbd.InvalidArgument: [errno 22] RBD invalid argument (error
  creating clone)

      ERROR cinder.volume.manager Traceback (most recent call last):
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
      ERROR cinder.volume.manager     result = task.execute(**arguments)
      ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1132, in execute
      ERROR cinder.volume.manager     model_update = self._create_from_image(context,
      ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/utils.py", line 638, in _wrapper
      ERROR cinder.volume.manager     return r.call(f, *args, **kwargs)
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 411, in call
      ERROR cinder.volume.manager     return self.__call__(*args, **kwargs)
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
      ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter
      ERROR cinder.volume.manager     return fut.result()
      ERROR cinder.volume.manager   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 438, in result
      ERROR cinder.volume.manager     return self.__get_result()
      ERROR cinder.volume.manager   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 390, in __get_result
      ERROR cinder.volume.manager     raise self._exception
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
      ERROR cinder.volume.manager     result = fn(*args, **kwargs)
      ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 998, in _create_from_image
      ERROR cinder.volume.manager     model_update, cloned = self.driver.clone_image(context,
      ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 1571, in clone_image
      ERROR cinder.volume.manager     volume_update = self._clone(volume, pool, image, snapshot)
      ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 1023, in _clone
      ERROR cinder.volume.manager     self.RBDProxy().clone(src_client.ioctx,
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
      ERROR cinder.volume.manager     result = proxy_call(self._autowrap, f, *args, **kwargs)
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
      ERROR cinder.volume.manager     rv = execute(f, *args, **kwargs)
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 129, in execute
      ERROR cinder.volume.manager     six.reraise(c, e, tb)
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/six.py", line 719, in reraise
      ERROR cinder.volume.manager     raise value
      ERROR cinder.volume.manager   File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 83, in tworker
      ERROR cinder.volume.manager     rv = meth(*args, **kwargs)
      ERROR cinder.volume.manager   File "rbd.pyx", line 698, in rbd.RBD.clone
      ERROR cinder.volume.manager rbd.InvalidArgument: [errno 22] RBD invalid argument (error creating clone)
      ERROR cinder.volume.manager

  In Pacific a check was added to make sure during a clone operation
  that the child's strip unit was not less than that of its parent.
  Failing this condition returns -EINVAL, which is then raised by
  python-rbd as an exception.  This maps to the 'order' argument in
  clone(), where order is log base 2 of the strip unit.  Ceph's default
  is 4 megabytes.  The reason we're seeing EINVAL exceptions in the
  Pacific CI is that: when Openstack is configured to use Ceph for both
  cinder and glance, volume-from-image tests fail because Glance's
  default stripe unit is 8 (distinctly larger than Cinder's 4). This
  results in an order calculation of 22, which is invalid for clone()
  (too small).

  I see two possible solutions and have proposed patches:

  1. Increase Cinder's default chunk size to match Glance's.  I think
  this makes sense for both consistency and performance.

  2. When doing a clone(), consider the configured chunk size /and/ the
  strip unit of the parent volume and choose the higher value.

  Either of these approaches prevent the failures we're seeing, I think
  they are both useful individually as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1931004/+subscriptions