[Bug 2064717] Re: ceph-volume needs "packaging" and "ceph" modules

Tue Aug 27 08:55:06 UTC 2024

I suspect the reason this was not picked up in the SRU test, is possibly
related to code from the Squid charm being used in the test instead of
the Reef Charm.

The squid charm merged a "tactical fix" to manually install
python3-packaging in this change:
https://review.opendev.org/c/openstack/charm-ceph-osd/+/918992

But not for Reef, it was originally proposed but abandoned when we
thought it wasn't needed for Reef:
https://review.opendev.org/c/openstack/charm-ceph-osd/+/919794

The Squid charm supports running/installing reef because you're expected
to upgrade the charm before Ceph itself, to orchestrate an upgrade. So
both the Reef and Squid charm branches have a test for Reef
(tests/bundles/jammy-bobcat).

IMHO merging this charm change was a bad idea and it should be reverted
once all the packages are fixed. The package should simply have been
fixed immediately in the first instance.

While I can appreciate this might have been done as a stop-gap to get
the charm CI working while the issue was not yet fixed in an SRU, the
problem is that we are using the charm tests to verify the SRU of the
Ubuntu package which is potentially (and actually, even in the cloud-
archive) used by people without the charms, so this is likely to hide
such an issue as it did here. It also means we don't have a functional
test to actually test that the issue is fixed, both in the Reef and
Squid SRUs.

I can't quite figure out exactly how this test was done though. The
original message said it was tested with the ceph-osd charm tests, but
the zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest test listed
in the output only exists in charm-ceph-mon.   Then those tests all use
the reef branch of the charm.. I am guessing maybe since we had to test
with bobcat-proposed that the squid branch was used by with openstack-
origin overriden to bobcat-proposed or something?  Luciano: Would be
great if you can clarify/reverse engineer exactly how you managed to do
this so we can learn for next time. I also wonder if we'd be better
using charmed-openstack-tester or something like that, instead of purely
the charm-ceph-mon tests, for validating SRUs?

A few possible lessons for future SRU verificaiton:
- We need to ensure we verify SRUs with all GIT/charmhub branches of the charm that support a release. So generally that would be both the matching and newer version. It's not sufficient to check with only one of those.
- Thinking more about the charm users that are the majority, I think ideally we also need to run both the charmhub stable AND candidate branches for both of those releases. Currently the test bundles use the '/edge' channel (which maps to candidate) and would only test the candidate charm, and won't show up if we're about to release a package that is broken wtith the stable charms. Especially for the latest release of Ceph, due to the Solutions QA process, sometimes the stable channel is lagging the edge channel by weeks or even months. So this is not unlikely.
- Using the charm tests to verify the Ubuntu package in general has some limits, in that it may miss scenarios that would still effect non-charm users. I am not proposing we stop using it, but we should be aware of that.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2064717

Title:
  ceph-volume needs "packaging" and "ceph" modules

Status in Ceph OSD Charm:
  Fix Released
Status in Ceph OSD Charm reef series:
  New
Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive bobcat series:
  New
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Noble:
  Fix Released
Status in ceph source package in Oracular:
  Fix Released

Bug description:
  [ Impact ]
  ceph-volume tool is not usable directly after install due to a missing dependencies.

  [ Test Plan ]
  sudo apt install ceph-volume
  ceph-volume --help

  [ Where problems could occur ]
  The missing packaging module is immediately obvious - the ceph dependency less so as ceph-volume is usually installed with ceph-osd (which already pulls this in).  Direct users of ceph-volume will get new depends pulled in.

  [ Original Bug Report ]
  The ceph-volume program needs python3-packaging but it looks like we're not installing it in jammy-caracal

  https://github.com/ceph/ceph/pull/54423/commits/0985e201342fa53c014a811156aed661b4b8f994
  https://openstack-ci-reports.ubuntu.com/artifacts/dcf/917920/4/check/jammy-caracal/dcf9973/index.html

  Traceback excerpt:

  2024-05-02 19:31:54.624912 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 INFO unit.ceph-osd/0.juju-log server.go:316 mon:27: osdize cmd: ['ceph-volume', 'lvm', 'create', '--osd-fsid', 'aef29aff-df24-4bb8-bfb3-bcd607761b2e', '--bluestore', '--data', 'ceph-aef29aff-df24-4bb8-bfb3-bcd607761b2e/osd-block-aef29aff-df24-4bb8-bfb3-bcd607761b2e']
  2024-05-02 19:31:54.624972 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 Traceback (most recent call last):
  2024-05-02 19:31:54.624990 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/sbin/ceph-volume", line 33, in <module>
  2024-05-02 19:31:54.625002 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')())
  2024-05-02 19:31:54.625014 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/sbin/ceph-volume", line 25, in importlib_load_entry_point
  2024-05-02 19:31:54.625189 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     return next(matches).load()
  2024-05-02 19:31:54.625210 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
  2024-05-02 19:31:54.625222 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     module = import_module(match.group('module'))
  2024-05-02 19:31:54.625234 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
  2024-05-02 19:31:54.625247 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     return _bootstrap._gcd_import(name[level:], package, level)
  2024-05-02 19:31:54.625259 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  2024-05-02 19:31:54.625541 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  2024-05-02 19:31:54.625598 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  2024-05-02 19:31:54.625611 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  2024-05-02 19:31:54.625622 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  2024-05-02 19:31:54.625633 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  2024-05-02 19:31:54.625975 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 9, in <module>
  2024-05-02 19:31:54.626007 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     from ceph_volume import log, devices, configuration, conf, exceptions, terminal, inventory, drive_group, activate
  2024-05-02 19:31:54.626057 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3/dist-packages/ceph_volume/devices/__init__.py", line 1, in <module>
  2024-05-02 19:31:54.626072 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     from . import lvm, simple, raw # noqa
  2024-05-02 19:31:54.626280 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/__init__.py", line 1, in <module>
  2024-05-02 19:31:54.626313 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     from .main import LVM # noqa
  2024-05-02 19:31:54.626327 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 4, in <module>
  2024-05-02 19:31:54.626339 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     from . import activate
  2024-05-02 19:31:54.626359 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/activate.py", line 9, in <module>
  2024-05-02 19:31:54.626371 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     from ceph_volume.util import encryption as encryption_utils
  2024-05-02 19:31:54.626465 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60   File "/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py", line 9, in <module>
  2024-05-02 19:31:54.626490 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60     from packaging import version
  2024-05-02 19:31:54.626502 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 ModuleNotFoundError: No module named 'packaging'

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-osd/+bug/2064717/+subscriptions