[Bug 2150762] Re: Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3 subinterpreter incompatibility)

[utkarsh bhatt]

Upstream references for the fix:

- Pull request: https://github.com/ceph/ceph/pull/66244
  ("mgr: add config to load modules in main interpreter instead of subinterpreter")
  Merged 2026-03-20 into ceph/ceph master.

- Commit: 239b0dc8a9c42449ee1faa1bf78bdcc380345ae2
  https://github.com/ceph/ceph/commit/239b0dc8a9c42449ee1faa1bf78bdcc380345ae2

The change adds a new mgr config `mgr_subinterpreter_modules` and inverts
the default so mgr modules load in the main interpreter; modules listed in
the config (or `*`) opt back into a subinterpreter.

The fix is currently in master only. Verified absent from the v20.2.0 and
v20.2.1 release tags by reading src/common/options/mgr.yaml.in at both
tags.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2150762

Title:
  Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3
  subinterpreter incompatibility)

Status in ceph package in Ubuntu:
  New

Bug description:
  Package: ceph 20.2.0-0ubuntu2 (resolute)
  Related: python3-cryptography 46.0.5-1ubuntu2

  On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
  the `mgr module enable cephadm` step with:

    Error ENOENT: module 'cephadm' reports that it cannot run on the active
    manager daemon: PyO3 modules do not yet support subinterpreters, see
    https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

  Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
  against a rock built from the same archive. The mgr daemon log
  (via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
  traceback when the cephadm module is loaded in a subinterpreter:

    File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
        from .module import CephadmOrchestrator
    File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
        from cephadm.cert_mgr import CertMgr
    File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
        from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
    File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
        from cryptography import x509
    File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
        in <module>
        from cryptography.x509 import certificate_transparency, verification
    File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
        line 8, in <module>
        from cryptography.hazmat.bindings._rust import x509 as rust_x509
    ImportError: PyO3 modules do not yet support subinterpreters,
        see https://github.com/PyO3/pyo3/issues/576

  The same failure mode applies to the diskprediction_local mgr module,
  where the failing import is `numpy._core._multiarray_umath` (also
  PyO3-built in resolute's python3-numpy 2.3.x).

  Root cause:
  - Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
    subinterpreter by default.
  - Resolute's python3-cryptography 46.0.5 includes Rust extensions built
    with PyO3, which raises ImportError when imported into a subinterpreter
    (PyO3 currently does not support PEP 684 multi-interpreter modules).

  Upstream fix:
  ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
  merged 2025-11-07) inverts the default: mgr modules load in the main
  interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
  is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
  by reading src/common/options/mgr.yaml.in at both tags). It is therefore
  absent from resolute's ceph 20.2.0-0ubuntu2.

  Requested action:
  Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
  `mgr module enable cephadm` succeeds on resolute. Until that lands,
  canonical/ceph-containers's Tentacle rock works around this by switching
  back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
  MicroCeph uses).

  Reproduction (LXD VM):
    1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
    2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
    3. cephadm --image <resolute-based-tentacle-OCI> bootstrap --mon-ip <ip> \
         --single-host-defaults --skip-dashboard --skip-monitoring-stack
    4. Observe the PyO3 error during the `mgr module enable cephadm` step.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2150762/+subscriptions