[Bug 2150762] [NEW] Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3 subinterpreter incompatibility)

[utkarsh bhatt]

Public bug reported:

Package: ceph 20.2.0-0ubuntu2 (resolute)
Related: python3-cryptography 46.0.5-1ubuntu2

On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
the `mgr module enable cephadm` step with:

  Error ENOENT: module 'cephadm' reports that it cannot run on the active
  manager daemon: PyO3 modules do not yet support subinterpreters, see
  https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
against a rock built from the same archive. The mgr daemon log
(via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
traceback when the cephadm module is loaded in a subinterpreter:

  File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
      from .module import CephadmOrchestrator
  File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
      from cephadm.cert_mgr import CertMgr
  File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
      from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
  File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
      from cryptography import x509
  File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
      in <module>
      from cryptography.x509 import certificate_transparency, verification
  File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
      line 8, in <module>
      from cryptography.hazmat.bindings._rust import x509 as rust_x509
  ImportError: PyO3 modules do not yet support subinterpreters,
      see https://github.com/PyO3/pyo3/issues/576

The same failure mode applies to the diskprediction_local mgr module,
where the failing import is `numpy._core._multiarray_umath` (also
PyO3-built in resolute's python3-numpy 2.3.x).

Root cause:
- Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
  subinterpreter by default.
- Resolute's python3-cryptography 46.0.5 includes Rust extensions built
  with PyO3, which raises ImportError when imported into a subinterpreter
  (PyO3 currently does not support PEP 684 multi-interpreter modules).

Upstream fix:
ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
merged 2025-11-07) inverts the default: mgr modules load in the main
interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
by reading src/common/options/mgr.yaml.in at both tags). It is therefore
absent from resolute's ceph 20.2.0-0ubuntu2.

Requested action:
Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
`mgr module enable cephadm` succeeds on resolute. Until that lands,
canonical/ceph-containers's Tentacle rock works around this by switching
back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
MicroCeph uses).

Reproduction (LXD VM):
  1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
  2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
  3. cephadm --image <any-tentacle-image> bootstrap --mon-ip <ip> \
       --single-host-defaults --skip-dashboard --skip-monitoring-stack
  4. Observe the PyO3 error during the `mgr module enable cephadm` step.

** Affects: ceph (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: pyo3 resolute tentacle

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2150762

Title:
  Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3
  subinterpreter incompatibility)

Status in ceph package in Ubuntu:
  New

Bug description:
  Package: ceph 20.2.0-0ubuntu2 (resolute)
  Related: python3-cryptography 46.0.5-1ubuntu2

  On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
  the `mgr module enable cephadm` step with:

    Error ENOENT: module 'cephadm' reports that it cannot run on the active
    manager daemon: PyO3 modules do not yet support subinterpreters, see
    https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

  Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
  against a rock built from the same archive. The mgr daemon log
  (via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
  traceback when the cephadm module is loaded in a subinterpreter:

    File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
        from .module import CephadmOrchestrator
    File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
        from cephadm.cert_mgr import CertMgr
    File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
        from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
    File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
        from cryptography import x509
    File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
        in <module>
        from cryptography.x509 import certificate_transparency, verification
    File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
        line 8, in <module>
        from cryptography.hazmat.bindings._rust import x509 as rust_x509
    ImportError: PyO3 modules do not yet support subinterpreters,
        see https://github.com/PyO3/pyo3/issues/576

  The same failure mode applies to the diskprediction_local mgr module,
  where the failing import is `numpy._core._multiarray_umath` (also
  PyO3-built in resolute's python3-numpy 2.3.x).

  Root cause:
  - Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
    subinterpreter by default.
  - Resolute's python3-cryptography 46.0.5 includes Rust extensions built
    with PyO3, which raises ImportError when imported into a subinterpreter
    (PyO3 currently does not support PEP 684 multi-interpreter modules).

  Upstream fix:
  ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
  merged 2025-11-07) inverts the default: mgr modules load in the main
  interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
  is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
  by reading src/common/options/mgr.yaml.in at both tags). It is therefore
  absent from resolute's ceph 20.2.0-0ubuntu2.

  Requested action:
  Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
  `mgr module enable cephadm` succeeds on resolute. Until that lands,
  canonical/ceph-containers's Tentacle rock works around this by switching
  back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
  MicroCeph uses).

  Reproduction (LXD VM):
    1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
    2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
    3. cephadm --image <any-tentacle-image> bootstrap --mon-ip <ip> \
         --single-host-defaults --skip-dashboard --skip-monitoring-stack
    4. Observe the PyO3 error during the `mgr module enable cephadm` step.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2150762/+subscriptions