[Bug 2150762] Re: Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3 subinterpreter incompatibility)

utkarsh bhatt 2150762 at bugs.launchpad.net
Thu Apr 30 17:10:43 UTC 2026 

** Description changed:

  Package: ceph 20.2.0-0ubuntu2 (resolute)
  Related: python3-cryptography 46.0.5-1ubuntu2
  
  On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
  the `mgr module enable cephadm` step with:
  
-   Error ENOENT: module 'cephadm' reports that it cannot run on the active
-   manager daemon: PyO3 modules do not yet support subinterpreters, see
-   https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)
+   Error ENOENT: module 'cephadm' reports that it cannot run on the active
+   manager daemon: PyO3 modules do not yet support subinterpreters, see
+   https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)
  
  Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
  against a rock built from the same archive. The mgr daemon log
  (via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
  traceback when the cephadm module is loaded in a subinterpreter:
  
-   File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
-       from .module import CephadmOrchestrator
-   File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
-       from cephadm.cert_mgr import CertMgr
-   File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
-       from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
-   File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
-       from cryptography import x509
-   File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
-       in <module>
-       from cryptography.x509 import certificate_transparency, verification
-   File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
-       line 8, in <module>
-       from cryptography.hazmat.bindings._rust import x509 as rust_x509
-   ImportError: PyO3 modules do not yet support subinterpreters,
-       see https://github.com/PyO3/pyo3/issues/576
+   File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
+       from .module import CephadmOrchestrator
+   File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
+       from cephadm.cert_mgr import CertMgr
+   File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
+       from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
+   File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
+       from cryptography import x509
+   File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
+       in <module>
+       from cryptography.x509 import certificate_transparency, verification
+   File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
+       line 8, in <module>
+       from cryptography.hazmat.bindings._rust import x509 as rust_x509
+   ImportError: PyO3 modules do not yet support subinterpreters,
+       see https://github.com/PyO3/pyo3/issues/576
  
  The same failure mode applies to the diskprediction_local mgr module,
  where the failing import is `numpy._core._multiarray_umath` (also
  PyO3-built in resolute's python3-numpy 2.3.x).
  
  Root cause:
  - Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
-   subinterpreter by default.
+   subinterpreter by default.
  - Resolute's python3-cryptography 46.0.5 includes Rust extensions built
-   with PyO3, which raises ImportError when imported into a subinterpreter
-   (PyO3 currently does not support PEP 684 multi-interpreter modules).
+   with PyO3, which raises ImportError when imported into a subinterpreter
+   (PyO3 currently does not support PEP 684 multi-interpreter modules).
  
  Upstream fix:
  ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
  merged 2025-11-07) inverts the default: mgr modules load in the main
  interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
  is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
  by reading src/common/options/mgr.yaml.in at both tags). It is therefore
  absent from resolute's ceph 20.2.0-0ubuntu2.
  
  Requested action:
  Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
  `mgr module enable cephadm` succeeds on resolute. Until that lands,
  canonical/ceph-containers's Tentacle rock works around this by switching
  back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
  MicroCeph uses).
  
  Reproduction (LXD VM):
-   1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
-   2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
-   3. cephadm --image <any-tentacle-image> bootstrap --mon-ip <ip> \
-        --single-host-defaults --skip-dashboard --skip-monitoring-stack
-   4. Observe the PyO3 error during the `mgr module enable cephadm` step.
+   1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
+   2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
+   3. cephadm --image <resolute-based-tentacle-OCI> bootstrap --mon-ip <ip> \
+        --single-host-defaults --skip-dashboard --skip-monitoring-stack
+   4. Observe the PyO3 error during the `mgr module enable cephadm` step.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2150762

Title:
  Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3
  subinterpreter incompatibility)

Status in ceph package in Ubuntu:
  New

Bug description:
  Package: ceph 20.2.0-0ubuntu2 (resolute)
  Related: python3-cryptography 46.0.5-1ubuntu2

  On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
  the `mgr module enable cephadm` step with:

    Error ENOENT: module 'cephadm' reports that it cannot run on the active
    manager daemon: PyO3 modules do not yet support subinterpreters, see
    https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

  Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
  against a rock built from the same archive. The mgr daemon log
  (via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
  traceback when the cephadm module is loaded in a subinterpreter:

    File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
        from .module import CephadmOrchestrator
    File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
        from cephadm.cert_mgr import CertMgr
    File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
        from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
    File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
        from cryptography import x509
    File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
        in <module>
        from cryptography.x509 import certificate_transparency, verification
    File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
        line 8, in <module>
        from cryptography.hazmat.bindings._rust import x509 as rust_x509
    ImportError: PyO3 modules do not yet support subinterpreters,
        see https://github.com/PyO3/pyo3/issues/576

  The same failure mode applies to the diskprediction_local mgr module,
  where the failing import is `numpy._core._multiarray_umath` (also
  PyO3-built in resolute's python3-numpy 2.3.x).

  Root cause:
  - Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
    subinterpreter by default.
  - Resolute's python3-cryptography 46.0.5 includes Rust extensions built
    with PyO3, which raises ImportError when imported into a subinterpreter
    (PyO3 currently does not support PEP 684 multi-interpreter modules).

  Upstream fix:
  ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
  merged 2025-11-07) inverts the default: mgr modules load in the main
  interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
  is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
  by reading src/common/options/mgr.yaml.in at both tags). It is therefore
  absent from resolute's ceph 20.2.0-0ubuntu2.

  Requested action:
  Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
  `mgr module enable cephadm` succeeds on resolute. Until that lands,
  canonical/ceph-containers's Tentacle rock works around this by switching
  back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
  MicroCeph uses).

  Reproduction (LXD VM):
    1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
    2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
    3. cephadm --image <resolute-based-tentacle-OCI> bootstrap --mon-ip <ip> \
         --single-host-defaults --skip-dashboard --skip-monitoring-stack
    4. Observe the PyO3 error during the `mgr module enable cephadm` step.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2150762/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list