[Bug 2112506] [NEW] Segmentation fault while multithreading and multiprocessing (Python 3.8, 3.9; pytest-xdist; opentelemetry-sdk), intermittently reproducible

Dima Tisnek 2112506 at bugs.launchpad.net
Thu Jun 5 00:57:26 UTC 2025 

Public bug reported:

We're facing an issue where sometimes Python 3.8 and 3.9 crash while
running out suite of pure-Python unit tests.

Python 3.10 and higher are not affected.
The key seems to be multithreading and forking/multiprocessing.
The segmentation fault seem to happen in ContextVar.set

Symptom when using pytest-xdist: worker 'gw0' crashed while running
'testing/tests/test_e2e/test_...'

I believe it's the same issue in py 3.8 and 3.9.
I don't have a backtrace for 3.8 though.
3.9 provides a magic env var where Python dumps stack of all threads and I get this:

```
2025-05-30T00:34:40.0891620Z Fatal Python error: Segmentation fault
2025-05-30T00:34:40.0892247Z testing/tests/test_e2e/test_state.py::test_bare_event 
2025-05-30T00:34:40.0892663Z 
2025-05-30T00:34:40.0893997Z [gw0]�[36m [ 85%] �[0m�[32mPASSED�[0m testing/tests/test_e2e/test_state.py::test_bare_event 
2025-05-30T00:34:40.0895862Z Thread 0x00007f1a631b66c0 (most recent call first):
2025-05-30T00:34:40.0898168Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 316 in wait
2025-05-30T00:34:40.0899660Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 258 in worker
2025-05-30T00:34:40.0906958Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 917 in run
2025-05-30T00:34:40.0908249Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 980 in _bootstrap_inner
2025-05-30T00:34:40.0909611Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 937 in _bootstrap
2025-05-30T00:34:40.0910341Z 
2025-05-30T00:34:40.0910646Z Thread 0x00007f1a6a9ff6c0 (most recent call first):
2025-05-30T00:34:40.0911883Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 534 in read
2025-05-30T00:34:40.0913638Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 567 in from_io
2025-05-30T00:34:40.0915652Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
2025-05-30T00:34:40.0917394Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 341 in run
2025-05-30T00:34:40.0919223Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
2025-05-30T00:34:40.0920247Z 
2025-05-30T00:34:40.0935905Z Current thread 0x00007f1a6be30b80 (most recent call first):
2025-05-30T00:34:40.0937432Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/context/contextvars_context.py", line 41 in attach
2025-05-30T00:34:40.0939416Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/context/__init__.py", line 144 in attach
2025-05-30T00:34:40.0941588Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/trace/__init__.py", line 585 in use_span
2025-05-30T00:34:40.0943532Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__
2025-05-30T00:34:40.0945978Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/sdk/trace/__init__.py", line 1099 in start_as_current_span
2025-05-30T00:34:40.0948097Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__
2025-05-30T00:34:40.0950167Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/trace/__init__.py", line 451 in start_as_current_span
2025-05-30T00:34:40.0952114Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__
2025-05-30T00:34:40.0966905Z   File "/home/runner/work/operator/operator/ops/framework.py", line 1014 in _reemit
2025-05-30T00:34:40.0967960Z   File "/home/runner/work/operator/operator/ops/framework.py", line 924 in _emit
2025-05-30T00:34:40.0969147Z   File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit
2025-05-30T00:34:40.0970440Z   File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit
2025-05-30T00:34:40.0971891Z   File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit
2025-05-30T00:34:40.0988312Z   File "/home/runner/work/operator/operator/ops/framework.py", line 351 in emit
2025-05-30T00:34:40.0989384Z   File "/home/runner/work/operator/operator/ops/charm.py", line 1432 in _evaluate_status
2025-05-30T00:34:40.0990396Z   File "/home/runner/work/operator/operator/ops/_main.py", line 405 in _emit
2025-05-30T00:34:40.0991303Z   File "/home/runner/work/operator/operator/ops/_main.py", line 459 in run
2025-05-30T00:34:40.0992346Z   File "/home/runner/work/operator/operator/testing/src/scenario/context.py", line 121 in run
2025-05-30T00:34:40.0993485Z   File "/home/runner/work/operator/operator/testing/tests/helpers.py", line 58 in trigger
2025-05-30T00:34:40.0994803Z   File "/home/runner/work/operator/operator/testing/tests/test_e2e/test_state.py", line 118 in test_status_setting
2025-05-30T00:34:40.0996537Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
2025-05-30T00:34:40.0998158Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
2025-05-30T00:34:40.0999724Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
2025-05-30T00:34:40.1001281Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
2025-05-30T00:34:40.1002849Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/python.py", line 1792 in runtest
2025-05-30T00:34:40.1004433Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
2025-05-30T00:34:40.1006300Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
2025-05-30T00:34:40.1007899Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
2025-05-30T00:34:40.1009392Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
2025-05-30T00:34:40.1010951Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 262 in <lambda>
2025-05-30T00:34:40.1012530Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 341 in from_call
2025-05-30T00:34:40.1014321Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
2025-05-30T00:34:40.1016168Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 222 in call_and_report
2025-05-30T00:34:40.1018049Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 133 in runtestprotocol
2025-05-30T00:34:40.1019805Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
2025-05-30T00:34:40.1021411Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
2025-05-30T00:34:40.1022944Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
2025-05-30T00:34:40.1024443Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
2025-05-30T00:34:40.1026253Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 195 in run_one_test
2025-05-30T00:34:40.1027834Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 174 in pytest_runtestloop
2025-05-30T00:34:40.1029603Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
2025-05-30T00:34:40.1031155Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
2025-05-30T00:34:40.1032647Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
2025-05-30T00:34:40.1034098Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 325 in _main
2025-05-30T00:34:40.1035736Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 271 in wrap_session
2025-05-30T00:34:40.1037330Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
2025-05-30T00:34:40.1038988Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
2025-05-30T00:34:40.1040543Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
2025-05-30T00:34:40.1042109Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
2025-05-30T00:34:40.1043572Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 393 in <module>
2025-05-30T00:34:40.1045389Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1291 in executetask
2025-05-30T00:34:40.1047023Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 341 in run
2025-05-30T00:34:40.1048665Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
2025-05-30T00:34:40.1050508Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
2025-05-30T00:34:40.1052307Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1273 in serve
2025-05-30T00:34:40.1053855Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1806 in serve
2025-05-30T00:34:40.1054927Z   File "<string>", line 8 in <module>
2025-05-30T00:34:40.1075786Z   File "<string>", line 1 in <module>
2025-05-30T00:34:40.1123245Z testing/tests/test_e2e/test_state.py::test_leader_get 
```

We can hit this both in local env (python from astral) as well as github actions (ubuntu latest).
We're tracking it here https://github.com/canonical/operator/issues/1771
More backtraces, etc., are attached there.

Incidence: takes about one hour on a multi-core system running unit
tests back to back. That's between once per 100K to once per 10K tests.

An upstream ticket, https://github.com/python/cpython/issues/122529 looks very similar to my report.
CPython no longer accepts bug fixes for 3.8 through 3.12 howerver, so if this is ever going to be fixed, that would be in the context of Ubuntu LTS.

** Affects: python3.8 (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python3.8 in Ubuntu.
https://bugs.launchpad.net/bugs/2112506

Title:
  Segmentation fault while multithreading and multiprocessing (Python
  3.8, 3.9; pytest-xdist; opentelemetry-sdk), intermittently
  reproducible

Status in python3.8 package in Ubuntu:
  New

Bug description:
  We're facing an issue where sometimes Python 3.8 and 3.9 crash while
  running out suite of pure-Python unit tests.

  Python 3.10 and higher are not affected.
  The key seems to be multithreading and forking/multiprocessing.
  The segmentation fault seem to happen in ContextVar.set

  Symptom when using pytest-xdist: worker 'gw0' crashed while running
  'testing/tests/test_e2e/test_...'

  I believe it's the same issue in py 3.8 and 3.9.
  I don't have a backtrace for 3.8 though.
  3.9 provides a magic env var where Python dumps stack of all threads and I get this:

  ```
  2025-05-30T00:34:40.0891620Z Fatal Python error: Segmentation fault
  2025-05-30T00:34:40.0892247Z testing/tests/test_e2e/test_state.py::test_bare_event 
  2025-05-30T00:34:40.0892663Z 
  2025-05-30T00:34:40.0893997Z [gw0]�[36m [ 85%] �[0m�[32mPASSED�[0m testing/tests/test_e2e/test_state.py::test_bare_event 
  2025-05-30T00:34:40.0895862Z Thread 0x00007f1a631b66c0 (most recent call first):
  2025-05-30T00:34:40.0898168Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 316 in wait
  2025-05-30T00:34:40.0899660Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 258 in worker
  2025-05-30T00:34:40.0906958Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 917 in run
  2025-05-30T00:34:40.0908249Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  2025-05-30T00:34:40.0909611Z   File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 937 in _bootstrap
  2025-05-30T00:34:40.0910341Z 
  2025-05-30T00:34:40.0910646Z Thread 0x00007f1a6a9ff6c0 (most recent call first):
  2025-05-30T00:34:40.0911883Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 534 in read
  2025-05-30T00:34:40.0913638Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 567 in from_io
  2025-05-30T00:34:40.0915652Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
  2025-05-30T00:34:40.0917394Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 341 in run
  2025-05-30T00:34:40.0919223Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  2025-05-30T00:34:40.0920247Z 
  2025-05-30T00:34:40.0935905Z Current thread 0x00007f1a6be30b80 (most recent call first):
  2025-05-30T00:34:40.0937432Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/context/contextvars_context.py", line 41 in attach
  2025-05-30T00:34:40.0939416Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/context/__init__.py", line 144 in attach
  2025-05-30T00:34:40.0941588Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/trace/__init__.py", line 585 in use_span
  2025-05-30T00:34:40.0943532Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__
  2025-05-30T00:34:40.0945978Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/sdk/trace/__init__.py", line 1099 in start_as_current_span
  2025-05-30T00:34:40.0948097Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__
  2025-05-30T00:34:40.0950167Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/trace/__init__.py", line 451 in start_as_current_span
  2025-05-30T00:34:40.0952114Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__
  2025-05-30T00:34:40.0966905Z   File "/home/runner/work/operator/operator/ops/framework.py", line 1014 in _reemit
  2025-05-30T00:34:40.0967960Z   File "/home/runner/work/operator/operator/ops/framework.py", line 924 in _emit
  2025-05-30T00:34:40.0969147Z   File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit
  2025-05-30T00:34:40.0970440Z   File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit
  2025-05-30T00:34:40.0971891Z   File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit
  2025-05-30T00:34:40.0988312Z   File "/home/runner/work/operator/operator/ops/framework.py", line 351 in emit
  2025-05-30T00:34:40.0989384Z   File "/home/runner/work/operator/operator/ops/charm.py", line 1432 in _evaluate_status
  2025-05-30T00:34:40.0990396Z   File "/home/runner/work/operator/operator/ops/_main.py", line 405 in _emit
  2025-05-30T00:34:40.0991303Z   File "/home/runner/work/operator/operator/ops/_main.py", line 459 in run
  2025-05-30T00:34:40.0992346Z   File "/home/runner/work/operator/operator/testing/src/scenario/context.py", line 121 in run
  2025-05-30T00:34:40.0993485Z   File "/home/runner/work/operator/operator/testing/tests/helpers.py", line 58 in trigger
  2025-05-30T00:34:40.0994803Z   File "/home/runner/work/operator/operator/testing/tests/test_e2e/test_state.py", line 118 in test_status_setting
  2025-05-30T00:34:40.0996537Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  2025-05-30T00:34:40.0998158Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
  2025-05-30T00:34:40.0999724Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
  2025-05-30T00:34:40.1001281Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
  2025-05-30T00:34:40.1002849Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/python.py", line 1792 in runtest
  2025-05-30T00:34:40.1004433Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  2025-05-30T00:34:40.1006300Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
  2025-05-30T00:34:40.1007899Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
  2025-05-30T00:34:40.1009392Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
  2025-05-30T00:34:40.1010951Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 262 in <lambda>
  2025-05-30T00:34:40.1012530Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 341 in from_call
  2025-05-30T00:34:40.1014321Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  2025-05-30T00:34:40.1016168Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 222 in call_and_report
  2025-05-30T00:34:40.1018049Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  2025-05-30T00:34:40.1019805Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  2025-05-30T00:34:40.1021411Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
  2025-05-30T00:34:40.1022944Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
  2025-05-30T00:34:40.1024443Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
  2025-05-30T00:34:40.1026253Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 195 in run_one_test
  2025-05-30T00:34:40.1027834Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 174 in pytest_runtestloop
  2025-05-30T00:34:40.1029603Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
  2025-05-30T00:34:40.1031155Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
  2025-05-30T00:34:40.1032647Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
  2025-05-30T00:34:40.1034098Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 325 in _main
  2025-05-30T00:34:40.1035736Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 271 in wrap_session
  2025-05-30T00:34:40.1037330Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  2025-05-30T00:34:40.1038988Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall
  2025-05-30T00:34:40.1040543Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec
  2025-05-30T00:34:40.1042109Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__
  2025-05-30T00:34:40.1043572Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 393 in <module>
  2025-05-30T00:34:40.1045389Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1291 in executetask
  2025-05-30T00:34:40.1047023Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 341 in run
  2025-05-30T00:34:40.1048665Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  2025-05-30T00:34:40.1050508Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
  2025-05-30T00:34:40.1052307Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1273 in serve
  2025-05-30T00:34:40.1053855Z   File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1806 in serve
  2025-05-30T00:34:40.1054927Z   File "<string>", line 8 in <module>
  2025-05-30T00:34:40.1075786Z   File "<string>", line 1 in <module>
  2025-05-30T00:34:40.1123245Z testing/tests/test_e2e/test_state.py::test_leader_get 
  ```

  We can hit this both in local env (python from astral) as well as github actions (ubuntu latest).
  We're tracking it here https://github.com/canonical/operator/issues/1771
  More backtraces, etc., are attached there.

  Incidence: takes about one hour on a multi-core system running unit
  tests back to back. That's between once per 100K to once per 10K
  tests.

  An upstream ticket, https://github.com/python/cpython/issues/122529 looks very similar to my report.
  CPython no longer accepts bug fixes for 3.8 through 3.12 howerver, so if this is ever going to be fixed, that would be in the context of Ubuntu LTS.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2112506/+subscriptions

More information about the foundations-bugs mailing list