[Bug 2119024] Re: [SRU] Squid: Ceph new point release 19.2.3

Mon Nov 24 03:35:09 UTC 2025

Hello,
The whole test-suite for noble-proposed has completed and all tests have passed. The results can be seen here: https://github.com/canonical/ceph-charms/actions/runs/19547255675?pr=120
(Please ignore the COS integration test, it isn't related to this SRU and it's been broken for a while now).

As for the tests regarding LTTNG, the following procedure was used:

- After deploying a Ceph cluster with 19.2.3, we install the lttng binaries:
> sudo apt install lttng-tools

- Next, we run the lttng daemons:
> lttng-sessiond --daemonize

- We can then list the available trace points:
> lttng list -u

  UST events:
  -------------

  PID: 14531 - Name: /usr/bin/ceph-osd

  ust_baddr_statedump:soinfo (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  mutex:lock_exit (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  mutex:lock_enter (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)

- We can now create a session, and begin tracing:

> lttng create
  Session auto-31240637-925731 created.

> lttng enable-event -u -a
  All UST events are enabled in channel channel0

> lttng start
  Tracing started for session auto-31240637-925731

- Next, we'll generate some activity on the Ceph cluster. We'll use
`rados bench` but any other type of I/O over objects can be applied:

> sudo rados -p data bench 10 write

- After the above is done, we'll stop tracing and verify the output:

> lttng stop
  Waiting for data availability.
  Tracing stopped for session auto-31240637-925731

> lttng view | head
[18:42:17.027535265] (+0.000001786) mutex:lock_enter: { cpu_id = 0 }, { addr = 0x1fc2408, name = "SimpleMessenger::DispatchQeueu::lock" }
[18:42:17.027535846] (+0.000000581) mutex:lock_exit: { cpu_id = 0 }, { addr = 0x1fc2408, name = "SimpleMessenger::DispatchQeueu::lock" }
[18:42:23.027538829] (+0.000000576) mutex:lock_exit: { cpu_id = 0 }, { addr = 0x205a610, name = "OSD::recovery_tp::lock" }

Finally, we can note that if we disable tracing and run the `rados
bench` program again, no tracing output will be generated.

** Tags removed: verification-needed-noble
** Tags added: verification-done-noble

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2119024

Title:
  [SRU] Squid: Ceph new point release 19.2.3

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive caracal series:
  New
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Noble:
  Fix Committed
Status in ceph source package in Plucky:
  Fix Committed
Status in ceph source package in Questing:
  Fix Committed
Status in ceph source package in Resolute:
  Fix Released

Bug description:
  [Impact]
  This release fixes several bugs. We would like to make sure all of our users have access to these improvements.

  The update contains the following package updates:
     * ceph 19.2.2 https://ceph.io/en/news/blog/2025/v19-2-2-squid-released/
     * ceph 19.2.3 https://ceph.io/en/news/blog/2025/v19-2-3-squid-released/

  Notable changes:
      * This fixes a silent RGW data loss regression when CopyObject is used to copy an object onto itself. The regression results in objects past the original 4mb head object being added to the garbage collection queue and deleted permanently, and are not recoverable.
      * LTTNG has been enabled for easier tracing and debugging.

  Key differences from upstream:
      * libnbd has been disabled from the build system. libnbd is in universe in Ubuntu, and cannot be a dependency for 19.2.3.
      * On questing, adjustments had to be made to build with boost1.88. On plucky and noble, these adjustments have been removed to build with standard boost 1.83 and 1.82.
      * python-cryptography uses PyO3 to call some rust based bcrypt and TLS functions, and has problems being called in parallel and in sub-interpreters, necessary for ceph-dashboard to work correctly. This was fixed by backporting:
    commit 717d0a6f3530ad3e07f4423002810327b2addcf1
    From: "Paulo E. Castro" <pecastro at wormholenet.com>
    Date: Sat, 5 Apr 2025 21:47:55 +0100
    Subject: pybind/mgr: Hack around the 'ImportError: PyO3 modules
     may only be initialized once per interpreter process' issue.
    Link: https://github.com/ceph/ceph/commit/7094a5a44d90e705141dbae9739e6c0835bf7ce3
    See also:
     https://tracker.ceph.com/issues/64213
     https://github.com/bazaah/aur-ceph/issues/20
     https://bugs.launchpad.net/charm-ceph-dashboard/+bug/2073115
      * Noble also receives the bugfix for hardened CFLAGS for python modules. See https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2078244

  [Test Case]

  This SRU will run all functional tests with the packages in 'proposed'
  for every Ceph charm maintained by Canonical's Ceph team, so that
  there's a higher chance of catching bugs associated with any Ceph
  component.

  The charms for which the functional tests will be run are: ceph-mon, ceph-osd,
  ceph-radosgw, ceph-proxy, ceph-nfs, ceph-nvme, ceph-fs and ceph-rbd-mirror.

  The functional test suite includes upgrade testing from the current stable
  release to the proposed (in this case, from 19.2.1 to 19.2.3)

  Since our functional tests do not include testing for LTTNG, this will be
  tested manually. This involves the following steps:

  - Deploy a Ceph cluster with 19.2.3.
  - Enable OSD tracing by adding the option `osd_tracing` to /etc/ceph.conf
  - Run `lttng create test`
  - Run `lttng enable-event --userspace osd:*`
  - At this point, tracing is enabled. Do some writes to an OSD, e.g: By creating an RBD object or via ceph bench.
  - Stop tracing with `lttng stop`
  - View the logs with `lttng view`.
  - We can then do some more writes on an OSD and verify that no more tracing output is generated.

  [Regression Potential]

  ceph is a critical package as it underpins large scale data storage
  and integrity checking, and care must be taken to not introduce any
  data loss regressions. ceph also is a common backing store for
  OpenStack virtual machines through ceph-rbd, and care must be taken to
  not break any API or ABI compatibility with current OpenStack
  releases.

  In order to mitigate the regression potential, the results of the
  aforementioned tests will be attached to this bug.

  [Other info]

  Microrelease exception:
  https://documentation.ubuntu.com/project/SRU/reference/exception-
  OpenStack-Updates/

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2119024/+subscriptions