[Bug 2065515] Re: [SRU] Ceph Squid release candidate

[Jay Ring]

I was able to roll the installation back to the previous version and the
system came back.  But I am stuck at this old version and can not
upgrade.

As this is currently the top search result for the error, I want to
share my method, which took me longer than I would like to admit to work
out.  Maybe it will save someone else some trouble.

1. Stop and disable all servers except for the monitor(ceph-mgr, ceph-
osd, ceph-mds)

2. Roll back the entire ceph installation, uninstalling everything.
"aptitude install ceph-common=19.2.0~git20240301.4c76c50-0ubuntu6"

3. Install just the monitors
"aptitude install ceph-mon=19.2.0~git20240301.4c76c50-0ubuntu6"
You need to look through the different solutions aptitude offers, it was not the first one for me, it was the second.  Look for the solution that results in ceph-mon being installed.  It should bring the older, working python installs with it.

4. The monitors should now start.  Wait for them to come back online.

5. Reinstall the OSD service.
"aptitude install ceph-osd=19.2.0~git20240301.4c76c50-0ubuntu6"
Again, for me, it was the second solution, but you will need to use some judgement.  Restart the OSDs and wait for them to come online.

6. On some of my servers, the OSDs started just fine.  On others they did not.  On the ones where they did not restart properly it was because of ceph-volume"
"aptitude install ceph-volume=19.2.0~git20240301.4c76c50-0ubuntu6"
Again you need to choose the proper solution, probably not the first.

7. Now install and activate the MGR and MDS again with a similar
aptitude command.  Everything should be back.

I tried several times a simple "aptitude install
ceph=19.2.0~git20240301.4c76c50-0ubuntu6" command but never got it to
work.  I think starting the monitors before everything else was
necessary, I am not sure.

Best of luck to anyone else affected.

Thanks in advance to anyone working on a solution.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2065515

Title:
  [SRU] Ceph Squid release candidate

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive caracal series:
  Fix Committed
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Noble:
  Fix Released
Status in ceph source package in Oracular:
  Fix Released

Bug description:
  [Impact]
  Ubuntu 24.04 LTS users are currently using a development snapshot of Ceph; the upstream project are working towards a release and produced a 19.1.0 RC which we should include in 24.04 as a stepping stone towards release.  There are current know upgrade issues with the development snapshot included.

  [Test Case]
  No reproducer for this bug; RC will be tested in the following ways:

  - As part of a core24 based microceph built from proposed
  - Using the standard OpenStack Regression Testing utilised for validation of OpenStack and Ceph updates under the MRE already in place:

  https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates

  Test results will all be attached to this bug report.

  [Where things might go wrong]
  As this is an RC there will likely be outstanding bugs that will need to be resolved by the upstream project before release; This is expected but the RC will be an improvement on the current development snapshot in terms of quality.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2065515/+subscriptions