[Bug 1909162] Re: cluster log slow request spam

[Corey Bryant]

This bug was fixed in the package ceph - 16.1.0-0ubuntu3~cloud0
---------------

 ceph (16.1.0-0ubuntu3~cloud0) focal-wallaby; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 ceph (16.1.0-0ubuntu3) hirsute; urgency=medium
 .
   * d/p/issue49494.patch: Cherry pick fix for issue with preprocessor
     logic which causes backport failures to focal.
   * d/p/bug1917414.patch: Cherry pick fix to isa-l to remove use of text
     relocation calls which cause ceph-osd and ceph-mon daemons to fail
     to start (LP: #1917414).
 .
 ceph (16.1.0-0ubuntu2) hirsute; urgency=medium
 .
   * No change rebuild with fixed ownership.
 .
 ceph (16.1.0-0ubuntu1) hirsute; urgency=medium
 .
   * New interim release in preparation for Ceph Pacific.
   * d/p/*: Refresh, drop any patches included upstream.
   * d/control,ceph-mgr-diskprediction-cloud.*: Drop ceph-mgr-
     diskprediction-cloud package, feature dropped upstream.
   * d/ceph-mgr-modules-core.install: Include new snap_schedule and stats
     modules.
   * d/ceph-osd.install: Include ceph-erasure-code-tool binary.
   * d/control: Add libcryptsetup-dev to BD's.
   * d/control: Add liblua5.3-dev and luarocks to BD's.
   * d/control: Drop use of python3-six.
   * d/control: Add python3-jinja2 to Depends of ceph-mgr-cephadm.
   * d/libcephfs-dev.install: Add new Types.h header.
   * d/librgw{2,-dev}.install: Drop header and so for librgw_admin_user.
   * d/python3-cephfs.install: Drop install of ceph_volume_client.py.
   * New upstream snapshot for Pacific release.
   * d/control: Add libboost-filesystem-dev to BD's, bump boost minimum
     version to 1.74.0.
   * d/rules: Install grafana dashboards.
   * d/p/fix-boost-1.74-build.patch: Resolve build failure with boost
     1.74/c++ 17.
   * d/rules: Drop install of cephadm sudoers configuration.
   * d/cephadm.install: Drop sudoers file, include manpage.
   * d/*.symbols: Update for new release.
   * d/control,rules: Enable use of boost context for riscv64 as its no
     longer an optional dependency.
   * d/p/32bit-fixes.patch: Fix issues with mismatched size_t max
     comparision on armhf.
   * d/p/disable-log-slow-requests.patch: Remove logging every slow request
     details to monitors LP: #1909162).


** Changed in: cloud-archive
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1909162

Title:
  cluster log slow request spam

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Won't Fix
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Focal:
  Fix Released
Status in ceph source package in Groovy:
  Fix Released
Status in ceph source package in Hirsute:
  Fix Released

Bug description:
  [Impact]

  A recent change (issue#43975 [0]) was made to slow request logging to
  include detail on each operation in the cluster logs. With this
  change, detail for every slow request is always sent to the monitors
  and added to the cluster logs.

  This does not scale. Large, high-throughput clusters can overwhelm
  their monitors with spurious logs in the event of a performance issue.
  Disrupting the monitors can then cause further instability in the
  cluster.

  This SRU reverts the cluster logging of every slow request the osd is
  processing.

  The slow request clog change was added in nautilus (14.2.10) and
  octopus (15.2.0).

  [Test Case]

  Stress the cluster with a benchmarking tool to generate slow requests
  and observe the cluster logs.

  [Where problems could occur]

  The cluster logs contain detailed debug information on slow requests
  that is useful for smaller, low-throughput clusters. While these logs
  are not used by ceph, they may be used by the cluster administrators
  (for monitoring or alerts). Changing this logging behavior may be
  unexpected.

  [Other Info]

  The intent is to re-enable this feature behind a configurable setting,
  but the solution must be discussed upstream.

  The same slow request detail can be enabled for each osd by raising
  the "debug osd" log level to 20.

  [0] https://tracker.ceph.com/issues/43975

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1909162/+subscriptions