[Bug 1972159] Re: systemd-oomd frequently kills firefox and visual studio code

[Tim Richardson]

For me, systemd-oomd no longer kills at all. The memory pressure
threshold is still active, but I think the default of 50% on the user
slice is way too high. I can put a 4gb test VM under extreme memory load
and get so much swap activity that CPU load in a two core VM gets > 50,
yet the memory pressure score is 14%. I can not conceive of what type of
load would get it to 50%.

I have set the user slice threshold to 10%, and when I attempt to load
100 tabs, the browser is killed a couple of minutes after memory and
swap is exhausted. It's not an aggressive kill, but it lets systemd-oomd
actually kill something.

So far it has only ever killed the guilty app. I think if the aim is not
have systemd-oomd ever kill anything, 50% memory threshold and swap kill
off achieves the goal, but if you want it to kill baes on memory
pressure, the memory threshold needs to be much lower. killing on memory
pressure was supposed to be one of the great things about systemd-oomd,
I thought.

I note the systemd-cgtop shows there are many tasks under the user slice
(I have about 400 when idle, and about 1200 when the brower is trying to
load all those tabs). All the system slices have < 5 tasks. So one or
two of those processes being stalled will result is a steep increase in
memory pressure KPI. But perhaps with so many tasks in the user slice,
the KPI is highly "diluted" and needs a much lower threshold to be
meaningful.

Maybe this is all very different on a raspberry PI.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1972159

Title:
  systemd-oomd frequently kills firefox and visual studio code

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Jammy:
  Fix Committed
Status in systemd source package in Kinetic:
  Fix Released
Status in systemd package in Fedora:
  Unknown

Bug description:
  [Impact]

  The "swap kill" side of systemd-oomd has caused unexpected behavior
  for desktop users. A user's browser, desktop session, or some other
  desktop application may be killed by systemd-oomd when SwapUsedLimit
  is reached, but system performance otherwise appears unaffected. This
  leaves users confused as to why their application was killed, and has
  a negative impact on their desktop experience.

  For now, let's disable the swap kill functionality by default.

  [Test Plan]

  On Jammy desktop, check the ManagedOOMSwap property on -.slice:

  $ systemctl show -- "-.slice" | grep "^ManagedOOMSwap"
  ManagedOOMSwap=kill # After the fix, this should print ManagedOOMSwap=auto

  [Where problems could occur]

  Disabling swap kill by default means that users may experience
  degraded system performance due to high swap usage, because systemd-
  oomd will no longer act on cgroups with high swap usage.

  [Other Info]

  If a user wishes to restore the original systemd-oomd behavior, they
  can do so by creating the following overrides file:

   $ cat /etc/systemd/system/-.slice.d/10-oomd-root-slice-defaults.conf
   [Slice]
   ManagedOOMSwap=kill

  [Original Description]

  Since I installed Ubuntu 22.04, firefox and visual studio code are
  frequently killed by systemd-oomd (every 2hours).

  I have 8 GB memory and never experienced this before the upgrade to
  Ubuntu 22.04. I thus assume that the claim that there is not enough
  memory is abusive. Did 64GB of memory become the minimum requirement
  to run Ubuntu ?

  The second problem is that it gives a very bad user experience which
  is critical for new Ubuntu users.

  There should be a warning prior killing apps to give the opportunity
  to save the app data. There should at least be an apologize and an
  explanation after killing the app.

  The current behavior gives the impression that Ubuntu 22.04 is
  unreliable and unsafe to use which is a problem for an LTS release
  that many people might want to use for critical production context.

  There might be a configuration problem with systemd-oomd or simply a
  bogus behavior. I would recommend to disable it or remove it
  completely until this problem is resolved. This is what I will do for
  myself because I have work to do.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159/+subscriptions