[Bug 1977652] Re: systemd loops trying to start systemd-ask-password-plymouth

[Walter]

Funny you should ask. I tempted to ask where you think I got the output
from in the first place. But I shall refrain from sarcasm as that won't
do any good.

This does put me in a bit of a pickle. Because in order for the
journalctl output to be more useful than just the 40.000 message I
referenced, I should leave it untouched. But I cannot give it to you
unsanitized either.

I'm attaching the log uniq -c'd, starting just before the issue at hand,
so I could inspect it for private details. Observe that the `1501` on
the left means 1501x the same message in lines like:

   1501 jun 04 15:01:55 myhostname systemd[1]: Condition check resulted
in Forward Password Requests to Plymouth being skipped.

I hope this is convincing enough that I'm not making this up.


As for the:

> I believe the commit you referenced has been in jammy since
249.9-0ubuntu1.

Okay. Well. If I apt-get source systemd, I would assume that I'd get
that as a patch file. But grepping for
"unit_has_failed_condition_or_assert" in ./debian/patches yields
nothing. Grepping for -i "condition": no relevant hits.

Are you sure this patch is included? If so, then only parts if it maybe?


Regards,
Walter

** Attachment added: "journalctl-demonstrating-plymouth-being-skipped-flood.txt"
   https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1977652/+attachment/5596096/+files/journalctl-demonstrating-plymouth-being-skipped-flood.txt

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1977652

Title:
  systemd loops trying to start systemd-ask-password-plymouth

Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  Short story:
  - after boot
  - when a systemd service wants a password
  - systemd-ask-password is invoked
  - systemd-ask-password-plymouth.path is reached
  - systemd-ask-password-plymouth.service is started

  Except:
  - the conditions for the latter are (apparently) not met

  End result:
  - 40.000x "Condition check resulted in Forward Password Requests to Plymouth being skipped."
  - 1.5 minutes of 100% cpu usage

  ----------------
  AFFECTED VERSION
  ----------------

  systemd 249.11-0ubuntu3.1 on Ubuntu/Jammy 22.04

  ----------------
  HOW TO REPRODUCE
  ----------------

  If I leave the `systemd-ask-password-plymouth.path` unmasked/enabled
  and reboot. OpenVPN (calling systemd-ask-password) will trigger the
  condition.

  This causes high CPU usage for 1.5 minutes.

  It looks like /run/plymouth/pid exists (for the .path file) at first,
  when that is invoked, but then the resultant target (the .service
  file) checks again, and finds that it is gone.

  Manually reproducing:

  # ls /run/systemd/ask-password
  ask.5hW6rb  sck.79cfe1203518610

  # mkdir /run/plymouth/pid
  # systemctl start systemd-ask-password-plymouth.service systemd-ask-password-plymouth.path

  # systemctl show --value --property=MainPID systemd-ask-password-plymouth.service
  24777

  # rmdir /run/plymouth/pid ; kill 24777

  Result: systemd going into a loop.

  Stop the loop with:

  # systemctl stop systemd-ask-password-plymouth.path

  
  --------
  ANALYSIS
  --------

  It looks like this:
  https://github.com/systemd/systemd/issues/21025

  which is fixed by:
  https://github.com/systemd/systemd/pull/21030

  Alternative bug reports:
  https://bugzilla.redhat.com/show_bug.cgi?id=1919538

  Systemd's own analysis of the situation:

  # systemd-analyze critical-chain systemd-ask-password-plymouth.service
  ...
  systemd-ask-password-plymouth.service @1min 35.823s
  └─systemd-ask-password-plymouth.path @593ms
    └─plymouth-start.service @571ms +21ms
      └─systemd-udevd.service @450ms +119ms
        └─systemd-tmpfiles-setup-dev.service @429ms +18ms
          └─systemd-sysusers.service @382ms +46ms
            └─systemd-remount-fs.service @353ms +22ms
              └─systemd-journald.socket @318ms
                └─system.slice @264ms
                  └─-.slice @264ms

  -----------
  WORKAROUNDS
  -----------

  This works as long as you don't need the plymouth-ask-password:

  # systemctl disable systemd-ask-password-plymouth.path
  # systemctl mask systemd-ask-password-plymouth.path


  Could you get the relevant patches from upstream sorted in Jammy?

  Thanks!

  Walter Doekes
  OSSO B.V.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1977652/+subscriptions