[Bug 1893716] Re: scripts in /etc/update-motd.d/ run even on login via non-interactive scp and sftp sessions

[Christian Ehrhardt ]

I wanted to get a better feeling about this before jumping to action.
Therefore I have created three 1G/1vcpu KVM guests Bionic/Focal/Jammy to test and compare this on.

I do not need hot-loop analysis or anything down to instructions, so no debug symbols needed.
For now I only want to know:
1. how much time a bunch of low effort logins take (we measure only the overhead)
2. how much cpu is utilized while doing that
3. how that work is spread across programs (disable them one by one and look at data)


The two I see most are:
- /usr/lib/update-notifier/update-motd-hwe-eol
    apt-config shell SourceList Dir::Etc::sourcelist
- /etc/update-motd.d/50-landscape-sysinfo
    /usr/bin/python3 /usr/bin/landscape-sysinfo

Very non-pro data gathering:

$ cat tracestart.sh
#!/bin/bash
sudo rm perf.data perf.log log.vmstat
nohup vmstat -wt 5 &> log.vmstat &
nohup perf record --event cpu-clock --all-cpus &> perf.log &
sleep 5

$ cat traceend.sh
#!/bin/bash
killall perf
killall vmstat
cat perf.log
cat log.vmstat
#perf report --sort comm --stdio

Most simple load involving those helpers.
for sys in login-bionic login-focal login-jammy; do ssh $sys "sudo ~/tracestart.sh"; time for i in $(seq 1 100); do ssh $sys "/bin/true"; done; ssh $sys "sudo ~/traceend.sh"; done

I'll get those numbers for Bionic/Focal/Jammy and enabling/disabling it
all and/or individual elements.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to pam in Ubuntu.
https://bugs.launchpad.net/bugs/1893716

Title:
  scripts in /etc/update-motd.d/ run even on login via non-interactive
  scp and sftp sessions

Status in pam package in Ubuntu:
  Confirmed
Status in update-motd package in Ubuntu:
  Confirmed

Bug description:
  My client has 200+ devices automatically uploading information via
  sftp and scp to a server every few minutes. After a recent update, I
  noticed the load on their server spiking through the roof. Upon
  investigation, I discovered a horde of landscape-sysinfo and
  /usr/bin/lsb_release processes running that correlated with login
  session notifications in /var/log/syslog and the load spikes.

  It appears that even in non-interactive sessions where this
  information will never be seen, the configuration options below in
  /etc/pam.d/sshd cause these items to be launched (in fact, probably
  everything in /etc/update-motd.d). This only started on the system in
  question after a recent set of system updates were installed.

  The content of /etc/update-motd.d/* really, really, really shouldn't
  be executed if the session in question is not interactive, as it
  provides no value at all. Unfortunately, to disable it for these non-
  interactive sessions, we also have to disable it for the interactive
  ones as well where it has some value (though not enough to make
  spiking the load on this server through the roof an acceptable
  tradeoff).

  # Print the message of the day upon successful login.
  # This includes a dynamically generated part from /run/motd.dynamic
  # and a static (admin-editable) part from /etc/motd.
  #session    optional     pam_motd.so  motd=/run/motd.dynamic
  #session    optional     pam_motd.so noupdate

  Also, looking at the script 00-header in /etc/update-motd.d/,
  /usr/bin/lsb_release is being improperly launched, as /etc/lsb_release
  does include the necessary information:

  [ -r /etc/lsb-release ] && . /etc/lsb-release

  if [ -z "$DISTRIB_DESCRIPTION" ] && [ -x /usr/bin/lsb_release ]; then
          # Fall back to using the very slow lsb_release utility
          DISTRIB_DESCRIPTION=$(lsb_release -s -d)
  fi

  # cat /etc/lsb-release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=16.04
  DISTRIB_CODENAME=xenial
  DISTRIB_DESCRIPTION="Ubuntu 16.04.7 LTS"

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/pam/+bug/1893716/+subscriptions