[Bug 1893716] Re: scripts in /etc/update-motd.d/ run even on login via non-interactive scp and sftp sessions

[Christian Ehrhardt ]

Overall the majority of this is now fixed and mitigated with the combination of:
 ubuntu-release-upgrader | 1:22.04.8  | jammy           | source
 landscape-client | 19.12-0ubuntu13        | jammy           | source, amd64, arm64, armhf, ppc64el, riscv64, s390x
 update-notifier | 3.192.54       | jammy           | source, amd64, arm64, armhf, ppc64el, riscv64, s390x

There is still the IMHO valid feature request to pam_motd to not run at
all in non-interactive sessions which I'll need to file upstream.

But already in a system with these updates:
ubuntu at login-jammy:~$ dpkg -l ubuntu-release-upgrader-core landscape-common update-notifier-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                         Version         Architecture Description
+++-============================-===============-============-=======================================================
ii  landscape-common             19.12-0ubuntu13 amd64        Landscape administration system client - Common files
ii  ubuntu-release-upgrader-core 1:22.04.8       all          manage release upgrades
ii  update-notifier-common       3.192.54        all          Files shared between update-notifier and other packages


I now get reasonable results.
Down from 70-80 seconds to ~20-25 => almost down to 1/4 of the time.
At the same time the system is ~16% less busy, so other things running won't stall it that much either and vice versa.


What is left looks as in the test sessions.
This now mostly comes down to the fact that logging in for every command will in general have overhead to spawn the session. For another gain pam_motd can be disabled as shown above, but that does not reduce it to zero overhead - so as explained any mutli-command submitting solution should still - even with the fix - try to use one login for all of them.

# Overhead  Command        
# ........  ...............
#
    32.50%  swapper        
    26.67%  sshd           
     3.53%  dbus-daemon    
     3.37%  systemd        
     2.36%  run-parts      
     2.02%  systemd-logind 
     1.87%  find           
     1.85%  gdbus          
     1.48%  cat            
     1.47%  update-motd-fsc
     1.22%  50-motd-news   
     1.17%  awk            
     1.15%  systemd-journal
     1.11%  grep           
     1.10%  bash           
     1.05%  uname          
     0.98%  00-header      
     0.93%  91-release-upgr
     0.92%  97-overlayroot 
     0.81%  90-updates-avai
     0.80%  date           
     0.72%  cut            
     0.68%  50-landscape-sy
     0.62%  env            
     0.59%  ksoftirqd/0    
     0.58%  95-hwe-eol     
     0.53%  stat           
     0.51%  id

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to update-notifier in Ubuntu.
https://bugs.launchpad.net/bugs/1893716

Title:
  scripts in /etc/update-motd.d/ run even on login via non-interactive
  scp and sftp sessions

Status in landscape-client package in Ubuntu:
  Fix Released
Status in pam package in Ubuntu:
  Confirmed
Status in ubuntu-release-upgrader package in Ubuntu:
  Fix Released
Status in update-motd package in Ubuntu:
  Invalid
Status in update-notifier package in Ubuntu:
  Fix Released

Bug description:
  My client has 200+ devices automatically uploading information via
  sftp and scp to a server every few minutes. After a recent update, I
  noticed the load on their server spiking through the roof. Upon
  investigation, I discovered a horde of landscape-sysinfo and
  /usr/bin/lsb_release processes running that correlated with login
  session notifications in /var/log/syslog and the load spikes.

  It appears that even in non-interactive sessions where this
  information will never be seen, the configuration options below in
  /etc/pam.d/sshd cause these items to be launched (in fact, probably
  everything in /etc/update-motd.d). This only started on the system in
  question after a recent set of system updates were installed.

  The content of /etc/update-motd.d/* really, really, really shouldn't
  be executed if the session in question is not interactive, as it
  provides no value at all. Unfortunately, to disable it for these non-
  interactive sessions, we also have to disable it for the interactive
  ones as well where it has some value (though not enough to make
  spiking the load on this server through the roof an acceptable
  tradeoff).

  # Print the message of the day upon successful login.
  # This includes a dynamically generated part from /run/motd.dynamic
  # and a static (admin-editable) part from /etc/motd.
  #session    optional     pam_motd.so  motd=/run/motd.dynamic
  #session    optional     pam_motd.so noupdate

  Also, looking at the script 00-header in /etc/update-motd.d/,
  /usr/bin/lsb_release is being improperly launched, as /etc/lsb_release
  does include the necessary information:

  [ -r /etc/lsb-release ] && . /etc/lsb-release

  if [ -z "$DISTRIB_DESCRIPTION" ] && [ -x /usr/bin/lsb_release ]; then
          # Fall back to using the very slow lsb_release utility
          DISTRIB_DESCRIPTION=$(lsb_release -s -d)
  fi

  # cat /etc/lsb-release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=16.04
  DISTRIB_CODENAME=xenial
  DISTRIB_DESCRIPTION="Ubuntu 16.04.7 LTS"

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/landscape-client/+bug/1893716/+subscriptions