[Bug 1905393] Re: Ubuntu 20.04: opal-prd fails to start on 20.04

Mon Jan 11 22:53:01 UTC 2021

Hello bugproxy, or anyone else affected,

Accepted skiboot into groovy-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/skiboot/6.5.2-1ubuntu0.20.10.1 in a
few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
groovy to verification-done-groovy. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-groovy. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: skiboot (Ubuntu Groovy)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-groovy

** Changed in: skiboot (Ubuntu Focal)
       Status: In Progress => Fix Committed

** Tags added: verification-needed-focal

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to skiboot in Ubuntu.
Matching subscriptions: foundations-bugs-skiboot
https://bugs.launchpad.net/bugs/1905393

Title:
  Ubuntu 20.04: opal-prd fails to start on 20.04

Status in The Ubuntu-power-systems project:
  In Progress
Status in skiboot package in Ubuntu:
  Fix Released
Status in skiboot source package in Focal:
  Fix Committed
Status in skiboot source package in Groovy:
  Fix Committed
Status in skiboot source package in Hirsute:
  Fix Released

Bug description:
  [Impact]
  opal-prd (the daemon on a power system that listens for hw diagnostic system events at the /dev/opal-prd device) fails to start.
  The reason is that opal-prd is not able to properly handle devtmpfs, mounted with noexec in /dev, which is the case on recent versions of systemd (like used in focal or newer)..
  Currently such a mount prevents mapping HBRT image code region as 'rwx' from /dev.

  [Fix]
  This patch/commit attempts to work around the situation by copying HBRT image to a non mmapped memory region and sets mprotect rwx on it, allowing opal-prd to successfully execute the code region (as suggested here: https://github.com/open-power/skiboot/issues/258):
  47005e8d4c9aeda5826c17c4a013cfbda1a3f2de 47005e8 "opal-prd: handle devtmpfs mounted with noexec"

  [Test Case]
  Since the opal-prd daemon must be running in the background as a separate process, the test is to:
  - install the updated package that includes the patched opal-prd daemon (e.g. from the PPA mentioned below)
  - double check the installed package version (dpkg -l) and maybe the opal-pd version that's in place (opal-prd --version)
  - start opal-prd as daemon: 'service opal-prd start' (if not started automatically)
  - verify the opal-prd status and check if it's running or not, by for example 'service opal-prd status'

  [What could go wrong]
  Things can go wrong in case the HBRT image copy is done wrong; in case it's accidentally copied to a wrong memory area (e.g. to an already mapped range, or erroneously calculated address/size), a seg. fault will happen and the system would core dump.
  The mprotect code is pretty straight forward, but the fact that mprotect rwx is set on it, allows opal-prd to successfully execute the code region. It's not generally a perfect approach to map memory as RWX, but HBRT requires the ability to write into the image at runtime - and it got upstream accepted that way with skiboot v6.7.
  The fix was released back in October and was pre-tested by the IBM Power team.
  On top a patched Ubuntu package was build and shared in a PPA (see comment #1) and again successfully validated on focal and groovy.
  __________

  [Original Description]

  == Comment: #0 - VASANT HEGDE <hegdevasant at in.ibm.com> - 2020-11-23 23:23:22 ==
  ---Problem Description---
  opal-prd fails to start on 20.04

  Contact Information = Vasant hegde <hegdevasant at linux.vnet.ibm.com>

  ---uname output---
  Ubuntu 20.04

  Machine Type = All Power System

  ---Steps to Reproduce---
   opal-prd fails to start on 20.04

  Userspace tool common name: opal-prd

  The userspace tool has the following bit modes: 64bit

  Userspace rpm: opal-prd

  This is fixed in upstream by below commit. Please backport this patch
  to 20.04 LTS release. Also applicable for 20.10.

  commit 47005e8d4c9aeda5826c17c4a013cfbda1a3f2de
  Author: Georgy Yakovlev <gyakovlev at gentoo.org>
  Date:   Mon Oct 12 14:29:17 2020 -0700

      opal-prd: handle devtmpfs mounted with noexec

      On systems using recent versions of systemd /dev (devtmpfs) is mounted with
      noexec option. Such mount prevents mapping HBRT image code region as RWX
      from /dev. This commit, as suggested in github PR linked below, attempts to
      work around the situation by copying HBRT image to anon mmaped memory
      region and sets mprotect rwx on it, allowing opal-prd to sucessfully
      execute the code region.

      Having memory region set as RWX is not ideal for security, but fixing that
      is a separate and hard to solve problem. Original code also mmaped region
      as RWX, so this PR does not make things worse at least.

      Closes: https://github.com/open-power/skiboot/issues/258
      Signed-off-by: Georgy Yakovlev <gyakovlev at gentoo.org>
      Reviewed-by: Vasant Hegde <hegdevasant at linux.vnet.ibm.com>
      [oliver: whitespace fix, add a comment, reflow commit message]
      Signed-off-by: Oliver O'Halloran <oohall at gmail.com>

  -Vasant

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1905393/+subscriptions