ACK: [SRU bionic/focal/groovy/hirsure LRM] [PATCH 0/9] LP#1918134 -- LRMv4 switch to signing with Ubuntu Kernel Modules signing key

Tue Mar 9 09:36:12 UTC 2021

On 08.03.21 15:59, Andy Whitcroft wrote:
> Currently we build Nvidia packages in the main kernel package (linux,
> linux-aws etc).  We do this to sign the resulting .ko's into the kernel
> ephemeral modules signing key.  We package up only the added signature
> data (as .sig files) and destroy the remainder.  We later rebuild the .o's
> (and temporarily the .ko's) in linux-restricted-modules* and include the
> .o's so produced and the .sig's as produced by the main kernel build in
> LRM's binary packages.  These are then reconstructed on installation on
> the end-user system.
> 
> Now this has several unfortunate properties.  Firstly we have to access
> and build dkms packages from the restricted component in the main kernel
> package build (which is hard and error prone).  Secondly, we must be able
> to reproducibly build the .ko's as we build them in both the main package
> and LRM.  (Recently a toolchain change seems to have made this build no
> longer reproducible leading to modules which do not match the produced
> signatures and are thus not loadable.)  Finally, in order to bump the
> version of any particular Nvidia stream (or add or remove a stream) we
> have to rebuild the main kernel package in order to get signatures which
> is very costly and leads to unnecessary kernel updates for people who do
> not even use Nvidia modules.
> 
> As background, in recent kernel uploads we have included two new static
> keys into the kernel `.builtin_trusted_keys`; the Canonical Livepatch key,
> and the Ubuntu Kernel Modules key.  This second key is designed to allow
> signing of driver .ko's into the kernel even after the build is completed.
> With this in place it is no longer necessary for us to build the Nvidia
> packages in the main kernel package to obtain signatures.
> 
> Signing of .ko's with the Ubuntu Drivers Key (UDK) is performed via a
> signing custom binary upload containing those .ko's.  This signing either
> produces a signed .ko file, or optionally a detached .sig signature.
> For our purposes the latter is our desired form as we cannot distribute
> the combined .ko files.  It is very important to note that as we
> cannot distribute the combined .ko files for these Nvidia packages
> (due to liscencing constraints) that we also cannot allow the resultant
> custom binary upload to be distributed either; ie it cannot end up on the
> launchpad librarian or published into the public archive.  This constraint
> is a key design feature if the implementation described below.
> 
> In the new world order (LRMv4) the linux-restricted-modules (LRM) package
> is responsible for the only build of the nvidia .o's and the helpers which
> allow those to be combined on the end-user system to produce final loadable
> .ko's; this avoids the build reproducibility issues.  Two new packages are
> introduced, linux-restricted-generate (LRG) and linux-restricted-signatures
> (LRS).  LRG consumes the raw .o's and helpers and uses those to reconstruct
> the unsigned .ko's and places those into the signing custom binary upload.
> LRS simply downloads the resulting signatures from the archive and
> expresses those directly into the filesystem in the same directory as the
> .o's expressed by LRM.  LRM's binary packages then directly depend on these
> signatures such that when the postinst there runs we are able to add those
> signatures to the unsigned .ko's forming loadable .ko's for the kernel.
> 
> In order to allow LRG to simply consume the raw .o's and build helpers via
> a Build-Depend: we have restructured LRM's binary packages.  We split the
> raw .o's and build helpers from the build dependancies and postinst/prerm
> support.  The former is needed both by LRG and on the end-user system,
> the latter is only needed on the end-user system to construct and
> destroy the signed .ko's.  The raw .o's and build scripts are moved
> into new linux-objects-nvidia-460-ABI-FLAVOUR packages.  The existing
> linux-modules-nvidia-460-ABI-FLAVOUR packages simply depend on that and
> the associated signature package, and carry the postinst/prerm support.
> 
> Building this package set is complex as the LRG package cannot be built
> in a public PPA as that would cause the signing custom binary upload to
> be redistributed by the launchpad librarian.  We therefore must build
> LRG and LRS in a private PPA.  The built LRG and LRS must then be signed
> using the Ubuntu Kernel Modules key (again privatly) and the resultant
> LRS returned for distribution; this occurs via a private signing PPA
> containing the appropriate key; LRG is abandoned at this stage and takes
> no futher part.  As LRM and LRS are redistributable (and indeed both are
> required to install the drivers) LRS is returned to the public build PPA
> where LRM was built and enters the normal pipeline.
> 
> As LRG depends on binary packages produced by LRM the private PPA in which
> LRG will build must depend on the PPA in which LRM builds.  To support
> this we introduce an adjunct PPA for each public build PPA which will
> produce kernels which contain an LRM component.  These are needed for
> both of the stable PPAs (ppa and ppa2) and the development PPAs (bootstrap
> and unstable); each is simply named using the primary PPA name with a -ps
> (Private Signing) suffix.  Below is a flow-chart showing the lifecycles
> for these packages:
> 
>           LRM                 LRS LRG
>            |   .LRS(-b).       |   |
>            v   v       |       v   v
>      +------------+    |   +------------+
>      |    ppa     |    |   |   ppa-ps   |
>      +------------+    |   +------------+
>            |           |       |   |
>            |           |      LRS LRG(-b)
>            v           |       v   v
>      +------------+    |   +------------+
>      | -proposed  |    `---|  signing   |
>      +------------+        +------------+
> 
> This new proceedure is complex and error prone as (nominally) this requires
> the addition and maintenance of two new packages.  Also of the three
> packages in this set one is initially uploaded to the public PPA with the
> rest of the kernel (linux, linux-signed, and linux-meta) and the other two
> must be uploaded to a specific related private PPA (it must depend on the
> associated public PPA); indeed if LRG is uploaded to the public PPA we may
> end up redistributing nvidia .ko's which must be prevented at all costs.
> 
> To avoid the complexity and associated risk of misplacing these new
> packages the LRG and LRS package source are folded into the LRM package.
> This package will continue to be maintained in the normal way and uploaded
> to the public build PPA as normal, meaning no process change for crankers.
> LRS and LRG are (intentionally) generable from LRM simply by changing the
> name of the source package in debian/changelog.  A cron based helper will
> monitor the primary PPAs for new LRM uploads and automatically convert
> that into the necessary LRG/LRS pairing and upload it to the correct
> adjunct PPA.  The tracking bugs associated with our uploads will drive
> the process from there to sign and return LRS to the primary build PPA.
> 
> The patch stack below represents the nominal conversion of the
> hirsute:linux package from LRMv3 to LRMv4.  However (as we did with LRMv3)
> I proposed to apply this update programatically to all LRM packages
> once this is reviewed.
> 
> This stack does not contain the (further) changes we will apply to the main
> kernel packages to stop building the nvidia signatures there.  We also
> continue to assume that the primary kernel package for a version-series
> (bionic:linux) still carries the definative debian/dkms-versions file so
> that we retain consistent versioning on these packages[1]; though this
> does not preclude LRMs being respun independently during a cycle.
> 
> -apw
> 
> [1] the next phase of this work will revisit the home for this version
>      data going forward, and the aquisition of the DKMS packages during
>      the builds, and later.
> ---
> 
> Branch: autogen4
> 
> Andy Whitcroft (9):
>    UBUNTU: [Packaging] parameterise package generator based on package
>    UBUNTU: [Packaging] gen-rules -- clean up debian/rules fragment
>    UBUNTU: [Packaging] dkms-build--nvidia-N -- handle symbol versioning being enabled
>    UBUNTU: [Packaging] dkms-build--nvidia-N -- follow changes to lds naming
>    UBUNTU: [Packaging] dkms-build--nvidia-N -- add unsigned build mode to helpers
>    UBUNTU: [Packaging] linux-restricted-generate -- generate unsigned modules for signing
>    UBUNTU: [Packaging] linux-restricted-signatures -- publish clean signatures
>    UBUNTU: [Packaging] linux-restricted-modules -- consume published signatures
>    UBUNTU: [Packaging] update-version -- handle forward-backport versioning
> 
>   debian/rules.in                     |  26 +---
>   debian/rules.lrg                    |  31 ++++
>   debian/rules.lrs                    |  12 ++
>   debian/scripts/dkms-build--nvidia-N |  63 ++++----
>   debian/scripts/gen-rules            | 179 +---------------------
>   debian/scripts/gen-rules.lrg        | 138 +++++++++++++++++
>   debian/scripts/gen-rules.lrm        | 226 ++++++++++++++++++++++++++++
>   debian/scripts/gen-rules.lrs        | 100 ++++++++++++
>   debian/source/options               |   2 +-
>   download-signed                     | 183 ++++++++++++++++++++++
>   scripts/module-common.lds           |  26 ----
>   update-version                      |   5 +-
>   12 files changed, 741 insertions(+), 250 deletions(-)
>   create mode 100755 debian/rules.lrg
>   create mode 100755 debian/rules.lrs
>   create mode 100755 debian/scripts/gen-rules.lrg
>   create mode 100755 debian/scripts/gen-rules.lrm
>   create mode 100755 debian/scripts/gen-rules.lrs
>   create mode 100755 download-signed
>   delete mode 100644 scripts/module-common.lds
> 
Some details I ignored as those should either work and be tested or not. When we 
follow the approach of first see this in action on devel and then roll this out 
programmatically, then we should be on the safe side.

Acked-by: Stefan Bader <stefan.bader at canonical.com>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20210309/a015a1e5/attachment-0001.sig>