[Bug 1711203] Re: Deployments fail when Secure Boot enabled
Andres Rodriguez
andreserl at ubuntu-pe.org
Fri Feb 23 01:13:42 UTC 2018
On Thu, Feb 22, 2018 at 7:55 PM, Jeff Lane <jeffrey.lane at canonical.com>
wrote:
> On Thu, Feb 22, 2018 at 6:28 PM, Steve Langasek
> <steve.langasek at canonical.com> wrote:
> > On Thu, Feb 22, 2018 at 11:06:51PM -0000, Jeff Lane wrote:
> >> > Is /efi/ubuntu/grubx64.efi on your EFI System Partition definitely the
> >> > Canonical-signed image from grub-efi-amd64-signed?
> >
> >> I presume so? dpkg says it is:They look the same to me:
> >
> >> ubuntu at xwing:/boot/efi/EFI/ubuntu$ dpkg -S grubx64.efi
> >> grub-efi-amd64-signed: /usr/lib/grub/x86_64-efi-
> signed/grubx64.efi.signed
> >
> > That doesn't establish that /usr/lib/grub/x86_64-efi-
> signed/grubx64.efi.signed
> > and /boot/efi/EFI/ubuntu/grubx64.efi match. Can you please verify that
> they
> > do?
>
> Doh!... indeed.
> ubuntu at xwing:~$ md5sum /boot/efi/EFI/ubuntu/grubx64.efi
> /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
> 474a3900382e54c2129626683f12f3b5 /boot/efi/EFI/ubuntu/grubx64.efi
> 474a3900382e54c2129626683f12f3b5
> /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
> ubuntu at xwing:~$ diff -s /boot/efi/EFI/ubuntu/grubx64.efi
> /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
> Files /boot/efi/EFI/ubuntu/grubx64.efi and
> /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed are identical
>
> >> > Which version of Ubuntu's grub are you booting via pxe?
> >
> >> ubuntu at xwing:/boot/efi/EFI/ubuntu$ dpkg -l |grep grub|awk '{print
> $2": "$3}'
> >> grub-common: 2.02~beta2-36ubuntu3.16
> >> grub-efi-amd64: 2.02~beta2-36ubuntu3.16
> >> grub-efi-amd64-bin: 2.02~beta2-36ubuntu3.16
> >> grub-efi-amd64-signed: 1.66.16+2.02~beta2-36ubuntu3.16
> >> grub-pc: 2.02~beta2-36ubuntu3.16
> >> grub-pc-bin: 2.02~beta2-36ubuntu3.16
> >> grub2-common: 2.02~beta2-36ubuntu3.16
> >
> >> That is what is installed on the node.
> >
> > Sorry, I was asking about the other end of this: what version of
> > grubnetx64.efi is being served by maas?
>
> I have no idea. Andres?
>
> As far as I can tell, it's serving up a copy of grubx64.efi out of
> /var/lib/maas/boot-resources/current
>
> which has files dated Feb 5.
> bladernr at critical-maas:/var/lib/maas/boot-resources/
> current/bootloader/uefi/amd64$
> ll
> total 2328
> drwxr-xr-x 2 maas maas 4096 Feb 22 17:34 ./
> drwxr-xr-x 4 maas maas 4096 Feb 22 17:34 ../
> -rw-r--r-- 2 maas maas 1196736 Feb 5 07:29 bootx64.efi
> -rw-r--r-- 2 maas maas 1173368 Feb 5 07:29 grubx64.efi
>
> That all comes from maas.io.
>
> I presume its one of these?
>
> http://images.maas.io/ephemeral-v3/daily/streams/v1/
> com.ubuntu.maas:daily:1
> :bootloader-download.json
Whichever is the latest version in -updates at the time the streams were
created.
But yes, the latest version on the bootloader stream.
>
>
>
> >
> > (But it is also good to confirm what version of grub is installed on the
> > node's disk.)
> >
> >> So I re-enabled SecureBoot and removed all NICs from the boot order. I
> >> added in the HDD (since this is an EFI boot, the HDD is an entry called
> >> "Ubuntu" under "OTHER" in the boot order)
> >
> >> This fails to boot, I get an error from the system:
> >
> >> Error 1962: No operating system found. Boot sequence will automatically
> >> repeat.
> >
> >> Because I have no NICs listed in the boot order, this just churns as it
> >> keeps retrying the HDD entry.
> >
> >> So next, I went back and disabled SecureBoot once more. It immediately
> >> booted straight from the HDD.
> >
> >> I also just tried a USB install with Secure Boot enabled. I was able to
> >> install bionic from USB, but it too fails to boot with the same error.
> >
> >> To be fair at this point, given that this does work elsewhere, I'm
> >> suspicious that this is possibly an issue with my server.
> >
> > Agreed. Something is wrong with the boot configuration of this node,
> which
> > is independent of the question of whether we have a viable workaround for
> > the netboot chainloading bug.
>
> I'm going to see if I can update the firmware on this node and maybe
> that will make a difference. Otherwise, we'll need to try that C240
> in the lab.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1711203
>
> Title:
> Deployments fail when Secure Boot enabled
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1711203/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=curtin; status=Invalid; importance=Undecided;
> assignee=None;
> Launchpad-Bug: product=dellserver; status=New; importance=Undecided;
> assignee=None;
> Launchpad-Bug: product=maas; milestone=2.3.0; status=In Progress;
> importance=High; assignee=andreserl at ubuntu-pe.org;
> Launchpad-Bug: product=maas; productseries=2.3; milestone=2.3.1; status=In
> Progress; importance=High; assignee=andreserl at ubuntu-pe.org;
> Launchpad-Bug: product=maas-images; status=Fix Released;
> importance=Critical; assignee=lee.trager at canonical.com;
> Launchpad-Bug: distribution=ubuntu; sourcepackage=shim; component=main;
> status=In Progress; importance=High; assignee=mathieu.tl at gmail.com;
> Launchpad-Bug-Tags: blocks-hwcert-server id-5a28802797729aedf99dcd37
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl bladernr cyphermox jwezel ltrager
> narindergupta raharper rodsmith vorlon
> Launchpad-Bug-Reporter: Rod Smith (rodsmith)
> Launchpad-Bug-Modifier: Jeff Lane (bladernr)
> Launchpad-Message-Rationale: Assignee
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to shim in Ubuntu.
https://bugs.launchpad.net/bugs/1711203
Title:
Deployments fail when Secure Boot enabled
Status in curtin:
Invalid
Status in dellserver:
New
Status in MAAS:
In Progress
Status in MAAS 2.3 series:
In Progress
Status in maas-images:
Fix Released
Status in shim package in Ubuntu:
In Progress
Bug description:
I've recently encountered a problem with deploying nodes on which
Secure Boot is enabled. The symptoms are:
1. The node enlists and commissions fine
2. The node boots and begin deploying fine
3. After deployment completes, the node reboots
4. When booting at this point, after showing a few routine messages,
including a GRUB menu, the node displays the following text on its
screen:
error: invalid video mode specification `text'.
Booting in blind mode
Bootloader has not verified loaded image
System is compromised. halting
Disabling Secure Boot on the node enables it to boot. If this is done
quickly enough, deployment will succeed.
I've encountered this problem on two systems managed by two MAAS
servers: An Intel NUC DC53247HYE and a Cisco UCS C-240 M4 (VIC). One
MAAS server is running 2.2.2 (6099-g8751f91-0ubuntu1~16.04.1) and the
other is running 2.2.1 (6078-g2a6d96e-0ubuntu1~16.04.1). I'm attaching
log files from the first server to this bug report. The affected node
is brennan on that server.
Further observations:
* Once booted, I see that there's no kernel with a .efi.signed extension on
the hard disk. Installing such a kernel does NOT fix the problem;
however, it may be necessary to install such a kernel for a proper fix.
* If I force a boot directly through the Shim and GRUB installed on the
hard disk, the system boots correctly, even with Secure Boot enabled.
I found a copy of the error message in Shim source code, and reports
of this message on Fedora as early as 2014:
* https://github.com/rhboot/shim/blob/master/replacements.c
* https://ask.fedoraproject.org/en/question/39126/bootloader-has-not-verified-loaded-image/
It looks to me as if the Shim that MAAS uses for the post-deployment
boot has been updated/changed to include this strict verification that
the kernel is honoring Secure Boot rules; but the Shim installed to
the hard disk, and used during enlistment and commissioning, does not
perform this check. OTOH, I can't find any evidence of separate Shim
binaries on the MAAS server.
MAAS version information from one server:
$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================================-============-==================================================
ii maas 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cert-server 0.2.30-0~76~ubuntu16.04.1 all Ubuntu certification support files for MAAS server
ii maas-cli 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS server common files
ii maas-dhcp 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS DHCP server
ii maas-dns 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS DNS server
ii maas-proxy 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all Rack Controller for MAAS
ii maas-region-api 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.2.2-6099-g8751f91-0ubuntu1~16.04.1 all MAAS server provisioning libraries (Python 3)
To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1711203/+subscriptions
More information about the foundations-bugs
mailing list