[Bug 1895131] Re: Groovy Desktop *BREAKS* the most common method of creating UEFI bootable drives for Ubuntu installation

[Akeo]

Hi Thomas. Glad to see you onboard here as well. ;)

First of all, I guess I should point out that UEFI bootloader boot from
non ESP partitions is hardly a "trick". It is de-facto. That behaviour
comes from the EDK2 source itself and, as my recent dealings with EDK2
indicate (in [1]) that the EDK2/UEFI folks are more inclined to have the
existing EDK2 code drive the specs, especially for boot, than the
opposite. Which means that, if the default firmware you get from EDK2
pretty much sees any /efi/boot/boot###.efi bootloader that resides on a
non-ESP partition it has a file system for as something it should
provide as a boot option, then this is as established as if it was part
of the specs. Hence "de facto".

So I see little point in trying to stir this discussion into asking whether distros may be technically entitled not to bother with such a method of creating UEFI boot media as:
1. People *ARE* going to be attempting to use these methods, since they will be supported by their platform's UEFI firmware, regardless of whether they are explicitly backed up by the specs or not.
2. Even if General Data partition boot was to be formally outlawed tomorrow, it's not that difficult to change the partition type to ESP when creating the media in the manner I highlighted, and then we're back to square one, with missing EFI bootloaders on the ISO-9660 file system preventing boot by simply extracting the content to an ESP partition.

So since you provided a comprehensive explanation (thanks!) as to why
distros took it to moving to using a boot/grub/efi.img for the EFI
bootloaders (I see where you're coming from now, but it's a shame you
didn't consider the need for folks to be able to create bootable media
by simply extracting the full ISO-9660 content onto their own data
partition) then let me provide my Linux ISO creation requirements, RFC
style:

For the purpose of providing a method of creation of UEFI boot
installation media by simply enabling the user to extract the ISO
content onto an existing data partition, a Linux installation ISO:

1. MUST NOT use symbolic links, for the file systems that it implements
on the ISO (through the "hybrid" part), for any critical part of the
boot or installation process. In other words, while one may create a
symbolic '/doc/' link at the root level that points to something like
'/release/documentation/', it should not use a symbolic link for
vmlinuz, initrd or any of the installation packages, as these are
unlikely to be preserved during ISO extraction onto an alternate file
system.

2. MUST ensure that, along with ISO-9660, it supports FAT32 as a file
system that can be used during installation (meaning that the default
installation kernel should not forcibly remove vfat as a supported file
system, and that the installation scripts should be able to handle the
retrieval of content that resides on FAT file systems, and not just
ISO-9660). Note that this point is conditional to not requiring the use
of any file that larger than 4 GB. If that cannot be achieved then the
next SHOULD becomes a MUST.

3. SHOULD ensure that, along with ISO-9660, it supports exFAT as a file
system that can be used during installation. The obvious reasoning
behind this has to do with the 4 GB limitation of FAT32 when it comes to
individual files, which, thankfully, mostly due to the use of squashfs,
is not yet a limit that is being encountered by distro installers, but
that it makes sense to plan for. With Microsoft having recently granted
"free use" of exFAT for the Linux kernel, it makes sense to foster the
use of exFAT over alternatives like NTFS, as exFAT has also the
advantage of being natively supported on MacOS.

4. MUST ensure that UEFI bootloaders, that enable the creation of
bootable installation media by extracting the ISO-9660 content straight
into a FAT32/exFAT, are present on the ISO-9660 file system in their
expected location ('/efi/boot/boot####.efi').


If the above is respected, and right now Ubuntu only needs to sort out point 4, then the simplest and, I will assert, safest method of creating a UEFI bootable installation media can be expected to work.

It should be noted that a direct corollary of points 2. and 3. is that
the naming scheme used for the ISO-9660 content needs to be more
restrictive than what Rock-Ridge or Joliet allows, as, for instance,
case sensitivity and special characters have to carefully considered,
else a file name lookup that might work in an ISO-9660 environment might
be broken for content that was extracted to FAT32/exFAT.

Also, whereas the exFAT push might seems a bit pointless for now, since
the EDK2 doesn't provide a native exFAT driver, it should be noted that
there does exist methods to load the required exFAT UEFI driver and then
automatically chain load the EFI bootloader from the exFAT partition
(I'm actually using such a method in Rufus, by creating a small 512KB
"ESP" at the end of the drive [2], though right now this is mostly used
for NTFS, which Ubuntu actually was supporting just fine last time I
tested) and, unless someone does it before me, I do have some medium-
term plan of bringing an exFAT driver proposal to EDK2. Once I do that,
and if integrated, we should start to see UEFI platforms with native
exFAT boot support, rendering the need to an additional hack for > 4 GB
file support obsolete.

I'm hoping that the proposal highlighted above makes sense to all the
parties involved.

Regards,

/Pete

[1] https://bugzilla.tianocore.org/show_bug.cgi?id=2831#c14
[2] https://github.com/pbatard/uefi-ntfs (Don't let the name fool you, this also supports exFAT)

** Bug watch added: bugzilla.tianocore.org/ #2831
   https://bugzilla.tianocore.org/show_bug.cgi?id=2831

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to casper in Ubuntu.
https://bugs.launchpad.net/bugs/1895131

Title:
  Groovy Desktop *BREAKS* the most common method of creating UEFI
  bootable drives for Ubuntu installation

Status in casper package in Ubuntu:
  Confirmed

Bug description:
  As opposed to 20.04, Groovy Desktop decided to do away with the EFI
  boot files that used to reside in `/EFI/BOOT/` on the ISO, and instead
  moved them away into a FAT image located at /boot/grub/efi.img.

  While this works fine when writing the image in DD mode, it completely
  **BREAKS** the common method, used by many many users on Windows,
  MacOS and other platforms, of simply formatting a USB drive to FAT32
  and then copying the whole ISO content there in order to create a UEFI
  bootable Ubuntu installation media.

  Please bear in mind that this method of creating UEFI boot media is favoured by many on account that:
  * It doesn't require the installation of third party software like balenaEtcher or Rufus to create the media, especially on Windows
  * It is less risky to use than dd, on account that it's less prone to making a mistake with regards to the target disk. Especially, not everyone has access to 'dd', or is familiar enough with it, or even wants to use it if there exists an alternative that lets them access the content of their drive (e.g. on Windows).
  * It leverages the *NATIVE* ability of UEFI to pick a bootloader from /EFI/BOOT/ which was introduced precisely to make the creation of bootable media through third party utilities (including dd) a thing of the past.

  So, let me start by giving a stern warning here:

  UBUNTU PEOPLE, BY HIDING THE EFI BOOT FILES AWAY IN THE ISO, YOU ARE
  **NOT** HELPING YOUR USERS. INSTEAD, YOU ARE ACTIVELY **DEGRADING**
  THE USER INSTALLATION EXPERIENCE. PLEASE DON'T DO THAT!

  My questions therefore are twofold:

  1. What on earth was the rationale behind this move? What exactly is
  to be gained here?? Ubuntu 20.4 was perfectly fine with the GRUB boot
  files in /EFI/BOOT/ on the ISO file system structure, and, as pointed
  out above, it's hard to see how hiding these files away in efi.img is
  going to improve user experience when this breaks the simplest method
  of creation of a UEFI bootable media. So what prompted this sudden
  unwarranted change, and why didn't anybody realize that this would
  make the Ubuntu media creation experience subpar in terms of UEFI
  install?

  2. Can this change please, please, **PLEASE**, be reverted? I know
  that drinking the ISOHybrid kool-aid and putting your eggs into one
  basket by declaring that `dd` is now the "one true way" of creating
  UEFI bootable media is very seducing from a maintainer's perspective.
  But don't remove features that helped foster the image of Ubuntu being
  focused on user-friendliness, and that are **ACTUALLY** used by more
  people than you realize. Else you may find that a move that actively
  prevents people from installing Linux in a manner they've been using
  for years, and that really has no reason to be broken because it's
  what UEFI was designed for, will be percieved as a **STRONG
  INDICATION** that Ubuntu is no longer caring about its users...

  Thank you.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1895131/+subscriptions