[Bug 2150354] [NEW] ZFS_LUKS_KEYSTORE installation drops to emergency shell on first boot due to missing hostid seeding in /target
Juanmi Taboada
2150354 at bugs.launchpad.net
Sun Apr 26 07:54:22 UTC 2026
Public bug reported:
AFFECTED PACKAGE
----------------
subiquity (likely also curtin, depending on which component owns the ZFS
pool creation step)
VERSIONS OBSERVED
-----------------
- Ubuntu 26.04 (Resolute Resilient) - desktop ISO, daily build of 2026-04-25
- Kernel: 7.0.0-14-generic
- ZFS: 2.4.1-1ubuntu5 (libzfs7linux, zfs-dracut, zfsutils-linux)
- Storage layout: ZFS_LUKS_KEYSTORE (guided full-disk encryption)
- Target disk: USB-attached SSD (SanDisk Portable SSD, 2 TB),
/dev/sda enumerated as usb-SanDisk_Portable_SSD_*
SEVERITY
--------
High - installed system is unbootable without manual intervention from the
dracut emergency shell. The user must know to run "zpool import -f" for both
pools, which is not obvious from the failure message and not covered in any
installer-facing documentation.
SUMMARY
-------
When installing Ubuntu 26.04 with guided storage option ZFS_LUKS_KEYSTORE
onto a USB-attached SSD, the freshly installed system fails to boot directly
on first boot. It drops into the dracut emergency shell with:
dracut-pre-mount[958]: Cannot get properties of rpool: no such pool available.
[FAILED] Failed to mount sysroot.mount - /sysroot.
>From the emergency shell, "zpool import" (without arguments) shows both pools
as ONLINE but flagged with ZFS-8000-EY:
pool: bpool
id: 11158846221490413465
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier
and the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
pool: rpool
id: 2362008108274967906
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier
and the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
After running "zpool import -f -N rpool && zpool import -f -N bpool" from the
emergency shell and pressing Ctrl+D, boot resumes correctly: the dracut
keystore module prompts for the passphrase via plymouth, opens the LUKS
keystore zvol, the raw ZFS key is loaded, and the system reaches the
desktop normally.
As a side effect of the forced import, ZFS rewrites the hostid stamped into
the pools to match the installed system's hostid. Subsequent boots therefore
succeed without intervention - the system self-heals after the first manual
recovery. The boot pipeline is intact; only the *initial* hostid handshake
between live ISO and installed target is broken.
ROOT CAUSE
----------
The installer creates both pools (bpool, rpool) inside the live ISO
environment using whichever hostid the live happens to have. It does NOT
seed /target/etc/hostid to match, nor does it propagate a coherent
zpool.cache into the target before the initramfs is built. As a result, on
first boot the installed system has a hostid that does not match the one
embedded in the pools' on-disk configuration, and ZFS refuses to import them
without -f.
EVIDENCE FROM THE INSTALLER SYSLOG
----------------------------------
Relevant timeline from /var/log/syslog of the installer session (full log
attached):
09:41:50 zpool create -o ashift=12 -o autotrim=on
-o feature at async_destroy=enabled ... -O canmount=off
-O mountpoint=/boot -R /target -d bpool
/dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part2
09:41:50 zpool set cachefile=/etc/zfs/zpool.cache bpool
(writes cachefile inside the live ISO, NOT /target)
09:41:52 zpool create -o ashift=12 -O encryption=on
-O keylocation=file:///tmp/tmpl569ywhx -O keyformat=raw
-O mountpoint=/ -R /target rpool
/dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part4
09:41:52 zpool set cachefile=/etc/zfs/zpool.cache rpool
09:41:52 cryptsetup luksFormat --offset 32768
/dev/zvol/rpool/keystore /tmp/zpool-key-m4oyo6or
09:42:00 cryptsetup open --type luks /dev/zvol/rpool/keystore
keystore-rpool --key-file /tmp/zpool-key-m4oyo6or
09:44:25 apt-get install of cryptsetup, zfs-dracut, zfsutils-linux,
grub-efi-amd64-signed, shim-signed into /target
09:44:45 dracut: Generating /boot/initrd.img-7.0.0-14-generic
(inside chroot)
09:45:00 dpkg-reconfigure grub-efi-amd64, update-grub
09:45:37 Postinst hook 50_zfs_cache runs, syncs zfs-list.cache only
A targeted search across the entire installer syslog returns ZERO hits for
any of the following:
$ grep -iE 'hostid|zgenhostid|/etc/hostid' syslog
(no output)
Specifically, none of these expected operations occur during
installation:
- No zgenhostid is ever invoked.
- /etc/hostid is never copied from the live ISO to /target/etc/hostid.
- No "chroot /target zgenhostid" is run before
"dpkg-reconfigure linux-image-..." triggers the dracut initramfs build.
- The zpool.cache file generated by "zpool set cachefile=..." lives only
in the live ISO's /etc/zfs/ and is never transferred (in a coherent
state) into /target/etc/zfs/.
The first sign of trouble on boot is therefore:
zfs-import-cache.service - Import ZFS pools by cache file skipped,
unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
zfs-import-scan.service then finds the pools but refuses to import them
under the running system's hostid, which does not match the one stamped
into the pools at creation time inside the live.
STEPS TO REPRODUCE
------------------
1. Boot the Ubuntu 26.04 desktop ISO from a USB stick.
2. In the installer, choose "Erase disk and install Ubuntu" with
"Advanced features -> Use ZFS and encryption" (ZFS_LUKS_KEYSTORE
capability).
3. Select a USB-attached SSD as the install target. (Note: USB targets
reproduce this reliably; the same logic likely affects internal disks
under any condition that produces a hostid mismatch between live and
installed system, e.g. live ISO sessions where /etc/hostid is
regenerated per boot.)
4. Set a passphrase for the keystore and complete the installation.
5. Reboot, removing the install media.
Expected: System boots through GRUB, prompts for the keystore passphrase
via plymouth, reaches the GDM login screen.
Actual: GRUB loads the kernel and initramfs; dracut imports neither pool;
the system drops to the emergency shell. No plymouth keystore prompt is
ever displayed because dracut bails out before reaching that stage.
PROPOSED FIX
------------
In the curtin/subiquity ZFS install path, between pool creation and
initramfs generation, the installer must ensure the target system's hostid
matches the hostid embedded in the pools, and that the zpool.cache reflects
this state. A minimal correct sequence is:
# Generate /etc/hostid in the target (or inherit the live's)
chroot /target zgenhostid -f
# Make the live use the same hostid as the target so subsequent
# zpool operations stamp pools with the correct value.
cp /target/etc/hostid /etc/hostid
# Re-write the on-disk pool configuration so the recorded hostid
# matches what the installed system will use at boot time.
zpool set cachefile=/etc/zfs/zpool.cache bpool
zpool set cachefile=/etc/zfs/zpool.cache rpool
# Propagate the resulting cachefile into the target so
# zfs-import-cache.service can succeed on first boot.
install -d /target/etc/zfs
cp /etc/zfs/zpool.cache /target/etc/zfs/zpool.cache
This block must run BEFORE "dpkg-reconfigure linux-image-..." triggers
dracut, so that the generated initramfs captures both the correct
/etc/hostid and the correct zpool.cache.
A more robust variant exports the pools and re-imports them under the
target's hostid before the initramfs is built, but the simpler form above
is sufficient because "zpool set cachefile" rewrites the cached
configuration with the active hostid.
The natural place for this logic is the same curtin step that already runs
"zpool set cachefile=..." (visible in the syslog at 09:41:50 and 09:41:52),
or alternatively a new postinst hook executed before 50_zfs_cache.
WORKAROUND FOR AFFECTED USERS
-----------------------------
Recover the current installation
. . . . . . . . . . . . . . . . .
>From the dracut emergency shell, after the failed boot:
zpool import -f -N rpool
zpool import -f -N bpool
Then press Ctrl+D to exit the emergency shell. Boot will resume; plymouth
will prompt for the keystore passphrase, the system will decrypt and
continue to the desktop.
Make the fix permanent
. . . . . . . . . . . .
Once the pools have been imported with -f from the emergency shell at least
once, the on-disk hostid recorded in each pool is rewritten to match the
installed system's hostid as a side effect of the forced import. Subsequent
boots no longer drop to emergency mode and no further action is required
from the user - the system self-heals after the first manual recovery.
ADDITIONAL NOTES
----------------
- The dracut keystore integration provided by zfs-dracut 2.4.1-1ubuntu5
works correctly. The keystore zvol, the LUKS layer, the raw key, the
plymouth prompt, and "zfs load-key" all function as designed once the
pools are imported. This bug is strictly about pool import, not about
encryption.
- syslog: full installer session log attached, with "grep -i hostid"
returning zero matches.
- Note: The attached syslog has been thoroughly sanitized over three
iterative review passes (75 redaction patterns verified with zero
residuals) before public submission. Replacements use clearly-marked
placeholders (REDACTED_USERNAME, [REDACTED-CPU], REDACTED_TARGET_SERIAL,
etc.) so structure and timestamps remain intact. Categories redacted
include: identity (username/hostname were exposed letter-by-letter via
validate_username GETs - note this itself may warrant a separate privacy
bug report), geolocation, hardware fingerprinting (CPU, GPU, BIOS,
memory, screen resolution and physical size, chipsets for WiFi/BT/audio/
Ethernet, USB peripherals), MAC addresses, WiFi SSIDs and BSSIDs, local
IPs, NetworkManager UUIDs, disk vendor/model/serial across all probed
devices, snap revision numbers, ZFS dataset random suffixes, ephemeral
systemd scope UUIDs, swap and FAT32 volume serials, and Windows BCD
identifiers from co-existing OS installations. The 8184-line structure,
all timestamps, and all technical content relevant to the bug are
intact. To verify the bug's root cause from this log:
"grep -i 'hostid' syslog_sanitized_v3.txt" returns zero hits,
demonstrating that no /etc/hostid write occurred during installation.
** Affects: subiquity (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "syslog has been sanitized to redact private information"
https://bugs.launchpad.net/bugs/2150354/+attachment/5965653/+files/syslog_sanitized.txt.gz
** Description changed:
- # ZFS_LUKS_KEYSTORE installation drops to emergency shell on first boot
+ ================================================================================
+ ZFS_LUKS_KEYSTORE installation drops to emergency shell on first boot
due to missing hostid seeding in /target
-
- ## Affected package
-
- `subiquity` (likely also `curtin`, depending on which component owns the
- ZFS pool creation step)
-
- ## Versions observed
+ ================================================================================
+
+
+ AFFECTED PACKAGE
+ ----------------
+
+ subiquity (likely also curtin, depending on which component owns the ZFS
+ pool creation step)
+
+
+ VERSIONS OBSERVED
+ -----------------
- Ubuntu 26.04 (Resolute Resilient) - desktop ISO, daily build of 2026-04-25
- - Kernel: `7.0.0-14-generic`
- - ZFS: `2.4.1-1ubuntu5` (`libzfs7linux`, `zfs-dracut`, `zfsutils-linux`)
- - Storage layout: `ZFS_LUKS_KEYSTORE` (guided full-disk encryption)
- - Target disk: USB-attached SSD (`SanDisk Portable SSD`, 2 TB), `/dev/sda` enumerated as `usb-SanDisk_Portable_SSD_*`
-
- ## Severity
-
- **High** - installed system is unbootable without manual intervention
- from the dracut emergency shell. The user must know to run `zpool import
- -f` for both pools, which is not obvious from the failure message and
- not covered in any installer-facing documentation.
-
- ## Summary
-
- When installing Ubuntu 26.04 with guided storage option
- `ZFS_LUKS_KEYSTORE` onto a USB-attached SSD, the freshly installed
- system fails to boot directly on first boot. It drops into the dracut
- emergency shell with:
-
- ```
- dracut-pre-mount[958]: Cannot get properties of rpool: no such pool available.
- [FAILED] Failed to mount sysroot.mount - /sysroot.
- ```
-
- From the emergency shell, `zpool import` (without arguments) shows both
- pools as `ONLINE` but flagged with `ZFS-8000-EY`:
-
- ```
- pool: bpool
- id: 11158846221490413465
- state: ONLINE
- status: The pool was last accessed by another system.
- action: The pool can be imported using its name or numeric identifier
- and the '-f' flag.
- see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
-
- pool: rpool
- id: 2362008108274967906
- state: ONLINE
- status: The pool was last accessed by another system.
- action: The pool can be imported using its name or numeric identifier
- and the '-f' flag.
- see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
- ```
-
- After running `zpool import -f -N rpool && zpool import -f -N bpool`
- from the emergency shell and pressing Ctrl+D, boot resumes correctly:
- the dracut keystore module prompts for the passphrase via plymouth,
- opens the LUKS keystore zvol, the raw ZFS key is loaded, and the system
- reaches the desktop normally.
-
- **As a side effect of the forced import, ZFS rewrites the hostid stamped
- into the pools to match the installed system's hostid. Subsequent boots
- therefore succeed without intervention, the system self-heals after the
- first manual recovery.** The boot pipeline is intact; only the *initial*
- hostid handshake between live ISO and installed target is broken.
-
- ## Root cause
-
- The installer creates both pools (`bpool`, `rpool`) inside the live ISO
- environment using whichever hostid the live happens to have. It does
- **not** seed `/target/etc/hostid` to match, nor does it propagate a
- coherent `zpool.cache` into the target before the initramfs is built. As
- a result, on first boot the installed system has a hostid that does not
- match the one embedded in the pools' on-disk configuration, and ZFS
- refuses to import them without `-f`.
-
- ### Evidence from the installer syslog
-
- Relevant timeline from `/var/log/syslog` of the installer session (full
- log attached):
-
- | Time | Action |
- |---|---|
- | 09:41:50 | `zpool create -o ashift=12 -o autotrim=on -o feature at async_destroy=enabled ... -O canmount=off -O mountpoint=/boot -R /target -d bpool /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part2` |
- | 09:41:50 | `zpool set cachefile=/etc/zfs/zpool.cache bpool` (writes cachefile **inside the live ISO**, not `/target`) |
- | 09:41:52 | `zpool create -o ashift=12 -O encryption=on -O keylocation=file:///tmp/tmpl569ywhx -O keyformat=raw -O mountpoint=/ -R /target rpool /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part4` |
- | 09:41:52 | `zpool set cachefile=/etc/zfs/zpool.cache rpool` |
- | 09:41:52 | `cryptsetup luksFormat --offset 32768 /dev/zvol/rpool/keystore /tmp/zpool-key-m4oyo6or` |
- | 09:42:00 | `cryptsetup open --type luks /dev/zvol/rpool/keystore keystore-rpool --key-file /tmp/zpool-key-m4oyo6or` |
- | 09:44:25 | `apt-get install` of `cryptsetup`, `zfs-dracut`, `zfsutils-linux`, `grub-efi-amd64-signed`, `shim-signed` into `/target` |
- | 09:44:45 | `dracut: Generating /boot/initrd.img-7.0.0-14-generic` (inside chroot) |
- | 09:45:00 | `dpkg-reconfigure grub-efi-amd64`, `update-grub` |
- | 09:45:37 | Postinst hook `50_zfs_cache` runs, syncs `zfs-list.cache` only |
-
- A targeted search across the entire installer syslog returns **zero
- hits** for any of the following:
-
- ```
- $ grep -iE 'hostid|zgenhostid|/etc/hostid' syslog
- (no output)
- ```
+ - Kernel: 7.0.0-14-generic
+ - ZFS: 2.4.1-1ubuntu5 (libzfs7linux, zfs-dracut, zfsutils-linux)
+ - Storage layout: ZFS_LUKS_KEYSTORE (guided full-disk encryption)
+ - Target disk: USB-attached SSD (SanDisk Portable SSD, 2 TB),
+ /dev/sda enumerated as usb-SanDisk_Portable_SSD_*
+
+
+ SEVERITY
+ --------
+
+ High - installed system is unbootable without manual intervention from the
+ dracut emergency shell. The user must know to run "zpool import -f" for both
+ pools, which is not obvious from the failure message and not covered in any
+ installer-facing documentation.
+
+
+ SUMMARY
+ -------
+
+ When installing Ubuntu 26.04 with guided storage option ZFS_LUKS_KEYSTORE
+ onto a USB-attached SSD, the freshly installed system fails to boot directly
+ on first boot. It drops into the dracut emergency shell with:
+
+ dracut-pre-mount[958]: Cannot get properties of rpool: no such pool available.
+ [FAILED] Failed to mount sysroot.mount - /sysroot.
+
+ From the emergency shell, "zpool import" (without arguments) shows both pools
+ as ONLINE but flagged with ZFS-8000-EY:
+
+ pool: bpool
+ id: 11158846221490413465
+ state: ONLINE
+ status: The pool was last accessed by another system.
+ action: The pool can be imported using its name or numeric identifier
+ and the '-f' flag.
+ see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
+
+ pool: rpool
+ id: 2362008108274967906
+ state: ONLINE
+ status: The pool was last accessed by another system.
+ action: The pool can be imported using its name or numeric identifier
+ and the '-f' flag.
+ see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
+
+ After running "zpool import -f -N rpool && zpool import -f -N bpool" from the
+ emergency shell and pressing Ctrl+D, boot resumes correctly: the dracut
+ keystore module prompts for the passphrase via plymouth, opens the LUKS
+ keystore zvol, the raw ZFS key is loaded, and the system reaches the
+ desktop normally.
+
+ As a side effect of the forced import, ZFS rewrites the hostid stamped into
+ the pools to match the installed system's hostid. Subsequent boots therefore
+ succeed without intervention - the system self-heals after the first manual
+ recovery. The boot pipeline is intact; only the *initial* hostid handshake
+ between live ISO and installed target is broken.
+
+
+ ROOT CAUSE
+ ----------
+
+ The installer creates both pools (bpool, rpool) inside the live ISO
+ environment using whichever hostid the live happens to have. It does NOT
+ seed /target/etc/hostid to match, nor does it propagate a coherent
+ zpool.cache into the target before the initramfs is built. As a result, on
+ first boot the installed system has a hostid that does not match the one
+ embedded in the pools' on-disk configuration, and ZFS refuses to import them
+ without -f.
+
+
+ EVIDENCE FROM THE INSTALLER SYSLOG
+ ----------------------------------
+
+ Relevant timeline from /var/log/syslog of the installer session (full log
+ attached):
+
+ 09:41:50 zpool create -o ashift=12 -o autotrim=on
+ -o feature at async_destroy=enabled ... -O canmount=off
+ -O mountpoint=/boot -R /target -d bpool
+ /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part2
+
+ 09:41:50 zpool set cachefile=/etc/zfs/zpool.cache bpool
+ (writes cachefile inside the live ISO, NOT /target)
+
+ 09:41:52 zpool create -o ashift=12 -O encryption=on
+ -O keylocation=file:///tmp/tmpl569ywhx -O keyformat=raw
+ -O mountpoint=/ -R /target rpool
+ /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part4
+
+ 09:41:52 zpool set cachefile=/etc/zfs/zpool.cache rpool
+
+ 09:41:52 cryptsetup luksFormat --offset 32768
+ /dev/zvol/rpool/keystore /tmp/zpool-key-m4oyo6or
+
+ 09:42:00 cryptsetup open --type luks /dev/zvol/rpool/keystore
+ keystore-rpool --key-file /tmp/zpool-key-m4oyo6or
+
+ 09:44:25 apt-get install of cryptsetup, zfs-dracut, zfsutils-linux,
+ grub-efi-amd64-signed, shim-signed into /target
+
+ 09:44:45 dracut: Generating /boot/initrd.img-7.0.0-14-generic
+ (inside chroot)
+
+ 09:45:00 dpkg-reconfigure grub-efi-amd64, update-grub
+
+ 09:45:37 Postinst hook 50_zfs_cache runs, syncs zfs-list.cache only
+
+ A targeted search across the entire installer syslog returns ZERO hits for
+ any of the following:
+
+ $ grep -iE 'hostid|zgenhostid|/etc/hostid' syslog
+ (no output)
Specifically, none of these expected operations occur during
installation:
- - No `zgenhostid` is ever invoked.
- - `/etc/hostid` is never copied from the live ISO to `/target/etc/hostid`.
- - No `chroot /target zgenhostid` is run before `dpkg-reconfigure linux-image-...` triggers the dracut initramfs build.
- - The `zpool.cache` file generated by `zpool set cachefile=...` lives only in the live ISO's `/etc/zfs/` and is never transferred (in a coherent state) into `/target/etc/zfs/`.
+ - No zgenhostid is ever invoked.
+ - /etc/hostid is never copied from the live ISO to /target/etc/hostid.
+ - No "chroot /target zgenhostid" is run before
+ "dpkg-reconfigure linux-image-..." triggers the dracut initramfs build.
+ - The zpool.cache file generated by "zpool set cachefile=..." lives only
+ in the live ISO's /etc/zfs/ and is never transferred (in a coherent
+ state) into /target/etc/zfs/.
The first sign of trouble on boot is therefore:
- ```
- zfs-import-cache.service - Import ZFS pools by cache file skipped,
- unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
- ```
-
- `zfs-import-scan.service` then finds the pools but refuses to import
- them under the running system's hostid, which does not match the one
- stamped into the pools at creation time inside the live.
-
- ## Steps to reproduce
+ zfs-import-cache.service - Import ZFS pools by cache file skipped,
+ unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
+
+ zfs-import-scan.service then finds the pools but refuses to import them
+ under the running system's hostid, which does not match the one stamped
+ into the pools at creation time inside the live.
+
+
+ STEPS TO REPRODUCE
+ ------------------
1. Boot the Ubuntu 26.04 desktop ISO from a USB stick.
- 2. In the installer, choose **Erase disk and install Ubuntu** with **Advanced features → Use ZFS and encryption** (`ZFS_LUKS_KEYSTORE` capability).
- 3. Select a USB-attached SSD as the install target. (Note: USB targets reproduce this reliably; the same logic likely affects internal disks under any condition that produces a hostid mismatch between live and installed system, e.g. live ISO sessions where `/etc/hostid` is regenerated per boot.)
+
+ 2. In the installer, choose "Erase disk and install Ubuntu" with
+ "Advanced features -> Use ZFS and encryption" (ZFS_LUKS_KEYSTORE
+ capability).
+
+ 3. Select a USB-attached SSD as the install target. (Note: USB targets
+ reproduce this reliably; the same logic likely affects internal disks
+ under any condition that produces a hostid mismatch between live and
+ installed system, e.g. live ISO sessions where /etc/hostid is
+ regenerated per boot.)
+
4. Set a passphrase for the keystore and complete the installation.
+
5. Reboot, removing the install media.
- **Expected:** System boots through GRUB, prompts for the keystore
- passphrase via plymouth, reaches the GDM login screen.
-
- **Actual:** GRUB loads the kernel and initramfs; dracut imports neither
- pool; the system drops to the emergency shell. No plymouth keystore
- prompt is ever displayed because dracut bails out before reaching that
- stage.
-
- ## Proposed fix
+ Expected: System boots through GRUB, prompts for the keystore passphrase
+ via plymouth, reaches the GDM login screen.
+
+ Actual: GRUB loads the kernel and initramfs; dracut imports neither pool;
+ the system drops to the emergency shell. No plymouth keystore prompt is
+ ever displayed because dracut bails out before reaching that stage.
+
+
+ PROPOSED FIX
+ ------------
In the curtin/subiquity ZFS install path, between pool creation and
- initramfs generation, the installer must ensure the target system's
- hostid matches the hostid embedded in the pools, and that the
- `zpool.cache` reflects this state. A minimal correct sequence is:
-
- ```sh
- # Generate /etc/hostid in the target (or inherit the live's)
- chroot /target zgenhostid -f
-
- # Make the live use the same hostid as the target so subsequent
- # zpool operations stamp pools with the correct value.
- cp /target/etc/hostid /etc/hostid
-
- # Re-write the on-disk pool configuration so the recorded hostid
- # matches what the installed system will use at boot time.
- zpool set cachefile=/etc/zfs/zpool.cache bpool
- zpool set cachefile=/etc/zfs/zpool.cache rpool
-
- # Propagate the resulting cachefile into the target so
- # zfs-import-cache.service can succeed on first boot.
- install -d /target/etc/zfs
- cp /etc/zfs/zpool.cache /target/etc/zfs/zpool.cache
- ```
-
- This block must run **before** `dpkg-reconfigure linux-image-...`
- triggers `dracut`, so that the generated initramfs captures both the
- correct `/etc/hostid` and the correct `zpool.cache`.
+ initramfs generation, the installer must ensure the target system's hostid
+ matches the hostid embedded in the pools, and that the zpool.cache reflects
+ this state. A minimal correct sequence is:
+
+ # Generate /etc/hostid in the target (or inherit the live's)
+ chroot /target zgenhostid -f
+
+ # Make the live use the same hostid as the target so subsequent
+ # zpool operations stamp pools with the correct value.
+ cp /target/etc/hostid /etc/hostid
+
+ # Re-write the on-disk pool configuration so the recorded hostid
+ # matches what the installed system will use at boot time.
+ zpool set cachefile=/etc/zfs/zpool.cache bpool
+ zpool set cachefile=/etc/zfs/zpool.cache rpool
+
+ # Propagate the resulting cachefile into the target so
+ # zfs-import-cache.service can succeed on first boot.
+ install -d /target/etc/zfs
+ cp /etc/zfs/zpool.cache /target/etc/zfs/zpool.cache
+
+ This block must run BEFORE "dpkg-reconfigure linux-image-..." triggers
+ dracut, so that the generated initramfs captures both the correct
+ /etc/hostid and the correct zpool.cache.
A more robust variant exports the pools and re-imports them under the
- target's hostid before the initramfs is built, but the simpler form
- above is sufficient because `zpool set cachefile` rewrites the cached
+ target's hostid before the initramfs is built, but the simpler form above
+ is sufficient because "zpool set cachefile" rewrites the cached
configuration with the active hostid.
- The natural place for this logic is the same curtin step that already
- runs `zpool set cachefile=...` (visible in the syslog at 09:41:50 and
- 09:41:52), or alternatively a new postinst hook executed before
- `50_zfs_cache`.
-
- ## Workaround for affected users
-
- ### Recover the current installation
+ The natural place for this logic is the same curtin step that already runs
+ "zpool set cachefile=..." (visible in the syslog at 09:41:50 and 09:41:52),
+ or alternatively a new postinst hook executed before 50_zfs_cache.
+
+
+ WORKAROUND FOR AFFECTED USERS
+ -----------------------------
+
+ Recover the current installation
+ . . . . . . . . . . . . . . . . .
From the dracut emergency shell, after the failed boot:
- ```sh
- zpool import -f -N rpool
- zpool import -f -N bpool
- ```
-
- Then press **Ctrl+D** to exit the emergency shell. Boot will resume;
- plymouth will prompt for the keystore passphrase, the system will
- decrypt and continue to the desktop.
-
- ### Make the fix permanent
-
- Once the pools have been imported with `-f` from the emergency shell at
- least once, the on-disk hostid recorded in each pool is rewritten to
- match the installed system's hostid as a side effect of the forced
- import. Subsequent boots no longer drop to emergency mode and no further
- action is required from the user - the system self-heals after the first
- manual recovery.
-
- ## Additional notes
+ zpool import -f -N rpool
+ zpool import -f -N bpool
+
+ Then press Ctrl+D to exit the emergency shell. Boot will resume; plymouth
+ will prompt for the keystore passphrase, the system will decrypt and
+ continue to the desktop.
+
+ Make the fix permanent
+ . . . . . . . . . . . .
+
+ Once the pools have been imported with -f from the emergency shell at least
+ once, the on-disk hostid recorded in each pool is rewritten to match the
+ installed system's hostid as a side effect of the forced import. Subsequent
+ boots no longer drop to emergency mode and no further action is required
+ from the user - the system self-heals after the first manual recovery.
+
+
+ ADDITIONAL NOTES
+ ----------------
- The dracut keystore integration provided by zfs-dracut 2.4.1-1ubuntu5
- works correctly. The keystore zvol, the LUKS layer, the raw key, the
- plymouth prompt, and zfs load-key all function as designed once the
- pools are imported. This bug is strictly about pool import, not about
- encryption.
-
- - `syslog`: full installer session log attached, with `grep -i hostid` returning zero matches.
- - **Note:** The attached syslog has been sanitized to redact private information (WiFi SSIDs, BSSIDs, local IPs, machine vendor/model, disk serial numbers, NetworkManager connection UUIDs). Replacements use clearly-marked tokens like `REDACTED_TARGET_SERIAL`. The structure, timestamps, line numbers, and all technical content relevant to the bug are intact. The string `grep -i hostid syslog_sanitized.txt` still returns zero hits, demonstrating the absence of any hostid-related operation during install.
+ works correctly. The keystore zvol, the LUKS layer, the raw key, the
+ plymouth prompt, and "zfs load-key" all function as designed once the
+ pools are imported. This bug is strictly about pool import, not about
+ encryption.
+
+ - syslog: full installer session log attached, with "grep -i hostid"
+ returning zero matches.
+
+ - Note: The attached syslog has been thoroughly sanitized over three
+ iterative review passes (75 redaction patterns verified with zero
+ residuals) before public submission. Replacements use clearly-marked
+ placeholders (REDACTED_USERNAME, [REDACTED-CPU], REDACTED_TARGET_SERIAL,
+ etc.) so structure and timestamps remain intact. Categories redacted
+ include: identity (username/hostname were exposed letter-by-letter via
+ validate_username GETs - note this itself may warrant a separate privacy
+ bug report), geolocation, hardware fingerprinting (CPU, GPU, BIOS,
+ memory, screen resolution and physical size, chipsets for WiFi/BT/audio/
+ Ethernet, USB peripherals), MAC addresses, WiFi SSIDs and BSSIDs, local
+ IPs, NetworkManager UUIDs, disk vendor/model/serial across all probed
+ devices, snap revision numbers, ZFS dataset random suffixes, ephemeral
+ systemd scope UUIDs, swap and FAT32 volume serials, and Windows BCD
+ identifiers from co-existing OS installations. The 8184-line structure,
+ all timestamps, and all technical content relevant to the bug are
+ intact. To verify the bug's root cause from this log:
+ "grep -i 'hostid' syslog_sanitized_v3.txt" returns zero hits,
+ demonstrating that no /etc/hostid write occurred during installation.
** Description changed:
- ================================================================================
- ZFS_LUKS_KEYSTORE installation drops to emergency shell on first boot
- due to missing hostid seeding in /target
- ================================================================================
-
-
AFFECTED PACKAGE
----------------
subiquity (likely also curtin, depending on which component owns the ZFS
pool creation step)
-
VERSIONS OBSERVED
-----------------
- Ubuntu 26.04 (Resolute Resilient) - desktop ISO, daily build of 2026-04-25
- Kernel: 7.0.0-14-generic
- ZFS: 2.4.1-1ubuntu5 (libzfs7linux, zfs-dracut, zfsutils-linux)
- Storage layout: ZFS_LUKS_KEYSTORE (guided full-disk encryption)
- Target disk: USB-attached SSD (SanDisk Portable SSD, 2 TB),
- /dev/sda enumerated as usb-SanDisk_Portable_SSD_*
-
+ /dev/sda enumerated as usb-SanDisk_Portable_SSD_*
SEVERITY
--------
High - installed system is unbootable without manual intervention from the
dracut emergency shell. The user must know to run "zpool import -f" for both
pools, which is not obvious from the failure message and not covered in any
installer-facing documentation.
-
SUMMARY
-------
When installing Ubuntu 26.04 with guided storage option ZFS_LUKS_KEYSTORE
onto a USB-attached SSD, the freshly installed system fails to boot directly
on first boot. It drops into the dracut emergency shell with:
- dracut-pre-mount[958]: Cannot get properties of rpool: no such pool available.
- [FAILED] Failed to mount sysroot.mount - /sysroot.
+ dracut-pre-mount[958]: Cannot get properties of rpool: no such pool available.
+ [FAILED] Failed to mount sysroot.mount - /sysroot.
From the emergency shell, "zpool import" (without arguments) shows both pools
as ONLINE but flagged with ZFS-8000-EY:
- pool: bpool
- id: 11158846221490413465
- state: ONLINE
- status: The pool was last accessed by another system.
- action: The pool can be imported using its name or numeric identifier
- and the '-f' flag.
- see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
-
- pool: rpool
- id: 2362008108274967906
- state: ONLINE
- status: The pool was last accessed by another system.
- action: The pool can be imported using its name or numeric identifier
- and the '-f' flag.
- see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
+ pool: bpool
+ id: 11158846221490413465
+ state: ONLINE
+ status: The pool was last accessed by another system.
+ action: The pool can be imported using its name or numeric identifier
+ and the '-f' flag.
+ see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
+
+ pool: rpool
+ id: 2362008108274967906
+ state: ONLINE
+ status: The pool was last accessed by another system.
+ action: The pool can be imported using its name or numeric identifier
+ and the '-f' flag.
+ see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
After running "zpool import -f -N rpool && zpool import -f -N bpool" from the
emergency shell and pressing Ctrl+D, boot resumes correctly: the dracut
keystore module prompts for the passphrase via plymouth, opens the LUKS
keystore zvol, the raw ZFS key is loaded, and the system reaches the
desktop normally.
As a side effect of the forced import, ZFS rewrites the hostid stamped into
the pools to match the installed system's hostid. Subsequent boots therefore
succeed without intervention - the system self-heals after the first manual
recovery. The boot pipeline is intact; only the *initial* hostid handshake
between live ISO and installed target is broken.
-
ROOT CAUSE
----------
The installer creates both pools (bpool, rpool) inside the live ISO
environment using whichever hostid the live happens to have. It does NOT
seed /target/etc/hostid to match, nor does it propagate a coherent
zpool.cache into the target before the initramfs is built. As a result, on
first boot the installed system has a hostid that does not match the one
embedded in the pools' on-disk configuration, and ZFS refuses to import them
without -f.
-
EVIDENCE FROM THE INSTALLER SYSLOG
----------------------------------
Relevant timeline from /var/log/syslog of the installer session (full log
attached):
- 09:41:50 zpool create -o ashift=12 -o autotrim=on
- -o feature at async_destroy=enabled ... -O canmount=off
- -O mountpoint=/boot -R /target -d bpool
- /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part2
-
- 09:41:50 zpool set cachefile=/etc/zfs/zpool.cache bpool
- (writes cachefile inside the live ISO, NOT /target)
-
- 09:41:52 zpool create -o ashift=12 -O encryption=on
- -O keylocation=file:///tmp/tmpl569ywhx -O keyformat=raw
- -O mountpoint=/ -R /target rpool
- /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part4
-
- 09:41:52 zpool set cachefile=/etc/zfs/zpool.cache rpool
-
- 09:41:52 cryptsetup luksFormat --offset 32768
- /dev/zvol/rpool/keystore /tmp/zpool-key-m4oyo6or
-
- 09:42:00 cryptsetup open --type luks /dev/zvol/rpool/keystore
- keystore-rpool --key-file /tmp/zpool-key-m4oyo6or
-
- 09:44:25 apt-get install of cryptsetup, zfs-dracut, zfsutils-linux,
- grub-efi-amd64-signed, shim-signed into /target
-
- 09:44:45 dracut: Generating /boot/initrd.img-7.0.0-14-generic
- (inside chroot)
-
- 09:45:00 dpkg-reconfigure grub-efi-amd64, update-grub
-
- 09:45:37 Postinst hook 50_zfs_cache runs, syncs zfs-list.cache only
+ 09:41:50 zpool create -o ashift=12 -o autotrim=on
+ -o feature at async_destroy=enabled ... -O canmount=off
+ -O mountpoint=/boot -R /target -d bpool
+ /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part2
+
+ 09:41:50 zpool set cachefile=/etc/zfs/zpool.cache bpool
+ (writes cachefile inside the live ISO, NOT /target)
+
+ 09:41:52 zpool create -o ashift=12 -O encryption=on
+ -O keylocation=file:///tmp/tmpl569ywhx -O keyformat=raw
+ -O mountpoint=/ -R /target rpool
+ /dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part4
+
+ 09:41:52 zpool set cachefile=/etc/zfs/zpool.cache rpool
+
+ 09:41:52 cryptsetup luksFormat --offset 32768
+ /dev/zvol/rpool/keystore /tmp/zpool-key-m4oyo6or
+
+ 09:42:00 cryptsetup open --type luks /dev/zvol/rpool/keystore
+ keystore-rpool --key-file /tmp/zpool-key-m4oyo6or
+
+ 09:44:25 apt-get install of cryptsetup, zfs-dracut, zfsutils-linux,
+ grub-efi-amd64-signed, shim-signed into /target
+
+ 09:44:45 dracut: Generating /boot/initrd.img-7.0.0-14-generic
+ (inside chroot)
+
+ 09:45:00 dpkg-reconfigure grub-efi-amd64, update-grub
+
+ 09:45:37 Postinst hook 50_zfs_cache runs, syncs zfs-list.cache only
A targeted search across the entire installer syslog returns ZERO hits for
any of the following:
- $ grep -iE 'hostid|zgenhostid|/etc/hostid' syslog
- (no output)
+ $ grep -iE 'hostid|zgenhostid|/etc/hostid' syslog
+ (no output)
Specifically, none of these expected operations occur during
installation:
- - No zgenhostid is ever invoked.
- - /etc/hostid is never copied from the live ISO to /target/etc/hostid.
- - No "chroot /target zgenhostid" is run before
- "dpkg-reconfigure linux-image-..." triggers the dracut initramfs build.
- - The zpool.cache file generated by "zpool set cachefile=..." lives only
- in the live ISO's /etc/zfs/ and is never transferred (in a coherent
- state) into /target/etc/zfs/.
+ - No zgenhostid is ever invoked.
+ - /etc/hostid is never copied from the live ISO to /target/etc/hostid.
+ - No "chroot /target zgenhostid" is run before
+ "dpkg-reconfigure linux-image-..." triggers the dracut initramfs build.
+ - The zpool.cache file generated by "zpool set cachefile=..." lives only
+ in the live ISO's /etc/zfs/ and is never transferred (in a coherent
+ state) into /target/etc/zfs/.
The first sign of trouble on boot is therefore:
- zfs-import-cache.service - Import ZFS pools by cache file skipped,
- unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
+ zfs-import-cache.service - Import ZFS pools by cache file skipped,
+ unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
zfs-import-scan.service then finds the pools but refuses to import them
under the running system's hostid, which does not match the one stamped
into the pools at creation time inside the live.
-
STEPS TO REPRODUCE
------------------
1. Boot the Ubuntu 26.04 desktop ISO from a USB stick.
2. In the installer, choose "Erase disk and install Ubuntu" with
- "Advanced features -> Use ZFS and encryption" (ZFS_LUKS_KEYSTORE
- capability).
+ "Advanced features -> Use ZFS and encryption" (ZFS_LUKS_KEYSTORE
+ capability).
3. Select a USB-attached SSD as the install target. (Note: USB targets
- reproduce this reliably; the same logic likely affects internal disks
- under any condition that produces a hostid mismatch between live and
- installed system, e.g. live ISO sessions where /etc/hostid is
- regenerated per boot.)
+ reproduce this reliably; the same logic likely affects internal disks
+ under any condition that produces a hostid mismatch between live and
+ installed system, e.g. live ISO sessions where /etc/hostid is
+ regenerated per boot.)
4. Set a passphrase for the keystore and complete the installation.
5. Reboot, removing the install media.
Expected: System boots through GRUB, prompts for the keystore passphrase
via plymouth, reaches the GDM login screen.
Actual: GRUB loads the kernel and initramfs; dracut imports neither pool;
the system drops to the emergency shell. No plymouth keystore prompt is
ever displayed because dracut bails out before reaching that stage.
-
PROPOSED FIX
------------
In the curtin/subiquity ZFS install path, between pool creation and
initramfs generation, the installer must ensure the target system's hostid
matches the hostid embedded in the pools, and that the zpool.cache reflects
this state. A minimal correct sequence is:
- # Generate /etc/hostid in the target (or inherit the live's)
- chroot /target zgenhostid -f
-
- # Make the live use the same hostid as the target so subsequent
- # zpool operations stamp pools with the correct value.
- cp /target/etc/hostid /etc/hostid
-
- # Re-write the on-disk pool configuration so the recorded hostid
- # matches what the installed system will use at boot time.
- zpool set cachefile=/etc/zfs/zpool.cache bpool
- zpool set cachefile=/etc/zfs/zpool.cache rpool
-
- # Propagate the resulting cachefile into the target so
- # zfs-import-cache.service can succeed on first boot.
- install -d /target/etc/zfs
- cp /etc/zfs/zpool.cache /target/etc/zfs/zpool.cache
+ # Generate /etc/hostid in the target (or inherit the live's)
+ chroot /target zgenhostid -f
+
+ # Make the live use the same hostid as the target so subsequent
+ # zpool operations stamp pools with the correct value.
+ cp /target/etc/hostid /etc/hostid
+
+ # Re-write the on-disk pool configuration so the recorded hostid
+ # matches what the installed system will use at boot time.
+ zpool set cachefile=/etc/zfs/zpool.cache bpool
+ zpool set cachefile=/etc/zfs/zpool.cache rpool
+
+ # Propagate the resulting cachefile into the target so
+ # zfs-import-cache.service can succeed on first boot.
+ install -d /target/etc/zfs
+ cp /etc/zfs/zpool.cache /target/etc/zfs/zpool.cache
This block must run BEFORE "dpkg-reconfigure linux-image-..." triggers
dracut, so that the generated initramfs captures both the correct
/etc/hostid and the correct zpool.cache.
A more robust variant exports the pools and re-imports them under the
target's hostid before the initramfs is built, but the simpler form above
is sufficient because "zpool set cachefile" rewrites the cached
configuration with the active hostid.
The natural place for this logic is the same curtin step that already runs
"zpool set cachefile=..." (visible in the syslog at 09:41:50 and 09:41:52),
or alternatively a new postinst hook executed before 50_zfs_cache.
-
WORKAROUND FOR AFFECTED USERS
-----------------------------
Recover the current installation
. . . . . . . . . . . . . . . . .
From the dracut emergency shell, after the failed boot:
- zpool import -f -N rpool
- zpool import -f -N bpool
+ zpool import -f -N rpool
+ zpool import -f -N bpool
Then press Ctrl+D to exit the emergency shell. Boot will resume; plymouth
will prompt for the keystore passphrase, the system will decrypt and
continue to the desktop.
Make the fix permanent
. . . . . . . . . . . .
Once the pools have been imported with -f from the emergency shell at least
once, the on-disk hostid recorded in each pool is rewritten to match the
installed system's hostid as a side effect of the forced import. Subsequent
boots no longer drop to emergency mode and no further action is required
from the user - the system self-heals after the first manual recovery.
-
ADDITIONAL NOTES
----------------
- The dracut keystore integration provided by zfs-dracut 2.4.1-1ubuntu5
- works correctly. The keystore zvol, the LUKS layer, the raw key, the
- plymouth prompt, and "zfs load-key" all function as designed once the
- pools are imported. This bug is strictly about pool import, not about
- encryption.
+ works correctly. The keystore zvol, the LUKS layer, the raw key, the
+ plymouth prompt, and "zfs load-key" all function as designed once the
+ pools are imported. This bug is strictly about pool import, not about
+ encryption.
- syslog: full installer session log attached, with "grep -i hostid"
- returning zero matches.
+ returning zero matches.
- Note: The attached syslog has been thoroughly sanitized over three
- iterative review passes (75 redaction patterns verified with zero
- residuals) before public submission. Replacements use clearly-marked
- placeholders (REDACTED_USERNAME, [REDACTED-CPU], REDACTED_TARGET_SERIAL,
- etc.) so structure and timestamps remain intact. Categories redacted
- include: identity (username/hostname were exposed letter-by-letter via
- validate_username GETs - note this itself may warrant a separate privacy
- bug report), geolocation, hardware fingerprinting (CPU, GPU, BIOS,
- memory, screen resolution and physical size, chipsets for WiFi/BT/audio/
- Ethernet, USB peripherals), MAC addresses, WiFi SSIDs and BSSIDs, local
- IPs, NetworkManager UUIDs, disk vendor/model/serial across all probed
- devices, snap revision numbers, ZFS dataset random suffixes, ephemeral
- systemd scope UUIDs, swap and FAT32 volume serials, and Windows BCD
- identifiers from co-existing OS installations. The 8184-line structure,
- all timestamps, and all technical content relevant to the bug are
- intact. To verify the bug's root cause from this log:
- "grep -i 'hostid' syslog_sanitized_v3.txt" returns zero hits,
- demonstrating that no /etc/hostid write occurred during installation.
+ iterative review passes (75 redaction patterns verified with zero
+ residuals) before public submission. Replacements use clearly-marked
+ placeholders (REDACTED_USERNAME, [REDACTED-CPU], REDACTED_TARGET_SERIAL,
+ etc.) so structure and timestamps remain intact. Categories redacted
+ include: identity (username/hostname were exposed letter-by-letter via
+ validate_username GETs - note this itself may warrant a separate privacy
+ bug report), geolocation, hardware fingerprinting (CPU, GPU, BIOS,
+ memory, screen resolution and physical size, chipsets for WiFi/BT/audio/
+ Ethernet, USB peripherals), MAC addresses, WiFi SSIDs and BSSIDs, local
+ IPs, NetworkManager UUIDs, disk vendor/model/serial across all probed
+ devices, snap revision numbers, ZFS dataset random suffixes, ephemeral
+ systemd scope UUIDs, swap and FAT32 volume serials, and Windows BCD
+ identifiers from co-existing OS installations. The 8184-line structure,
+ all timestamps, and all technical content relevant to the bug are
+ intact. To verify the bug's root cause from this log:
+ "grep -i 'hostid' syslog_sanitized_v3.txt" returns zero hits,
+ demonstrating that no /etc/hostid write occurred during installation.
** Package changed: ubiquity (Ubuntu) => subiquity (Ubuntu)
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubiquity in Ubuntu.
https://bugs.launchpad.net/bugs/2150354
Title:
ZFS_LUKS_KEYSTORE installation drops to emergency shell on first boot
due to missing hostid seeding in /target
Status in subiquity package in Ubuntu:
New
Bug description:
AFFECTED PACKAGE
----------------
subiquity (likely also curtin, depending on which component owns the ZFS
pool creation step)
VERSIONS OBSERVED
-----------------
- Ubuntu 26.04 (Resolute Resilient) - desktop ISO, daily build of 2026-04-25
- Kernel: 7.0.0-14-generic
- ZFS: 2.4.1-1ubuntu5 (libzfs7linux, zfs-dracut, zfsutils-linux)
- Storage layout: ZFS_LUKS_KEYSTORE (guided full-disk encryption)
- Target disk: USB-attached SSD (SanDisk Portable SSD, 2 TB),
/dev/sda enumerated as usb-SanDisk_Portable_SSD_*
SEVERITY
--------
High - installed system is unbootable without manual intervention from the
dracut emergency shell. The user must know to run "zpool import -f" for both
pools, which is not obvious from the failure message and not covered in any
installer-facing documentation.
SUMMARY
-------
When installing Ubuntu 26.04 with guided storage option ZFS_LUKS_KEYSTORE
onto a USB-attached SSD, the freshly installed system fails to boot directly
on first boot. It drops into the dracut emergency shell with:
dracut-pre-mount[958]: Cannot get properties of rpool: no such pool available.
[FAILED] Failed to mount sysroot.mount - /sysroot.
From the emergency shell, "zpool import" (without arguments) shows both pools
as ONLINE but flagged with ZFS-8000-EY:
pool: bpool
id: 11158846221490413465
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier
and the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
pool: rpool
id: 2362008108274967906
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier
and the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
After running "zpool import -f -N rpool && zpool import -f -N bpool" from the
emergency shell and pressing Ctrl+D, boot resumes correctly: the dracut
keystore module prompts for the passphrase via plymouth, opens the LUKS
keystore zvol, the raw ZFS key is loaded, and the system reaches the
desktop normally.
As a side effect of the forced import, ZFS rewrites the hostid stamped into
the pools to match the installed system's hostid. Subsequent boots therefore
succeed without intervention - the system self-heals after the first manual
recovery. The boot pipeline is intact; only the *initial* hostid handshake
between live ISO and installed target is broken.
ROOT CAUSE
----------
The installer creates both pools (bpool, rpool) inside the live ISO
environment using whichever hostid the live happens to have. It does NOT
seed /target/etc/hostid to match, nor does it propagate a coherent
zpool.cache into the target before the initramfs is built. As a result, on
first boot the installed system has a hostid that does not match the one
embedded in the pools' on-disk configuration, and ZFS refuses to import them
without -f.
EVIDENCE FROM THE INSTALLER SYSLOG
----------------------------------
Relevant timeline from /var/log/syslog of the installer session (full log
attached):
09:41:50 zpool create -o ashift=12 -o autotrim=on
-o feature at async_destroy=enabled ... -O canmount=off
-O mountpoint=/boot -R /target -d bpool
/dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part2
09:41:50 zpool set cachefile=/etc/zfs/zpool.cache bpool
(writes cachefile inside the live ISO, NOT /target)
09:41:52 zpool create -o ashift=12 -O encryption=on
-O keylocation=file:///tmp/tmpl569ywhx -O keyformat=raw
-O mountpoint=/ -R /target rpool
/dev/disk/by-id/usb-SanDisk_Portable_SSD_*-part4
09:41:52 zpool set cachefile=/etc/zfs/zpool.cache rpool
09:41:52 cryptsetup luksFormat --offset 32768
/dev/zvol/rpool/keystore /tmp/zpool-key-m4oyo6or
09:42:00 cryptsetup open --type luks /dev/zvol/rpool/keystore
keystore-rpool --key-file /tmp/zpool-key-m4oyo6or
09:44:25 apt-get install of cryptsetup, zfs-dracut, zfsutils-linux,
grub-efi-amd64-signed, shim-signed into /target
09:44:45 dracut: Generating /boot/initrd.img-7.0.0-14-generic
(inside chroot)
09:45:00 dpkg-reconfigure grub-efi-amd64, update-grub
09:45:37 Postinst hook 50_zfs_cache runs, syncs zfs-list.cache only
A targeted search across the entire installer syslog returns ZERO hits for
any of the following:
$ grep -iE 'hostid|zgenhostid|/etc/hostid' syslog
(no output)
Specifically, none of these expected operations occur during
installation:
- No zgenhostid is ever invoked.
- /etc/hostid is never copied from the live ISO to /target/etc/hostid.
- No "chroot /target zgenhostid" is run before
"dpkg-reconfigure linux-image-..." triggers the dracut initramfs build.
- The zpool.cache file generated by "zpool set cachefile=..." lives only
in the live ISO's /etc/zfs/ and is never transferred (in a coherent
state) into /target/etc/zfs/.
The first sign of trouble on boot is therefore:
zfs-import-cache.service - Import ZFS pools by cache file skipped,
unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
zfs-import-scan.service then finds the pools but refuses to import them
under the running system's hostid, which does not match the one stamped
into the pools at creation time inside the live.
STEPS TO REPRODUCE
------------------
1. Boot the Ubuntu 26.04 desktop ISO from a USB stick.
2. In the installer, choose "Erase disk and install Ubuntu" with
"Advanced features -> Use ZFS and encryption" (ZFS_LUKS_KEYSTORE
capability).
3. Select a USB-attached SSD as the install target. (Note: USB targets
reproduce this reliably; the same logic likely affects internal disks
under any condition that produces a hostid mismatch between live and
installed system, e.g. live ISO sessions where /etc/hostid is
regenerated per boot.)
4. Set a passphrase for the keystore and complete the installation.
5. Reboot, removing the install media.
Expected: System boots through GRUB, prompts for the keystore passphrase
via plymouth, reaches the GDM login screen.
Actual: GRUB loads the kernel and initramfs; dracut imports neither pool;
the system drops to the emergency shell. No plymouth keystore prompt is
ever displayed because dracut bails out before reaching that stage.
PROPOSED FIX
------------
In the curtin/subiquity ZFS install path, between pool creation and
initramfs generation, the installer must ensure the target system's hostid
matches the hostid embedded in the pools, and that the zpool.cache reflects
this state. A minimal correct sequence is:
# Generate /etc/hostid in the target (or inherit the live's)
chroot /target zgenhostid -f
# Make the live use the same hostid as the target so subsequent
# zpool operations stamp pools with the correct value.
cp /target/etc/hostid /etc/hostid
# Re-write the on-disk pool configuration so the recorded hostid
# matches what the installed system will use at boot time.
zpool set cachefile=/etc/zfs/zpool.cache bpool
zpool set cachefile=/etc/zfs/zpool.cache rpool
# Propagate the resulting cachefile into the target so
# zfs-import-cache.service can succeed on first boot.
install -d /target/etc/zfs
cp /etc/zfs/zpool.cache /target/etc/zfs/zpool.cache
This block must run BEFORE "dpkg-reconfigure linux-image-..." triggers
dracut, so that the generated initramfs captures both the correct
/etc/hostid and the correct zpool.cache.
A more robust variant exports the pools and re-imports them under the
target's hostid before the initramfs is built, but the simpler form above
is sufficient because "zpool set cachefile" rewrites the cached
configuration with the active hostid.
The natural place for this logic is the same curtin step that already runs
"zpool set cachefile=..." (visible in the syslog at 09:41:50 and 09:41:52),
or alternatively a new postinst hook executed before 50_zfs_cache.
WORKAROUND FOR AFFECTED USERS
-----------------------------
Recover the current installation
. . . . . . . . . . . . . . . . .
From the dracut emergency shell, after the failed boot:
zpool import -f -N rpool
zpool import -f -N bpool
Then press Ctrl+D to exit the emergency shell. Boot will resume; plymouth
will prompt for the keystore passphrase, the system will decrypt and
continue to the desktop.
Make the fix permanent
. . . . . . . . . . . .
Once the pools have been imported with -f from the emergency shell at least
once, the on-disk hostid recorded in each pool is rewritten to match the
installed system's hostid as a side effect of the forced import. Subsequent
boots no longer drop to emergency mode and no further action is required
from the user - the system self-heals after the first manual recovery.
ADDITIONAL NOTES
----------------
- The dracut keystore integration provided by zfs-dracut 2.4.1-1ubuntu5
works correctly. The keystore zvol, the LUKS layer, the raw key, the
plymouth prompt, and "zfs load-key" all function as designed once the
pools are imported. This bug is strictly about pool import, not about
encryption.
- syslog: full installer session log attached, with "grep -i hostid"
returning zero matches.
- Note: The attached syslog has been thoroughly sanitized over three
iterative review passes (75 redaction patterns verified with zero
residuals) before public submission. Replacements use clearly-marked
placeholders (REDACTED_USERNAME, [REDACTED-CPU], REDACTED_TARGET_SERIAL,
etc.) so structure and timestamps remain intact. Categories redacted
include: identity (username/hostname were exposed letter-by-letter via
validate_username GETs - note this itself may warrant a separate privacy
bug report), geolocation, hardware fingerprinting (CPU, GPU, BIOS,
memory, screen resolution and physical size, chipsets for WiFi/BT/audio/
Ethernet, USB peripherals), MAC addresses, WiFi SSIDs and BSSIDs, local
IPs, NetworkManager UUIDs, disk vendor/model/serial across all probed
devices, snap revision numbers, ZFS dataset random suffixes, ephemeral
systemd scope UUIDs, swap and FAT32 volume serials, and Windows BCD
identifiers from co-existing OS installations. The 8184-line structure,
all timestamps, and all technical content relevant to the bug are
intact. To verify the bug's root cause from this log:
"grep -i 'hostid' syslog_sanitized_v3.txt" returns zero hits,
demonstrating that no /etc/hostid write occurred during installation.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/subiquity/+bug/2150354/+subscriptions
More information about the foundations-bugs
mailing list