[Bug 1489855] Re: Change to mount sequence order breaks persistence on casper-rw partitions

Fri Jan 24 14:23:49 UTC 2020

Since the patch didn't make it in time for 18.04.3 but 18.04 is the
latest LTS release available before 20.04, I'd wager there'd be at least
a few others looking to create a persistent LTS USB drive facing this
issue. Also, I spent a few hours figuring this out so hopefully this
saves someone some time.

I've made a patched version of the "casper/initrd" file that one can
simply replace with in an already-created USB drive. Its SHA1 is
d70c6189ae6497422b704219c3f8dd1a8fba6fe6.

*Only works from bootable drives made from Ubuntu 18.04.3 ISO!*
https://drive.google.com/file/d/1X25ZafWhUn9ZpDNS1QHWmCaDS7Fl9sOT

If you do not trust a random binary, here are the steps required to
reproduce it:

1. Extract "casper/initrd" from the ISO or USB drive to a temporary directory
(You can use unmkinitramfs but let's extract manually.)
(Based on mwhudson's incantation a few comments up, but for LZMA used in 18.04.)

$ cd /path/to/temp/dir; (cpio -t; cpio -t; lzcat | cpio -i ) <
/path/to/casper/initrd

2. (Optional) Apply 0e52b46 so that the actual patch won't be offset
(This patch is not relevant for our case but precedes the patch we want to apply. It does fix a bug though!)
(Ignore the error about not finding "debian/changelog", it is only included in the casper package and not the initrd.)

$ wget -qO -
https://git.launchpad.net/~mwhudson/ubuntu/+source/casper/patch/?id=0e52b46ea2a0eff48e0e8b26cb6a7174989ceb27
| patch -tp1

3. Apply a452b0b, which is what we want
(Again, ignore the error about not finding "debian/changelog", it is only included in the casper package and not the initrd.)

$ wget -qO -
https://git.launchpad.net/~mwhudson/ubuntu/+source/casper/patch/?id=a452b0b0d5874ae48ee7ea742e500e48ffd8d0a4
| patch -tp1

4. Repackage into LZMA file
(Again based on mwhudson's incantation a few comments up, though gzip should probably work fine too.)

$ find . | LC_ALL=C sort | cpio -R 0:0 -o -H newc | lzma -c >
/tmp/initrd.new.lz

[Here, you should be able to use the new initrd for almost all cases by
replacing the one on your USB drive. However, the original initrd has
microcode embedded before the actual initrd, so it will be nice if we
can restore that.]

5. Here it gets tricky as we try to extract the microcode segment
(If you have binwalk installed, here's a command to automatically do this for you)

$ TGT_FILE=/path/to/casper/initrd; dd if=$TGT_FILE
of=/tmp/initrd.microcode bs=`binwalk $TGT_FILE | grep LZMA | head -1 |
cut -d ' ' -f1` count=1

(Else, you'll have to trust me when I tell you the segment spans from
0-2441216 for the official 18.04.3 ISO)

$ dd if=/path/to/casper/initrd of=/tmp/initrd.microcode bs=2441216
count=1

6. Put the files together to get the initrd that's identical in
structure to the original

$ cat /tmp/initrd.microcode /tmp/initrd.new.lz > /tmp/initrd

[You should now be able to replace the initrd in the casper directory on
your USB drive with the newly-built one in /tmp]

HTH

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to casper in Ubuntu.
https://bugs.launchpad.net/bugs/1489855

Title:
  Change to mount sequence order breaks persistence on casper-rw
  partitions

Status in casper package in Ubuntu:
  Fix Released

Bug description:
  the system boots fine when using a casper-rw FILE but drops to a
  busybox when using a partition

  the short log would be:
  ___________________________________________________
  Begin: Running /scripts/casper-premount ... done.
  done.
  umount: can't umount /cdrom: Device or resource busy
  Warning: Unable to find the persistent home medium
  umount: can't umount /cdrom: Device or resource busy
  Warning: Impossible to include the casper-sn Snapshot
  umount: can't umount /cdrom: Device or resource busy
  Warning: Impossible to include the home-sn Snapshot
  done.
  ___________________________________________________

  removing the "persistence" keyword from the syslinux.cfg works and the live usb drive boots just fine.
  using a persistence file instead of the partition and the usb drive boots just fine.

  i found older bug reports concerning the same problem but the fix
  proposed in 2010 is already integrated into the script "casper".

  so i started the flashdrive with the casper debug= option and i see
  the following (after probing several other devices it finally finds
  the right device and partition (sdb3)

  ___________________________________________________

  + cow_backing_mp=/cdrom
  + [ -e /cdrom/casper-rw ]
  + umount /cdrom
  + sys2dev /sys/block/sdb/sdb3
  + sysdev=/block/sdb/sdb3
  + udevadm info -q name -p /block/sdb/sdb3
  + echo /dev/sdb3
  + devname=/dev/sdb3
  + /sbin/blkid -s LABEL -o value /dev/sdb3
  + [ casper-rw = casper-rw ]
  + echo /dev/sdb3
  + return
  + cowprobe=/dev/sdb3
  + [ -b /dev/sdb3 ]
  + cowdevice=/dev/sdb3
  + get_fstype /dev/sdb3
  + local FSTYPE
  + local FSSIZE
  + fstype
  + eval FSTYPE=ext4 FSSIZE=8458862592
  + FSTYPE=ext4 FSSIZE=8458862592
  + [ ext4 != unknown ]
  + echo ext4
  + return 0
  + cow_fstype=ext4
  + cow_mountopt=rw,noatime
  + mount -t ext4 -o rw,noatime /dev/sdb3 /cow
  + [ ! -d /cow/upper ]
  + mkdir -p /cow/upper
  + continue
  + continue
  + mkdir -p /cow/work
  + [ -f /cow/format ]
  + [ DEFAULT = DEFAULT ]
  + modprobe -q -b overlay
  + grep -q ^overlay$
  + cut -f2 /proc/filesystems
  + UNIONFS=overlay
  + break
  ___________________________________________________

  this looks fine to me.. it looks like it recognizes everything .. it's
  ext4 .. label casper-rw.. it's mounting it...

  a little bit further down in the loooong log file it states the
  following:

  ______________________________________________

  + cow_backing_mp=/home-rw-backing
  + [ -e /home-rw-backing/home-rw ]
  + umount /home-rw-backing
  + sys2dev /sys/block/sdb/sdb3
  + sysdev=/block/sdb/sdb3
  + udevadm info -q name -p /block/sdb/sdb3
  + echo /dev/sdb3
  + devname=/dev/sdb3
  + /sbin/blkid -s LABEL -o value /dev/sdb3
  + [ casper-rw = home-rw ]
  + get_fstype /dev/sdb3
  + local FSTYPE
  + local FSSIZE
  + fstype
  + eval FSTYPE=ext4 FSSIZE=8458862592
  + FSTYPE=ext4 FSSIZE=8458862592
  + [ ext4 != unknown ]
  + echo ext4
  + return 0
  + [ ext4 = vfat ]
  + homecow=
  + [ -b ]
  + [ n != y ]
  + log_warning_msg Unable to find the persistent home medium
  + _log_msg Warning: Unable to find the persistent home medium\n
  + [ n = y ]
  + printf Warning: Unable to find the persistent home medium\n
  Warning: Unable to find the persistent home medium
  __________________________________________________________

  a warning about the home medium is shown in both cases (file and partition) but the persistence file works never the less..

  my system:  

  kubuntu linux 15.10 beta1

  uname -a
  Linux wald 4.1.0-3-generic #3-Ubuntu SMP Tue Jul 28 12:25:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1489855/+subscriptions