[Bug 2073550] Re: "update_bootloader" step fails due to partition table not loaded

[Zhaoxuan Zhai]

Hi Paul,

We got 8 out of last 10 image build jobs failing due to this issue.

Do you have any plans to fix this issue soon? Or is there any workaround
we can try?


---
https://github.com/canonical/genio-image/actions?query=event%3Aschedule

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to Ubuntu Image.
https://bugs.launchpad.net/bugs/2073550

Title:
  "update_bootloader" step fails due to partition table not loaded

Status in Ubuntu Image:
  New

Bug description:
  We have a github action to build images using ubuntu-image. But it
  sometimes fails in the "update_bootloader" step.

  The error messages is as follows:
  ...
  [22] update_bootloader
  mount: /home/ubuntu/actions-runner/_work/genio-image/genio-image/work/scratch/loopback: special device /dev/loop6p9 does not exist.
  chroot: failed to run command ‘dpkg-divert’: No such file or directory
  mount: /home/ubuntu/actions-runner/_work/genio-image/genio-image/work/scratch/loopback/sys: not mount point or bad option.
  ...

  This error doesn't occur very often, but we triggered it many times. You can check full log at here:
  https://github.com/canonical/genio-image/actions/runs/9937829959/job/27448991654
  https://github.com/canonical/genio-image/actions/runs/9999251024/job/27639899366
  https://github.com/canonical/genio-image/actions/runs/9869259421/job/27252675072

  
  [Some Investigation]

  In the source code of the "update_bootloader" step, I found that
  ubuntu-image tries to mount the rootfs partition just after `losetup`.

  Then, I write a bash script `partscan.sh`, and wrap this script into
  an loop and run it on a arm64 virtual machine:

     while sudo ./partscan.sh ; do :;done

  After 20 minutes, I got a loopback device without partition devices
  created:

  ...
  ++ losetup --find --show --partscan --sector-size 4096 test.img                                                                                                                                                        
  + dev=/dev/loop4                                                                                                                                                                                                       
  + lsblk /dev/loop4                                                                                                                                                                                                     
  NAME  MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS                                                                                                                                                                              
  loop4   7:4    0  32M  0 loop                                                                                                                                                                                          
  + ls /dev/loop4                                                                                                                                                                                                        
  /dev/loop4                                                                                                                                                                                                             
  + '[' '!' -e /dev/loop4p1 ']'                                                                                                                                                                                          
  + echo 'It happends'                                                                                                                                                                                                   
  It happends                                                                                                                                                                                                            
  ...

  Then I tried `partprobe`, then the partition device has been created:
  ...
  ubuntu at ubuntu:~$ lsblk
  NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
  loop0     7:0    0 59.8M  1 loop /snap/core20/2321
  loop1     7:1    0 77.4M  1 loop /snap/lxd/28384
  loop2     7:2    0 33.7M  1 loop /snap/snapd/21761
  loop3     7:3    0 77.4M  1 loop /snap/lxd/29353
  loop4     7:4    0   32M  0 loop 
  vda     252:0    0  2.2G  0 disk 
  ├─vda1  252:1    0  2.1G  0 part /
  └─vda15 252:15   0   99M  0 part /boot/efi
  ubuntu at ubuntu:~$ sudo partprobe /dev/loop4
  ubuntu at ubuntu:~$ lsblk
  NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
  loop0       7:0    0 59.8M  1 loop /snap/core20/2321
  loop1       7:1    0 77.4M  1 loop /snap/lxd/28384
  loop2       7:2    0 33.7M  1 loop /snap/snapd/21761
  loop3       7:3    0 77.4M  1 loop /snap/lxd/29353
  loop4       7:4    0   32M  0 loop 
  └─loop4p1 259:0    0   31M  0 part 
  vda       252:0    0  2.2G  0 disk 
  ├─vda1    252:1    0  2.1G  0 part /
  └─vda15   252:15   0   99M  0 part /boot/efi
  ubuntu at ubuntu:~$ 
  ...

  So it means that the option "--partscan" in losetup cannot guarantee
  that the partition table is loaded. An additional "partprobe" is
  required.

  If ubuntu-image just calls losetup, then it may not be able to mount
  the rootfs. Because losetup may not create the partition device for
  ubuntu-image.

  
  BTW, I can only reproduce the losetup issue on an arm64 virtual machine. I have also tried the partscan.sh script on my amd64 laptop, but the issue of missing partition device did not occur.

  The way I create the arm64 virtual machine is followed here:
  https://ubuntu.com/server/docs/boot-arm64-virtual-machines-on-qemu

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-image/+bug/2073550/+subscriptions