[Bug 1732028] Re: timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests on LP Infra]

Thu Jul 19 16:03:11 UTC 2018

In a discussion with smoser we decided to try just clearing /etc/fstab - it might be sufficient for our test intention and at the same time avoid the issue.
With that done we still have the following failing with timeouts:
  [ TIME ] Timed out waiting for device dev-ttyS0.device.
  [FAILED] Failed to start Journal Service.

Not sure why the first happens, but the second is due to the lack of a
root FS.

In my slowed down environments it boots REALLY slow then, like 30 minutes.
Several services are unforgiving of the root not being there.
Especially snap that was getting into a loop of Starting/Stopping/... and by that did not let it complete.

The former try was too hard (intentionally), so I stepped back to just replace the labels with the (in this case) known partition IDs /dev/sda1 and /dev/sda15.
This I first booted non slowed to check if the config works at all.
It works

Note: non slowed the TCG-system is at 4*100% for the vcpus and 400% for
TCG. ~800% vs ~98% is a slowdown by 1/8 which matches what I see as
effective slowdown.

A few more tries together with Scott found that disabling the fstab entry for EFI, but keeping the main root as-is.
Again we verified this config is valid (boot without CPU limit) and then we set the limit.
With that finally it seems to work.
Lets code that up in open-iscsi and put it to a PPA to test on LP infra against it.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to open-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1732028

Title:
  timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests
  on LP Infra]

Status in open-iscsi package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  This issue keeps cropping up.  It shows itself in open-iscsi autopkg tests.
  I think it might just be "really slow system".  It seems the timeout is only
   1 minute 30 seconds for the disk to appear, and in a happy run you
  might see something very close:

  [K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for dev-disk…-UEFI.device (1min 29s / 1min 32s)
  [K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for dev-disk…-UEFI.device (1min 30s / 1min 32s)
  [K[[0;32m  OK  [0m] Found device VIRTUAL-DISK UEFI.
           Mounting /boot/efi...

  ---
  There is information on the open-iscsi tests at [1].
    [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/tests/README-boot-test.md

  The tests set up an iscsi target and boot a kvm guest off that read-
  only root with overlayroot.

  # get open-iscsi source
  $ apt-get source open-iscsi
  $ cd open-iscsi-2.0.874/

  $ sudo apt-get install -qy simplestreams tgt qemu-system-x86 \
       cloud-image-utils distro-info
  $ cd open-iscsi-2.0.874/

  ## we're now mostly following debian/tests/README-boot-test.md
  # download the image and get kernel/initrd
  $ PATH=$PWD/debian/tests:$PATH
  $ get-image bionic.d bionic
  $ sudo `which patch-image` \
      --kernel=bionic.d/kernel --initrd=bionic.d/initrd bionic.d/disk.img

  $ tgt-boot-test -v bionic.d/disk.img bionic.d/kernel bionic.d/initrd
  ....

  Success is being able to log in with 'ubuntu' and 'passw0rd'.
  Failure as seen in the log is dropping into an emergency shell.

  Once inside (this was successful) you'll see a mostly sane system.
  Some things to note:
  a.) tgt-boot-test boots without kvm enabled.  This is because using
   kvm with qemu in nested virt would cause system lockups. Its slower
  but more reliable to go wtihout.
  b.) under bug 1723183 I  made overlayroot comment out the root filesystem
  from the rendered /etc/fstab.  That was because systemd got confused and
  assumed that /media/root-ro had to be on top of /.
  c.) you can enable or disable kvm by setting _USE_KVM=0 or _USE_KVM=1
     in your environment.

  $ grep -v "^# " /etc/fstab
  #
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overl
  ay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
  LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported

  $ sudo blkid
  /dev/sda1: LABEL="cloudimg-rootfs" UUID="7b1980bd-9102-4356-8df0-ec7a0c062411" TYPE="ext4" PARTUUID="c0b5ace0-4703-4667-babb-3d38137cab88"
  /dev/sda15: LABEL="UEFI" UUID="B177-3CC9" TYPE="vfat" PARTUUID="0ab0b9fd-2c28-4724-857a-1559f0cf76ea"
  /dev/sda14: PARTUUID="221662d6-cab0-4290-ba1c-e72acf2bf193"

  $ cat /run/systemd/generator/local-fs.target.requires/boot-efi.mount
  # Automatically generated by systemd-fstab-generator

  [Unit]
  SourcePath=/etc/fstab
  Documentation=man:fstab(5) man:systemd-fstab-generator(8)
  Before=local-fs.target

  [Mount]
  Where=/boot/efi
  What=/dev/disk/by-label/UEFI
  Type=vfat

  Related bugs:
   * bug 1680197: Zesty deployments failing sporadically
   * bug 1723183: transient systemd ordering issue when using overlayroot
   * bug 1666573: transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root 

  ProblemType: Bug
  DistroRelease: Ubuntu 18.04
  Package: systemd 234-2ubuntu12
  ProcVersionSignature: User Name 4.13.0-16.19-generic 4.13.4
  Uname: Linux 4.13.0-16-generic x86_64
  ApportVersion: 2.20.7-0ubuntu4
  Architecture: amd64
  Date: Mon Nov 13 21:06:36 2017
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
  ProcEnviron:
   TERM=vt220
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=C.UTF-8
   SHELL=/bin/bash
  ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-7xuhwl iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-7xuhwl-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/
  SourcePackage: systemd
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 04/01/2014
  dmi.bios.vendor: SeaBIOS
  dmi.bios.version: 1.10.2-1ubuntu1
  dmi.chassis.type: 1
  dmi.chassis.vendor: QEMU
  dmi.chassis.version: pc-i440fx-artful
  dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-artful:cvnQEMU:ct1:cvrpc-i440fx-artful:
  dmi.product.name: Standard PC (i440FX + PIIX, 1996)
  dmi.product.version: pc-i440fx-artful
  dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1732028/+subscriptions