[Bug 2147326] Re: Cannot deploy VM hosts in 3.6.4 candidate

[Jacopo Rota]

** Also affects: curtin
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to curtin.
https://bugs.launchpad.net/bugs/2147326

Title:
  Cannot deploy VM hosts in 3.6.4 candidate

Status in curtin:
  New
Status in MAAS:
  New

Bug description:
  Describe the bug: 
  In our testing pipeline we designate one node to be LXD VM host and one to be KVM VM host.
  In current candidate it never went pass the deploying these two nodes

  
  Steps to reproduce: 

  Deploy KVM or LXD VM host


  Expected behavior (what should have happened?):
  Nodes deployed 

  
  Actual behavior (what actually happened?): 
  Nodes don't deploy within 30 minues


  MAAS version and installation type (deb, snap): 
  3.6.4, same failure in snap and deb


  MAAS setup (HA, single node, multiple regions/racks):
  HA


  Host OS distro and version:
  noble


  Additional context:
  Nothing special, regular release testing runs.

  Here is what AI gave me, take with a grain of salt
  This is from virtual maas environment, I see the same failures in baremetal as well

  # Workflow Run Analysis Report

  ## Run: 23910974753
  - **Repository:** canonical/sqa-cloud-deployment-pipeline
  - **Branch:** main
  - **Status:** ❌ Failed
  - **Duration:** ~35 minutes
  - **Solutions Run UUID:** f8251919-522e-47a0-abc8-3c36a045f8b5

  ---

  ## Failed Step: maas

  ### Executed Steps (relevant excerpt)
  - setup ✅
  - maas ❌ **← FAILED HERE**
  - (teardown / reporting ran post-failure)

  ---

  ## Failure Analysis

  ### Error
  ```
  Exception: Not all KVM hosts were deployed in time.

  FCE poll (every ~30s for 30 minutes):
    {'node5': 'Deploying', 'node6': 'Deploying'}
    …
    {'node5': 'Deploying', 'node6': 'Deploying'}   ← repeated ~60 times
  ```

  Curtin error on node5 (system ID `q686c3`, node6 equivalent):
  ```
  curtin.util.ProcessExecutionError: Unexpected error while running command.
  Command: ['unshare', '--fork', '--pid', '--', 'chroot',
            '/tmp/tmpyh1a9xjs/target', 'apt-get', '--quiet', '--assume-yes',
            '--option=Dpkg::options::=--force-unsafe-io',
            '--option=Dpkg::Options::=--force-confold',
            'install', '--download-only', 'linux-generic']
  Exit code: 100
  Stdout: ''
  Stderr: ''

  Preceded by:
    E: Unable to locate package linux-generic
  ```

  Stack trace:
  ```
  File "/curtin/curtin/commands/curthooks.py", line 1947, in curthooks
      builtin_curthooks(cfg, target, state)
  File "/curtin/curtin/commands/curthooks.py", line 1779, in builtin_curthooks
      install_kernel(cfg, target)
  File "/curtin/curtin/commands/curthooks.py", line 387, in install_kernel
      install(kernel_package)
  ```

  ### Environment Context
  - **Substrate:** tor3-sqa-virtual_maas
  - **Cluster:** cluster_6
  - **Environment:** tor3-sqa-virtual_maas-cluster_6
  - **MAAS version:** 3.6.4-17623-g.46a516275 (apt-installed, not snap)
  - **Curtin version:** 23.1.1-1124-g7324b43b-0ubuntu1~ubuntu24.04.1
  - **Ephemeral image:** ubuntu/amd64/ga-24.04/noble/20260223/squashfs (image hash `5f64c83`)
  - **Node5 IP:** 10.241.144.81 (system ID q686c3, `install_kvm=True`)
  - **Node6 IP:** 10.241.144.211 (system ID spn4d3, `register_vmhost=True`)
  - **Apt proxy:** http://10.241.144.3:8000/ (Squid on infra2)
  - **Storage layout:** bcache — sda (root), sdb (20 GB cache), sdc (bcache backing)

  ---

  ## Root Cause

  **Summary:** Curtin's `install_kernel` curthook failed because `apt-get update` made zero
  network requests, leaving the package database empty; consequently `apt-get install
  linux-generic` failed with "E: Unable to locate package linux-generic". Curtin never
  reached `late_commands`, so the `netboot_off` signal was never sent to MAAS, and both
  nodes continuously PXE-booted for the full 30-minute FCE timeout.

  **Detailed Explanation:**

  1. **Curtin ran successfully through all early stages**: partitioning (bcache layout with
     sda/sdb/sdc), network configuration, OS extraction from squashfs (23s), and early
     curthooks (apt config writing, missing-packages, iscsi, mdadm — all SUCCESS).

  2. **`install_kernel` curthook was reached at ~17:44:57** and ran in its entirety in
     ~246 milliseconds — far too fast for any network I/O. The sequence was:
     - `apt-get update` with `--option=Dir::Etc::sourcelist=/tmp/tmpsyktkyxm/sources.list`
       → "Reading package lists..." → returned immediately
     - `apt-get install --download-only linux-generic` → "E: Unable to locate package"

  3. **Zero Squid proxy requests from node5 during this window.** Confirmed by Squid logs
     on infra2: the last entry for node5 (10.241.144.81) was at 17:44:00 (ZFS packages
     downloaded in the ephemeral pre-curtin phase), and the next entry is not until 18:14:26
     (second PXE boot cycle). During the entire curtin run (17:44:11–17:44:57) — including
     the `apt-get update` in `install_kernel` — no HTTP requests reached the proxy.

  4. **The apt-get update ran with empty or inaccessible sources**, causing it to exit
     immediately without fetching any package lists. The freshly-extracted Noble squashfs
     target has no cached apt package lists (`/var/lib/apt/lists/` is empty), so after
     the no-op update, `linux-generic` could not be found.

  5. **The most likely cause of the empty sources**: The temp sources.list at
     `/tmp/tmpsyktkyxm/sources.list` (inside the chroot), created by Curtin
     `23.1.1-1124-g7324b43b` for the `apt-get update` call, was either empty, contained
     an unsubstituted `$RELEASE` literal (which APT silently ignores), or pointed to an
     unreachable location. The `writing-apt-config` stage logged
     `got primary mirror: None / got security mirror: None` — indicating curtin found no
     existing mirror configuration when examining the target's sources.list — suggesting
     the sources.list state was unexpected at that point.

  6. **Curthooks FAIL → curtin exits without running `late_commands`**. The `late_commands`
     contained the critical `wget … op=netboot_off` call that signals MAAS to flip the PXE
     boot order. Without it, MAAS assumed deployment was still in progress and kept serving
     the nodes the PXE boot image.

  7. **Both node5 and node6 exhibited identical behaviour**: each cycled through ~24 PXE
     boot/curtin-fail loops over 30 minutes. On each subsequent boot, the nodes' DHCP
     requests were served normally and they re-entered curtin, but the kernel install step
     failed identically each time.

  **Evidence:**

  - Finding 1: `finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: FAIL: installing kernel` at 17:44:57 (infra2 syslog)
  - Finding 2: `E: Unable to locate package linux-generic` from apt inside the chroot at 17:44:57
  - Finding 3: Zero Squid proxy entries for 10.241.144.81 between 17:44:01 and 18:14:26 (confirmed via infra2 and infra3 syslog)
  - Finding 4: All pre-`install_kernel` curthook stages were SUCCESS — this is not a network or commissioning failure
  - Finding 5: All bcache partitioning, OS extraction, and early curthooks passed without error
  - Finding 6: `apt-get update` took < 50ms (total `install_kernel` time was ~246ms), ruling out network timeout
  - Finding 7: Curtin version `23.1.1-1124-g7324b43b` — a development build beyond 23.1.1

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/2147326/+subscriptions