KVM support in Juju on ARM64

[Reed O'Brien]

KVM as a juju deploy target now works on AMD64 and ARM64. There is an issue
on ARM64 using trusty, so it only works with xenial on the hardware I could
test on atm. There are a number of issues that prevent running juju on
trusty on ARM64. Following is a support matrix, an outline of known issues,
and scenarios where we might run into the issue of trusty not working as
expected. Finally there are a couple thoughts about handling the issue. Any
input on options/solutions for this issue are appreciated.

ARM64 QEMU/KVM UEFI Support Matrix

Hardware Interrupt Controller

Ubuntu release

Kernel

Virt Stack

Ubuntu release

Kernel

qemu-efi

GICv2

*

*

*

*

*

Xenial

GICv3

Trusty

GA (3.13)

*

*

*

Xenial

GICv3

Trusty

*

Trusty

*

*

Xenial

GICv3

*

*

*

*

GA (3.13)

Xenial

GICv3

Trusty

hwe-*

Trusty + Cloud Archive (>=Newton)

Trusty

hwe-*

Xenial

GICv3

Trusty

hwe-*

Trusty + Cloud Archive (>=Newton)

Xenial

*

Xenial

GICv3

Xenial

*

*

Trusty

hwe-*

Xenial

GICv3

Xenial

*

*

Xenial

*

Xenial

   - EFI guests require qemu-efi, which is only available in xenial.


   - Trusty’s 3.13 kernel can’t boot on GICv3 systems, nor as a guest on
   GICv3 systems.


   - Trusty’s virt stack does not support GICv3 guests, but that support is
   available in the cloud archive.


The first issue is that there is a very limited set of hardware that was
certified for trusty on ARM64. So that severely limits the number of
systems where we would expect juju to be able to deploy to trusty. As noted
in the next issue is possible using the hwe-x kernel, but that is not
something juju can control.

The second issue is that those systems include GICv2 (General Interrupt
Controller v2), whereas newer systems include GICv3.  This would require
trusty to boot with hwe-x kernel. AFAIU there are no official cloud images
with that kernel pre-installed. Additionally, it currently seems it isn't
possible to set hwe-x as the minimum kernel version in MAAS:
https://bugs.launchpad.net/maas/+bug/1659694

Newer versions of qemu/libvirt support setting the GIC version to "host"
which allows the VM to use whatever the host supports as long as the guest
supports. Trusty guests will not boot on a GICv3 host.  Also "host" as a
GIC element’s version attribute isn't supported by the virt stack that
ships with trusty.

One final issue is that there is no qemu-efi package built for trusty. It
is feasible that someone could build their own ppa for use.  Other options
include:

- adding qemu-efi to trusty-backports which is not easy to use,
- push a stable release update back to trusty, which is a regression risk
for x86 by updating   the source that also builds ovmf,
- introduce qemu-efi in trusty which would require an exception from the
security team.


There are a few scenarios which involve deploying trusty.

- juju bootstrap a-cloud arm64-controller --default-series trusty
       This would a require a trusty cloud image with hwe-x kernel
pre-installed.
- juju add-machine --series trusty
       This would require the underlying provider to boot with hwe-x kernel
       minimum. This also seems like it is going to fail to deploy with the
       wily kernel.
- juju deploy wordpress --to kvm:0
       This would require the machine-0 to be able to install qemu-efi and
it
       would also require a trusty cloud image with hwe-x kernel
pre-installed.



So given all the possible knobs one would need to turn to make it work,
should we:

- block it from working in juju and return an error noting that trusty on
ARM64 isn’t supported?

This in conjunction with the deploy scenario in the last section seems to
be a real issue. One could decide to deploy only xenial hosts and guests on
ARM64, but it would hobble the usefulness of quite a few charms that
require trusty or bundles that require such charms. This seems draconian
when selecting a particular minimum kernel in MAAS could make it work --
that is if qemu-efi was available and there weren’t hardware version issues.

- warn that it won't work without a custom setup and document it as a known
issue pointing in to a wiki page in the warning/error that contains the
information above?
- something else/other ideas?


Cheers,
Reed

-- 
Reed O'Brien
✉ reed.obrien at canonical.com
✆ 415-562-6797
💻 redir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20170130/5d6757ec/attachment-0001.html>