Running an Ubuntu ppc64le virtual machine on x86

For an issue I'm debugging, I need a 64-bit little-endian powerpc machine. I don't have a physical one handy, so I wanted to run a virtual one on an x86 system.

We can use qemu and libvirt for this. Unfortunately, it will be very slow - we have to use qemu's TCG mode rather than hardware assisted virtualisation because the guest (POWER) and host (amd64) are different. Fortunately, libvirt can work with these VMs like normal, which I find very helpful as it allows me to use tools like virsh and virt-manager, rather than dealing with qemu's inscrutable command line arguments.

So I grabbed an Ubuntu ppc64el install ISO (Xenial, Bionic) and set to work.

Here's the virt-install incantation derived from much accumulated wisdom amongst my former colleagues at OzLabs:

virt-install --arch=ppc64le --name=GUESTVMNAME \
  --os-variant=linux --boot menu=on \
  --disk path=GUESTDISK.qcow2,size=20,format=qcow2,bus=virtio,cache=none,sparse=true \
  --memory=4096  --vcpu=2  --wait 0  --graphics none \
  --cdrom=/home/ubuntu/ubuntu-18.04.1-server-ppc64el.iso

If you run this on an x86 Xenial host, the installation will begin and you can interact with it by running virsh console (or use virt-manager).

Sadly, it does not work; the installer kernel panics. Removing 'quiet' from the boot commands in grub shows messages like the following:

modprobe[98]: unhandled signal 4 at 00007d96d2364fbc nip 00007d96d2364fbc lr 00007d96d2332ab8 code 1

Mikey Neuling pointed out that this was a SIGILL - illegal instruction - which reminded me that while Ubuntu only supports POWER8 for ppc64el, qemu will emulate a lot further back - and by default on Xenial I guess it uses an older ISA.

So I manually set the machine type to POWER8. I used virt-manager, which generated this snippet of libvirt xml under the domain tag:

  <cpu mode='custom' match='exact' check='none'>
    <model fallback='allow'>POWER8</model>
  </cpu>

This, however, won't start - virsh start and virt-manager report:

qemu-system-ppc64le: No Transactional Memory support in TCG, try cap-htm=off

Suraj Jitindar Singh explained some of the backstory here, and suggested that I try specifying a particular "machine type" to qemu that would default to having hardware transactional memory off. Fortunately, his suggestion - pseries-2.12 - worked: the machine started, the installer is installing, and I'm much happier.

I used virsh edit to do this - the relevant snippet is:

    <type arch='ppc64le' machine='pseries-2.12'>hvm</type>

This lives under os, under domain.

The story on Bionic

Fortunately Bionic does a better job of all this.

The virt-install command fails straight out of the box with the cap-htm error. Fortunately, it's easy to specify the machine type for virt-install: just append --machine=pseries-2.12:

virt-install --arch=ppc64le --name=GUESTVMNAME --machine=pseries-2.12 \
  --os-variant=linux --boot menu=on \
  --disk path=GUESTDISK.qcow2,size=20,format=qcow2,bus=virtio,cache=none,sparse=true \
  --memory=4096  --vcpu=2  --wait 0  --graphics none \
  --cdrom=/home/ubuntu/ubuntu-18.04.1-server-ppc64el.iso

This works! I suppose the machine type must imply a POWER8 cpu, because I didn't need to specify it.

Update

With a bionic guest, the installation worked on the serial console, but systemd failed to spawn a terminal on it for some reason. At least, it's probably systemd's fault, and it's convenient to blame it! I haven't actually confirmed this: I added a display in virt-manager and installed an openssh server so I can now access it in 2 different ways.

See also

• uvtool: I normally use this for my VMs, but it has some issues starting ppc64 guests. It flat-out fails on Xenial (it starts it as an x86 guest!) and on Bionic it fails too - it seems to assume that it will be running under KVM. I haven't dug into this any further.
• The very helpful instructions from Michael Ellerman on the linuxppc/linux wiki: Booting with Qemu. If I didn't need a pretty expansive user-space I would totally have just used this.