Eric Radman : a Journal

Bhyve and iPXE

webm | mp4

In software or systems engineering, the ability to spin up virtual machines is a valuable capability. Virtual machines provide a means of testing software or configuration in on multiple platforms, and in multiple configurations.

In production, a high-performance and reliable hypervisor allows services to be provisioned and updated one at a time. This capability avoids the added risk and complexity of fork-lift upgrades.

Bhyve is somewhat similar QEMU-KVM without the emulation required to boot SeaBIOS. Native UEFI support plus iPXE provides some very flexible means of bootstrapping VMs.

Boot Image

iPXE is a very capable UEFI application with very obscure documentation. Specifically, it is a challenging to determine what configuration iPXE it tries to find by default—the Internet is full of incorrect answers. strings(1) resolved this question!

$ strings /usr/local/share/ipxe/snp.efi-x86_64 | egrep '.ipxe$'

Bhyve does not have a built-in PXE boot support, but we can build our own boot image which will be added asn an nvme device when the VM is started

dhcp && goto netboot || goto dhcperror

prompt --key s --timeout 10000 DHCP failed, hit 's' for the iPXE shell; reboot in 10 seconds && shell || reboot

chain tftp://${next-server}/chainload/${hostname}.ipxe ||
prompt --key s --timeout 10000 Chainloading failed, hit 's' for the iPXE shell; reboot in 10 seconds && shell || reboot

The trick in this configuration is that iPXE will fetch further configuration based on next-server and host-name DHCP options.



truncate -s 4M $dst
mdconfig -a -t vnode -u 99 -f $dst
gpart create -s gpt /dev/md99
gpart add -t efi /dev/md99
newfs_msdos -F12 /dev/md99p1
mount -t msdosfs -o longnames /dev/md99p1 /mnt

mkdir -p /mnt/EFI/Boot
cp /usr/local/share/ipxe/snp.efi-x86_64 /mnt/EFI/Boot/BootX64.efi
cp autoexec.ipxe /mnt/autoexec.ipxe

umount /mnt
mdconfig -du md99

Process Status

In Bhyve, each virtual machine is a process, and each virtual CPU is a thread which may be listed using top -H or procstat(1)

# procstat -t 4470
  PID    TID COMM                TDNAME              CPU  PRI STATE   WCHAN
 4470 100259 bhyve               mevent               -1  120 sleep   kqread
 4470 100510 bhyve               blk-4:0-0            -1  120 sleep   uwait
 4470 100511 bhyve               blk-4:0-1            -1  120 sleep   uwait
 4470 100512 bhyve               blk-4:0-2            -1  120 sleep   uwait
 4470 100513 bhyve               blk-4:0-3            -1  120 sleep   uwait
 4470 100514 bhyve               blk-4:0-4            -1  120 sleep   uwait
 4470 100515 bhyve               blk-4:0-5            -1  120 sleep   uwait
 4470 100516 bhyve               blk-4:0-6            -1  120 sleep   uwait
 4470 100517 bhyve               blk-4:0-7            -1  120 sleep   uwait
 4470 100518 bhyve               vtnet-5:0 tx         -1  120 sleep   uwait
 4470 100519 bhyve               rfb                  -1  126 sleep   accept
 4470 100520 bhyve               vcpu 0               -1  128 sleep   vmidle
 4470 100521 bhyve               vcpu 1                1  124 run     -
 4470 100522 bhyve               vcpu 2                9  127 run     -
 4470 100523 bhyve               vcpu 3               -1  134 sleep   vmidle
 4470 100524 bhyve               vcpu 4                3  125 run     -
 4470 100525 bhyve               vcpu 5                4  136 run     -

VLAN Tagging and Bridging

On FreeBSD 802.1Q tagging is configured by defining a list of VLAN numbers for an interface

vlans_em0="80 81 82"


Now that VLAN interfaces are defined, create a bridge with VLAN 82 as it's first member

ifconfig_bridge2="addm em0.82 up"

A Complete Startup Script

There are a number of Bhyve frameworks, but my approach has been to construct a shell script to manage the lifetime using bhyve(8)


trap 'printf "$0: exit code $? on line $LINENO\n"' ERR

if [ $# -lt 1 ]; then
    echo "$0 name [-i]"
    exit 1


[ -f /vm/$name.img ] || [ $opt == '-i' ] || {
    echo "$name not found, add -i to initialize"
    exit 1

case $name in
    dev1)  # Ubuntu or Rocky Linux with one disk
        if [ $opt == '-i' ]; then
            install="-s 1,nvme,/root/uefi-ipxe-chainload.img"
            bhyvectl --destroy --vm=$name
            truncate -s 50G /vm/$name.img
        ifconfig $if create
        ifconfig bridge2 addm $if
        bhyve -c 4 -m 12G -w -H \
              -s 0,hostbridge \
              $install \
              -s 4,virtio-blk,/vm/$name.img \
              -s 5,virtio-net,$if,mac=00:0c:29:f9:6d:4e \
              -s 29,fbuf,tcp=$vp,w=800,h=600 \
              -s 30,xhci,tablet \
              -s 31,lpc \
              -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
               $name && exec $0 $1
        ifconfig $if destroy

    install)  # OpenBSD with two disks
        if [ $opt == '-i' ]; then
            install="-s 3,ahci-cd,/iso/miniroot73.img"
            bhyvectl --destroy --vm=$name
            truncate -s 20G /vm/$name.img
            truncate -s 80G /vm/$name-data.img
        ifconfig $if create
        ifconfig bridge2 addm $if
        bhyve -c 1 -m 512M -w -H \
              -s 0,hostbridge \
              $install \
              -s 4,virtio-blk,/vm/$name.img \
              -s 5,virtio-blk,/vm/$name-data.img \
              -s 6,virtio-net,$if,mac=00:0c:29:a5:19:b9 \
              -s 29,fbuf,tcp=$vp,w=800,h=600 \
              -s 30,xhci,tablet \
              -s 31,lpc \
              -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
               $name && exec $0 $1
        ifconfig $if destroy

This script can be refactored to be less repetitious, but will serve as a starting point. Key features:

OpenBSD's miniroot is deceptively easy to use for an automated network install since install.sub reads the install server and filename from /var/db/dhcpleased/$_if.