Before embarking its worth reviewing the requirements as running Windows on Bhyve does require a bit of work and not for the faint hearted. It also requires some pretty specific hardware requirements/features as Bhyve has been built from the ground up with modern hardware accelerated virtualisation features at the forefront leave all the legacy cruff in its wake.
- FreeBSD Current 11 r288524 (minimum)
- Kernel Modules
- vmm.ko (Bhyve)
- nmdm.ko (Null Modem)
- if_tap.ko (tap network interface)
- if_bridge.ko (bridge network interface)
- Microsoft Windows Server 2012 R2 Installation ISO
- An AutoUnattend.xml file
- To automate the installation as there is no graphical console.
- virtio Windows drivers
- CPU Supporting EPT with Unrestricted Guest Feature
- 7zip and cdrtools packages installed
- Some spare time and patients
Links to related articles
As mentioned in the requirements, EPT unrestricted guest feature is required so knowing you have EPT support may not be enough if you have a CPU older than 2010. For instance, I have a pair of HP DL 160 G6’s with 2 x Quad Core Xeon L5520’s in each server. The Xeon L5520 was initially release Q1 2009 and although has support for EPT, it does not have the crucial ‘unrestricted guest’ feature found in the Westmere released Q1 2010. The lacking the unrestricted guest feature has two drawbacks. The first is you cannot boot any operating system apart from FreeBSD and second, you can only allocate a single vCPU, multi vCPU vms would require unrestricted guest. The reason you can run a FreeBSD vm is because it boots directly into 64-bit mode; everything else (including multiple vCPU guests) requires either 16-bit support or 32-bit non-paged protected mode, both of which are only available on Intel with the ‘unrestricted guest’ feature.
Lacking the unrestricted guest feature will result in the following errors.
2 vCPUs requested but only 1 available
ROM boot failed: unrestricted guest capability not available
With the doom and gloom over, hopefully you have a recent modern CPU, most likely you’re using a workstation or desktop with a recent consumer CPU with all the bells and whistles.
First let move to our home directory if not already done so and add the required kernel modules to loader.conf to load at boot
echo 'vmm_load="YES"' >>/boot/loader.conf echo 'nmdm_load="YES"' >>/boot/loader.conf
Due to Bhyve lacking a graphical console and no really serial installation support (except for the out of band SAC Special Administration Console) an unattended install is required. After the installation of Windows is successful, console access can be facilitated with Remote Desktop Protocol (RDP) using the /admin. Execute mstsc from run or search box with the admin flag depending on your version of Windows. On old version of RDP you may need to use the console flag instead of admin. If using a non Microsoft OS, FreeRDP is widely available.
e.g. mstsc /admin e.g. mstsc /console
Although you can create your own AutoUnattend.xml file by hand, or using WAIK (Windows Automated Installation Kit) and SIM (System Image Manager), thankfully someone has already created one for the various Windows OS versions. If you have a WDS infrastructure, you most likely already have one which you spent alot of cycles on. Feel free to use that instead.
Download the following AutoUnattend.xml file using Git (shown below) or http. Feel free to edit selections as you see fit.
git clone https://github.com/nahanni/bhyve-windows-unattend-xml/blob/master/files/vista_AutoUnattend.xml
git clone https://github.com/nahanni/bhyve-windows-unattend-xml/blob/master/files/win10_AutoUnattend.xml
Windows Server 2012/2012 R2:
git clone https://github.com/nahanni/bhyve-windows-unattend-xml/blob/master/files/win2012r2_AutoUnattend.xml
Windows Server 2016 Preview:
git clone https://github.com/nahanni/bhyve-windows-unattend-xml/blob/master/files/win2016preview_AutoUnattend.xml
Windows 2008/2008 R2:
git clone https://github.com/nahanni/bhyve-windows-unattend-xml/blob/master/files/win2k8_AutoUnattend.xml
git clone https://github.com/nahanni/bhyve-windows-unattend-xml/blob/master/files/win7_AutoUnattend.xml
git clone https://github.com/nahanni/bhyve-windows-unattend-xml/blob/master/files/win8_AutoUnattend.xml
If you have a product key and want to add it to the AutoUnattend.xml file, find the following section
and replace with
<ProductKey> <WillShowUI>OnError</WillShowUI> <Key>PRODUCT KEY HERE</Key> </ProductKey>
We need to download the Virtio driver ISO, so again, download using fetch or the browser of your choice.
Virtio drivers for Windows 7, 8, 8.1, 10 and Server 2012, 2012 R2:
Virtio drivers for Windows Vista and Windows 2008:
Next we need to remaster a Microsoft Windows ISO so that it includes both the AutoUnattend.xml and the Windows Virtio drivers.
This can be done using FreeBSD as shown below, or you can use Windows with CDRTOOLS 4 Window which using cgywin, contains a Windows executable of mkisofs.
Create a directory called “iso-windows” which will be used as the source for the remastered Windows ISO. In addition, create a subdirectory that will contain the Windows Virtio drivers.
mkdir -p iso-windows/virtio
The next step requires 7zip for exacting the Windows ISO as tar shipped with FreeBSD base does not play well with UDF. Install cdrtools using pkg(8)
pkg install p7zip-9.38.1_2
With 7zip installed, go ahead and extract the content of your Windows ISO to the directory iso-windows. Replace <WINDOWS> with the file name of your ISO.
/usr/local/bin/7z x <WINDOWS>.iso -o iso-windows
Using tar extract the required drivers from the previously downloaded Virtio ISO. Replace <VERSION> with the virtio version number ISO you downloaded and <OS> with the Windows operating system you’re installing.
tar xf virtio-win-<VERSION>.iso --include="NetKVM/<OS>/amd64/*" -C iso-windows/virtio/ e.g. tar xf virtio-win-0.1.96.iso --include="NetKVM/2k12R2/amd64/*" -C iso-windows/virtio/
To use mkisofs, install cdrtools using pkg(8)
pkg install cdrtools
Copy and rename the AutoUnattend xml file you downloaded earlier. Replace <AUTOUNATTEND FILE> with the correct file you chose for your Windows OS install.
cp <AUTOUNATTEND FILE>.xml iso-windows/AutoUnattend.xml e.g. cp win2012r2_AutoUnattend.xml iso-remaster/AutoUnattend.xml
Now its time to remaster your Windows ISO. Replace <FILE NAME> with your own choice of file name.
mkisofs \ -b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \ -iso-level 4 -J -l -D \ -N -joliet-long \ -relaxed-filenames -v \ -V "Custom" -udf \ -boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \ -eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \ -no-emul-boot \ -o <FILE NAME>.iso iso-windows e.g. mkisofs \ -b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \ -iso-level 4 -J -l -D \ -N -joliet-long \ -relaxed-filenames -v \ -V "Custom" -udf \ -boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \ -eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \ -no-emul-boot \ -o Windows-2012R2.iso iso-windows
Creating a Virtual Hard Disk
Although you can use .img files created using truncate as a virtual hard disk, I prefer to use ZFS datasets and volume to organise both vm specific files and disks. If you’re using UFS then img files are really your only option.
Create a .img file for use as a virtual disk if not using ZFS.
truncate -s 40GB windows-disk0.img
Assuming you already have an existing pool, replace <POOL> with the name of your ZFS pool and <NAME OF VM> with your vm name, I like to use the hostname.
zfs create <POOL>/BHYVE e.g. zfs create TANK/BHYVE
zfs create <POOL>/BHYVE/VM e.g. zfs create TANK/BHYVE/VM
zfs create <POOL>/BHYVE/VM/<NAME OF VM> e.g. zfs create TANK/BHYVE/VM/WIN-SRV-01
Here we create ZFS Volume (ZVOL) of 40G in size
zfs create -V 40G <POOL>/BHYVE/VM/<NAME OF VM>/disk0 e.g. zfs create -V 40G TANK/BHYVE/VM/WIN-SRV-01/disk0
Its also worth creating an ISO dataset to store installation media. Once created, move your remaster Windows ISO to this directory.
Repository for installation media ISOs
zfs create <POOL>/BHYVE/ISO e.g. zfs create TANK/BHYVE/ISO
mv <FILE NAME>.iso <POOL>/BHYVE/ISO e.g. mv Windows-2012R2.iso TANK/BHYVE/ISO
Along with the installation ISO you will need the UEFI loader. I like to store the UEFI loader in the top level dataset of the vm I’m configuring. so in this example we’ll change directory to WIN-SRV-01
cd /TANK/BHYVE/VM/WIN-SRV-01/ fetch https://people.freebsd.org/~grehan/bhyve_uefi/BHYVE_UEFI_20151002.fd
Now rename the downloaded file for no other reason than its easier to read.
mv BHYVE_UEFI_20151002.fd BHYVE_UEFI.fd
We’re going to need to configure networking before creating and starting the VM. We will need a tap(4) device (L2 Network Interface), which will be used as the network interface inside the vm as well as a bridge(4). The bridge(4) will allow us to “bridge” two or more ethernet device thus creating a logical link between them at layer 2. You will want to bridge(4) your tap(4) device with a physical network interface that has LAN and Internet access.
Create a tap(4) device
ifconfig tap0 create
Create a bridge(4) device and add members. You will need to replace <PHYSICAL NETWORK INTERFACE> with your interface depending on the manufacture/driver. As an example I have a Broadcom NIC which uses the drive bge, so my interface is bge0.
ifconfig bridge0 create ifconfig bridge0 addm tap0 addm <PHYSICAL NETWORK INTERFACE>
There are three stages to installing Windows. Each stage will bring you back to the SAC (Special Administration Console) which looks like
Computer is booting, SAC started and initialized. Use the "ch -?" command for information about using channels. Use the "?" command for general help. SAC>
Between stages one and two, the VM will shutdown. Under normal circumstances, the vm would reboot, but right now Bhyve will simple stop the vm and we are required to manually start it. Before restarting the vm between each stage we must relinquish the resource and recreate them at each boot.
To destroy the previously running vm and relinquish the resources use the following
bhyvectl --vm <VM NAME> --destroy
To see if the stopped vm is still present and holding on to resources, running
If you received any result from ls destroy the vm using the above bhyvectl command.
As with any installation of Windows, the installation ISO is only required for the first boot. If a bootable ISO is present when creating/starting the vm e.g.
-s 4,ahci-cd,TANK/BHYVE/ISO/Windows-2012R2.iso the EFI loader will always prefer the ahci-cd device (CD-ROM), endlessly installing Windows every subsequent boot. This is a limitation of the EFI loader as boot selection is currently not present. Because of this we must remove the ahci-cd/CD-ROM device from the bhyve command after the first boot. Desktop editions of Windows require that a null install.iso file remains.
To create a null ISO file if installing a Windows desktop OS, do the following
Rather than removing the ahci-cd device for a Windows desktop OS, we must replace the bootable ISO with a null/0-bytes, non-bootable ISO using the following
Boot Windows Installer
With everything in place, lets go ahead and boot our Windows vm for the first time with the remastered Windows installation ISO. I will use my example vm “WIN-SRV-01” and directories in all subsequent commands, but change the Bold text to match your own environment.
bhyve \ -c 2 \ -s 0,hostbridge \ -s 3,ahci-hd,/dev/zvol/TANK/BHYVE/VM/WIN-SRV-01/disk0 \ -s 4,ahci-cd,TANK/BHYVE/ISO/Windows-2012R2.iso \ -s 10,virtio-net,tap0 \ -s 31,lpc \ -l com1,/dev/nmdm0A \ -l bootrom,BHYVE_UEFI_20151002.fd \ -m 2G -H -w \ WIN-SRV-01
One thing to remember is that Bhyve doesn’t exit (runs in the foreground) until the vm has been turned off/restarted so you will need another terminal session open alongside the one running Bhyve. This will be used for consoling in using the null modem (nmdm) device.
With another terminal session open, let use cu(1) to connect via
cu -l /dev/nmdm0B -s 9600
You will be greeted by debug output from the UEFI loader. Once successfully booted from the ISO you will be met with the SAC. The initial
SAC> prompt will appear immediately, followed by three EVENTs. Depending on your hardware these three channel creation EVENTs can take time, so be patient. Each SAC channel performs a different role. You can list available channels at any time with using
Computer is booting, SAC started and initialized. Use the "ch -?" command for information about using channels. Use the "?" command for general help. SAC> EVENT: The CMD command is now available. SAC> EVENT: A new channel has been created. Use "ch -?" for channel help. Channel: SACSetupAct SAC> EVENT: A new channel has been created. Use "ch -?" for channel help. Channel: SACSetupErr
TAB+ESC you can toggle between the channels. The most useful to gauging the installation progress is
SACSetupAct. Eventually you will see the WIM image being decompressed to disk, this can take time so once again patients is the key. This will be visible by running
gstat in another terminal session (do not close the current terminal session as it will kill Bhyve) as disk IO and CPU utilisation will be high.
... 2016-01-01 16:43:23, Info [0x06009e] IBS DeployWIMImage:Calling IDepWIMImageResolved::Apply... 2016-01-01 16:43:23, Info [0x0606cc] IBS Calling WIMApplyImage (flags = 0x180)...
At the end of the installation Bhyve will terminate and in the terminal window, you will be return to the shell prompt. Before booting the vm for stage two, we need to destroy the vm to relinquish the resrouces.
bhyvectl --vm WIN-SRV-01; --destroy
Restart the vm but this time without the ahci-cd/CD-ROM device and as such, the Windows Installation ISO
N.B. If you’re install a Windows desktop OS, be assure to leave the ahci-cd device in place and pass it the
null-install.iso file create earlier.
bhyve \ -c 2 \ -s 0,hostbridge \ -s 3,ahci-hd,/dev/zvol/TANK/BHYVE/VM/WIN-SRV-01/disk0 \ -s 10,virtio-net,tap0 \ -s 31,lpc \ -l com1,/dev/nmdm0A \ -l bootrom,BHYVE_UEFI_20151002.fd \ -m 2G -H -w \ WIN-SRV-01
After booting the vm for stage two, if you receive the following, it suggests the install in stage one did not complete successfully as EFI has unsuccessfully booted from the ahci-cd device, then ahci-hd device and is now trying to boot from the network.
Booting EFI Network InstallProtocolInterface: 245DCA21-FB7B-11D3-8F01-00A0C969723B BEC7C0E0 .