# f3s: Kubernetes with FreeBSD - Rocky Linux Bhyve VMs - Part 3 This is the third blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution we will use on FreeBSD-based physical machines. << template::inline::index f3s-kubernetes-with-freebsd-part => ./f3s-kubernetes-with-frhyveeebsd-part-1/f3slogo.png f3s logo << template::inline::toc ## Introduction In this blog post, we are going to install the Bhyve hypervisor. The FreeBSD Bhyve hypervisor is a lightweight, modern hypervisor that enables virtualization on FreeBSD systems. Bhyve's strengths include its minimal overhead, which allows it to achieve near-native performance for virtual machines. It is designed to be efficient and lightweight, leveraging the capabilities of the FreeBSD operating system for performance and network management. Bhyve supports running a variety of guest operating systems, including FreeBSD, Linux, and Windows, on hardware platforms that support hardware virtualization extensions (such as Intel VT-x or AMD-V). In our case, we are going to virtualize Rocky Linux, which later on in this series will be used to run k3s. ## Basic Bhyve setup For the management of the Bhyve VMs, we are using `vm-bhyve`, a tool not part of the FreeBSD operating system but available as a ready-to-use package. It eases VM management and reduces a lot of the overhead. We also install the required package to make Bhyve work with the UEFI firmware. => https://github.com/churchers/vm-bhyve The following commands are executed on all three hosts `f0`, `f1`, and `f2`, where `re0` is the name of the Ethernet interface (which may need to be adjusted if your hardware is different): ```sh paul@f0:~ % doas pkg install vm-bhyve bhyve-firmware paul@f0:~ % doas sysrc vm_enable=YES vm_enable: -> YES paul@f0:~ % doas sysrc vm_dir=zfs:zroot/bhyve vm_dir: -> zfs:zroot/bhyve paul@f0:~ % doas zfs create zroot/bhyve paul@f0:~ % doas vm init paul@f0:~ % doas vm switch create public paul@f0:~ % doas vm switch add public re0 ``` Bhyve stores all it's data in the `/bhyve` of the `zroot` ZFS pool: ```sh paul@f0:~ % zfs list | grep bhyve zroot/bhyve 1.74M 453G 1.74M /zroot/bhyve ``` For convenience, we also create this symlink: ```sh paul@f0:~ % doas ln -s /zroot/bhyve/ /bhyve ``` Now, Bhyve is ready to rumble, but no VMs are there yet: ```sh paul@f0:~ % doas vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE ``` ## Rocky Linux VMs ### ISO download We're going to install the Rocky Linux from the latest minimal iso: ```sh paul@f0:~ % doas vm iso \ https://download.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.5-x86_64-minimal.iso /zroot/bhyve/.iso/Rocky-9.5-x86_64-minimal.iso 1808 MB 4780 kBps 06m28s paul@f0:/bhyve % doas vm create rocky ``` ### VM configuration The default configuration looks like this now: ```sh paul@f0:/bhyve/rocky % cat rocky.conf loader="bhyveload" cpu=1 memory=256M network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0.img" uuid="1c4655ac-c828-11ef-a920-e8ff1ed71ca0" network0_mac="58:9c:fc:0d:13:3f" ``` Whereas the `uuid` and the `network0_mac` differ on each of the 3 hosts. but in order to make Rocky Linux boot it (plus some other adjustments, e.g. as I am intending to run the majority of the workload in the k3s cluster running on those linux VMs, I give them beefy specs like 4 CPU cores and 14GB RAM), I run `doas vm configure rocky` and modified it to: ``` guest="linux" loader="uefi" uefi_vars="yes" cpu=4 memory=14G network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0.img" graphics="yes" graphics_vga=io uuid="1c45400b-c828-11ef-8871-e8ff1ed71cac" network0_mac="58:9c:fc:0d:13:3f" ``` ### VM installation To start the installer from the downloaded ISO, I run: ```sh paul@f0:~ % doas vm install rocky Rocky-9.5-x86_64-minimal.iso Starting rocky * found guest in /zroot/bhyve/rocky * booting... paul@f0:/bhyve/rocky % doas vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE rocky default uefi 4 14G 0.0.0.0:5900 No Locked (f0.lan.buetow.org) paul@f0:/bhyve/rocky % doas sockstat -4 | grep 5900 root bhyve 6079 8 tcp4 *:5900 *:* ``` Port 5900 now also opened for VNC connections, so I connected to it with a VNC client and run through the installation dialogs. I'm sure this could be done unattended or more automated, there are only 3 VMs to install, and the automation doesn't seem worth it as we are doing it only once in a year or less often. ### Increase of the disk image By default the VMs disk image is only 20G, which is a bit small for my purposes, so I stopped the VMs again and run `truncate` on the image file to enlarge them to 100G, and re-started the installation: ```sh paul@f0:/bhyve/rocky % doas vm stop rocky paul@f0:/bhyve/rocky % doas truncate -s 100G disk0.img paul@f0:/bhyve/rocky % doas vm install rocky Rocky-9.5-x86_64-minimal.iso ``` ### Connect to VPN For the installation, I opened the VPN client on my Fedora laptop (GNOME comes with a simple VPN client) and ran through the base installation for each of the VMs manually. Again, I am sure this could have been automated a bit more, but there were just 3 VMs, and it wasn't worth the effort. The three VNC addresses of the VMs were: `vnc://f0:5900`, `vnc://f1:5900`, and `vnc://f0:5900`. I mostly selected the default settings (auto partitioning on the 100GB drive and a root user password). After the installation, the VMs were rebooted. ## After install I performed the following steps for all 3 VMs. In the following, the examples are all executed on `f0` (bzw the VM `r0` running on `f0`): ### VM auto-start after host reboot To automatically start the VM on the servers I added the following to the `rc.conf` on the FreeBSD hosts: ```sh paul@f0:/bhyve/rocky % cat <>/etc/hosts 192.168.1.120 r0 r0.lan r0.lan.buetow.org 192.168.1.121 r1 r1.lan r1.lan.buetow.org 192.168.1.122 r2 r2.lan r2.lan.buetow.org END ```` Whereas: * `192.168.1.120` is the IP of the VM itself (here: `r0.lan.buetow.org`) * `192.168.1.1` is the address of my home router, which also does DNS. ### Permitting root login As these VMs arent directly reachable via SSH from the internet, I enabled `root` login by adding a line with `PermitRootLogin yes` to `/etc/sshd/sshd_config`. Once done, I rebooted the VM by running `reboot` inside of the vm to test whether everything was configured and persisted correctly. After reboot, I copied my public key from my Laptop to the 3 VMs: ```sh % for i in 0 1 2; do ssh-copy-id root@r$i.lan.buetow.org; done ``` And then I edited the `/etc/ssh/sshd_config` file again on all 3 VMs and configured `PasswordAuthentication no`, to only allow SSH key authentication from now on. ### Install latest updates ```sh [root@r0 ~] % dnf update [root@r0 ~] % dreboot ``` CPU STRESS TESTER VM VS NOT VM Other *BSD-related posts: << template::inline::index bsd E-Mail your comments to `paul@nospam.buetow.org` :-) => ../ Back to the main site