From 5c503195b9d89ef1c392322f50e55addd0bed45d Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sun, 26 Jan 2025 23:22:26 +0200 Subject: Update content for gemtext --- gemfeed/4 | 239 ++++++++++++++ .../DRAFT-f3s-kubernetes-with-freebsd-part-3.gmi | 360 +++++++++++++++++++++ gemfeed/f3s-kubernetes-with-freebsd-part-4.gmi.tpl | 239 ++++++++++++++ 3 files changed, 838 insertions(+) create mode 100644 gemfeed/4 create mode 100644 gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-3.gmi create mode 100644 gemfeed/f3s-kubernetes-with-freebsd-part-4.gmi.tpl diff --git a/gemfeed/4 b/gemfeed/4 new file mode 100644 index 00000000..f4c59789 --- /dev/null +++ b/gemfeed/4 @@ -0,0 +1,239 @@ +# f3s: Kubernetes with FreeBSD - Rocky Linux Bhyve VMs - Part 3 + +This is the third blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution we will use on FreeBSD-based physical machines. + +<< template::inline::index f3s-kubernetes-with-freebsd-part + +=> ./f3s-kubernetes-with-frhyveeebsd-part-1/f3slogo.png f3s logo + +<< template::inline::toc + +## Introduction + +In this blog post, we are going to install the Bhyve hypervisor. + +The FreeBSD Bhyve hypervisor is a lightweight, modern hypervisor that enables virtualization on FreeBSD systems. Bhyve's strengths include its minimal overhead, which allows it to achieve near-native performance for virtual machines. It is designed to be efficient and lightweight, leveraging the capabilities of the FreeBSD operating system for performance and network management. + +Bhyve supports running a variety of guest operating systems, including FreeBSD, Linux, and Windows, on hardware platforms that support hardware virtualization extensions (such as Intel VT-x or AMD-V). In our case, we are going to virtualize Rocky Linux, which later on in this series will be used to run k3s. + +## Basic Bhyve setup + +For the management of the Bhyve VMs, we are using `vm-bhyve`, a tool not part of the FreeBSD operating system but available as a ready-to-use package. It eases VM management and reduces a lot of the overhead. We also install the required package to make Bhyve work with the UEFI firmware. + +=> https://github.com/churchers/vm-bhyve + +The following commands are executed on all three hosts `f0`, `f1`, and `f2`, where `re0` is the name of the Ethernet interface (which may need to be adjusted if your hardware is different): + +```sh +paul@f0:~ % doas pkg install vm-bhyve bhyve-firmware +paul@f0:~ % doas sysrc vm_enable=YES +vm_enable: -> YES +paul@f0:~ % doas sysrc vm_dir=zfs:zroot/bhyve +vm_dir: -> zfs:zroot/bhyve +paul@f0:~ % doas zfs create zroot/bhyve +paul@f0:~ % doas vm init +paul@f0:~ % doas vm switch create public +paul@f0:~ % doas vm switch add public re0 +``` + +Bhyve stores all it's data in the `/bhyve` of the `zroot` ZFS pool: + +```sh +paul@f0:~ % zfs list | grep bhyve +zroot/bhyve 1.74M 453G 1.74M /zroot/bhyve +``` + +For convenience, we also create this symlink: + +```sh +paul@f0:~ % doas ln -s /zroot/bhyve/ /bhyve + +``` + +Now, Bhyve is ready to rumble, but no VMs are there yet: + +```sh +paul@f0:~ % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +``` + +## Rocky Linux VMs + +### ISO download + +We're going to install the Rocky Linux from the latest minimal iso: + +```sh +paul@f0:~ % doas vm iso \ + https://download.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.5-x86_64-minimal.iso +/zroot/bhyve/.iso/Rocky-9.5-x86_64-minimal.iso 1808 MB 4780 kBps 06m28s +paul@f0:/bhyve % doas vm create rocky +``` +### VM configuration + +The default configuration looks like this now: + +```sh +paul@f0:/bhyve/rocky % cat rocky.conf +loader="bhyveload" +cpu=1 +memory=256M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +uuid="1c4655ac-c828-11ef-a920-e8ff1ed71ca0" +network0_mac="58:9c:fc:0d:13:3f" +``` + +Whereas the `uuid` and the `network0_mac` differ on each of the 3 hosts. + +but in order to make Rocky Linux boot it (plus some other adjustments, e.g. as I am intending to run the majority of the workload in the k3s cluster running on those linux VMs, I give them beefy specs like 4 CPU cores and 14GB RAM), I run `doas vm configure rocky` and modified it to: + +``` +guest="linux" +loader="uefi" +uefi_vars="yes" +cpu=4 +memory=14G +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +graphics="yes" +graphics_vga=io +uuid="1c45400b-c828-11ef-8871-e8ff1ed71cac" +network0_mac="58:9c:fc:0d:13:3f" +``` + +### VM installation + +To start the installer from the downloaded ISO, I run: + +```sh +paul@f0:~ % doas vm install rocky Rocky-9.5-x86_64-minimal.iso +Starting rocky + * found guest in /zroot/bhyve/rocky + * booting... + +paul@f0:/bhyve/rocky % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +rocky default uefi 4 14G 0.0.0.0:5900 No Locked (f0.lan.buetow.org) + +paul@f0:/bhyve/rocky % doas sockstat -4 | grep 5900 +root bhyve 6079 8 tcp4 *:5900 *:* +``` + +Port 5900 now also opened for VNC connections, so I connected to it with a VNC client and run through the installation dialogs. I'm sure this could be done unattended or more automated, there are only 3 VMs to install, and the automation doesn't seem worth it as we are doing it only once in a year or less often. + +### Increase of the disk image + +By default the VMs disk image is only 20G, which is a bit small for my purposes, so I stopped the VMs again and run `truncate` on the image file to enlarge them to 100G, and re-started the installation: + +```sh +paul@f0:/bhyve/rocky % doas vm stop rocky +paul@f0:/bhyve/rocky % doas truncate -s 100G disk0.img +paul@f0:/bhyve/rocky % doas vm install rocky Rocky-9.5-x86_64-minimal.iso +``` + +### Connect to VPN + +For the installation, I opened the VPN client on my Fedora laptop (GNOME comes with a simple VPN client) and ran through the base installation for each of the VMs manually. Again, I am sure this could have been automated a bit more, but there were just 3 VMs, and it wasn't worth the effort. The three VNC addresses of the VMs were: `vnc://f0:5900`, `vnc://f1:5900`, and `vnc://f0:5900`. + +I mostly selected the default settings (auto partitioning on the 100GB drive and a root user password). After the installation, the VMs were rebooted. + +## After install + +I performed the following steps for all 3 VMs. In the following, the examples are all executed on `f0` (bzw the VM `r0` running on `f0`): + +### VM auto-start after host reboot + +To automatically start the VM on the servers I added the following to the `rc.conf` on the FreeBSD hosts: + +```sh + +paul@f0:/bhyve/rocky % cat <>/etc/hosts +192.168.1.120 r0 r0.lan r0.lan.buetow.org +192.168.1.121 r1 r1.lan r1.lan.buetow.org +192.168.1.122 r2 r2.lan r2.lan.buetow.org +END +```` + +Whereas: + +* `192.168.1.120` is the IP of the VM itself (here: `r0.lan.buetow.org`) +* `192.168.1.1` is the address of my home router, which also does DNS. + +### Permitting root login + +As these VMs arent directly reachable via SSH from the internet, I enabled `root` login by adding a line with `PermitRootLogin yes` to `/etc/sshd/sshd_config`. + +Once done, I rebooted the VM by running `reboot` inside of the vm to test whether everything was configured and persisted correctly. + +After reboot, I copied my public key from my Laptop to the 3 VMs: + +```sh +% for i in 0 1 2; do ssh-copy-id root@r$i.lan.buetow.org; done +``` + +And then I edited the `/etc/ssh/sshd_config` file again on all 3 VMs and configured `PasswordAuthentication no`, to only allow SSH key authentication from now on. + +### Install latest updates + +```sh +[root@r0 ~] % dnf update +[root@r0 ~] % dreboot +``` + +CPU STRESS TESTER VM VS NOT VM + +Other *BSD-related posts: + +<< template::inline::index bsd + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +=> ../ Back to the main site diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-3.gmi b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-3.gmi new file mode 100644 index 00000000..e9ef4724 --- /dev/null +++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-3.gmi @@ -0,0 +1,360 @@ +# f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts + +This is the third blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution we will use on FreeBSD-based physical machines. + +=> ./2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi 2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage +=> ./2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi 2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation + +=> ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo + +## Table of Contents + +* ⇢ f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts +* ⇢ ⇢ Introduction +* ⇢ ⇢ Changes since last time +* ⇢ ⇢ ⇢ FreeBSD upgrade from 14.1 to 14.2 +* ⇢ ⇢ ⇢ A new home (behind the TV) +* ⇢ ⇢ The UPS hardware +* ⇢ ⇢ Configuring FreeBSD to Work with the UPS +* ⇢ ⇢ ⇢ USB Device Detection +* ⇢ ⇢ ⇢ `apcupsd` Installation +* ⇢ ⇢ ⇢ UPS Connectivity Test +* ⇢ ⇢ APC Info on Partner Nodes: +* ⇢ ⇢ ⇢ Installation on partners +* ⇢ ⇢ Power outage simulation +* ⇢ ⇢ ⇢ Pulling the plug +* ⇢ ⇢ ⇢ Restoring power + +## Introduction + +In this blog post, we are setting up the UPS for the cluster. A UPS, or Uninterruptible Power Supply, safeguards my cluster from unexpected power outages and surges. It acts as a backup battery that kicks in when the electricity cuts out—especially useful in my area, where power cuts are frequent—allowing for a graceful system shutdown and preventing data loss and corruption. This is especially important since I will also store some of my data on the f3s nodes. + +## Changes since last time + +### FreeBSD upgrade from 14.1 to 14.2 + +There has been a new release since the last blog post in this series. The upgrade from 14.1 was as easy as: + +```sh +paul@f0: ~ % doas freebsd-update fetch +paul@f0: ~ % doas freebsd-update install +paul@f0: ~ % doas freebsd-update -r 14.2-RELEASE upgrade +paul@f0: ~ % doas freebsd-update install +paul@f0: ~ % doas shutdown -r now +``` + +And after rebooting, I ran: + +```sh +paul@f0: ~ % doas freebsd-update install +paul@f0: ~ % doas pkg update +paul@f0: ~ % doas pkg upgrade +paul@f0: ~ % doas shutdown -r now +``` + +And after another reboot, I was on 14.2: + +```sh +paul@f0:~ % uname -a +FreeBSD f0.lan.buetow.org 14.2-RELEASE FreeBSD 14.2-RELEASE + releng/14.2-n269506-c8918d6c7412 GENERIC amd64 +``` + +And, of course, I ran this on all 3 nodes! + +### A new home (behind the TV) + +I've put all the infrastructure behind my TV, as plenty of space is available. The TV hides most of the setup, which drastically improved the SAF (spouse acceptance factor). + +=> ./f3s-kubernetes-with-freebsd-part-3/f3s-changes.jpg New hardware placement arrangement + +I got rid of the mini-switch I mentioned in the previous blog post. I have the TP-Link EAP615-Wall mounted on the wall nearby, which is my OpenWrt-powered Wi-Fi hotspot. It also has 3 Ethernet ports, to which I connected the Beelink nodes. That's the device you see at the very top. + +The Ethernet cables go downward through the cable boxes to the Beelink nodes. In addition to the Beelink f3s nodes, I connected the TP-Link to the UPS as well (not discussed further in this blog post, but the positive side effect is that my Wi-Fi will still work during a power loss for some time—and during a power cut, the Beelink nodes will still be able to communicate with each other). + +On the very left (the black box) is the UPS, with four power outlets. Three go to the Beelink nodes, and one goes to the TP-Link. A USB output is also connected to the first Beelink node, `f0`. + +On the very right (halfway hidden behind the TV) are the 3 Beelink nodes stacked on top of each other. The only downside (or upside?) is that my 14-month-old daughter is now chaos-testing the Beelink nodes, as the red power buttons (now reachable for her) are very attractive for her to press when passing by randomly. :-) Luckily, that will only cause graceful system shutdowns! + +## The UPS hardware + +I wanted a UPS that I could connect to via FreeBSD, and that would provide enough backup power to operate the cluster for a couple of minutes (it turned out to be around an hour, but this time will likely be shortened after future hardware upgrades, like additional drives and a backup enclosure) and to automatically initiate the shutdown of all the f3s nodes. + +I decided on the APC Back-UPS BX750MI model because: + +* Zero noise level when there is no power cut (some light noise when the battery is in operation during a power cut). +* Cost: It is relatively affordable (not costing thousands). +* USB connectivity: Can be connected via USB to one of the FreeBSD hosts to read the UPS status. +* A power output of 750VA (or 410 watts), suitable for an hour of runtime for my f3s nodes (plus the Wi-Fi router). +* Multiple power outlets: Can connect all 3 f3s nodes directly. +* User-replaceable batteries: I can replace the batteries myself after two years or more (depending on usage). +* Its compact design. Overall, I like how it looks. + +=> ./f3s-kubernetes-with-freebsd-part-3/apc-back-ups.jpg The APC Back-UPS BX750MI in operation. + +## Configuring FreeBSD to Work with the UPS + +### USB Device Detection + +Once plugged in via USB on FreeBSD, I could see the following in the kernel messages: + +```sh +paul@f0: ~ % doas dmesg | grep UPS +ugen0.2: at usbus0 +``` + +### `apcupsd` Installation + +To make use of the USB connection, the `apcupsd` package had to be installed: + +```sh +paul@f0: ~ % doas install apcupsd +``` + +I have made the following modifications to the configuration file so that the UPS can be used via the USB interface: + +```sh +paul@f0:/usr/local/etc/apcupsd % diff -u apcupsd.conf.sample apcupsd.conf +--- apcupsd.conf.sample 2024-11-01 16:40:42.000000000 +0200 ++++ apcupsd.conf 2024-12-03 10:58:24.009501000 +0200 +@@ -31,7 +31,7 @@ + # 940-1524C, 940-0024G, 940-0095A, 940-0095B, + # 940-0095C, 940-0625A, M-04-02-2000 + # +-UPSCABLE smart ++UPSCABLE usb + + # To get apcupsd to work, in addition to defining the cable + # above, you must also define a UPSTYPE, which corresponds to +@@ -88,8 +88,10 @@ + # that apcupsd binds to that particular unit + # (helpful if you have more than one USB UPS). + # +-UPSTYPE apcsmart +-DEVICE /dev/usv ++UPSTYPE usb ++DEVICE + + # POLLTIME + # Interval (in seconds) at which apcupsd polls the UPS for status. This +``` + +I left the remaining settings as the default ones; for example, the following are of main interest: + +``` +# If during a power failure, the remaining battery percentage +# (as reported by the UPS) is below or equal to BATTERYLEVEL, +# apcupsd will initiate a system shutdown. +BATTERYLEVEL 5 + +# If during a power failure, the remaining runtime in minutes +# (as calculated internally by the UPS) is below or equal to MINUTES, +# apcupsd, will initiate a system shutdown. +MINUTES 3 +``` + +I then enabled and started the daemon: + +```sh +paul@f0:/usr/local/etc/apcupsd % doas sysrc apcupsd_enable=YES +apcupsd_enable: -> YES +paul@f0:/usr/local/etc/apcupsd % doas service apcupsd start +Starting apcupsd. +``` + +### UPS Connectivity Test + +And voila, I could now access the UPS information via the `apcaccess` command; how convenient :-) (I also read through the manual page, which provides a good understanding of what else can be done with it!). + +```sh +paul@f0:~ % apcaccess +APC : 001,035,0857 +DATE : 2025-01-26 14:43:27 +0200 +HOSTNAME : f0.lan.buetow.org +VERSION : 3.14.14 (31 May 2016) freebsd +UPSNAME : f0.lan.buetow.org +CABLE : USB Cable +DRIVER : USB UPS Driver +UPSMODE : Stand Alone +STARTTIME: 2025-01-26 14:43:25 +0200 +MODEL : Back-UPS BX750MI +STATUS : ONLINE +LINEV : 230.0 Volts +LOADPCT : 4.0 Percent +BCHARGE : 100.0 Percent +TIMELEFT : 65.3 Minutes +MBATTCHG : 5 Percent +MINTIMEL : 3 Minutes +MAXTIME : 0 Seconds +SENSE : Medium +LOTRANS : 145.0 Volts +HITRANS : 295.0 Volts +ALARMDEL : No alarm +BATTV : 13.6 Volts +LASTXFER : Automatic or explicit self test +NUMXFERS : 0 +TONBATT : 0 Seconds +CUMONBATT: 0 Seconds +XOFFBATT : N/A +SELFTEST : NG +STATFLAG : 0x05000008 +SERIALNO : 9B2414A03599 +BATTDATE : 2001-01-01 +NOMINV : 230 Volts +NOMBATTV : 12.0 Volts +NOMPOWER : 410 Watts +END APC : 2025-01-26 14:44:06 +0200 +``` + +## APC Info on Partner Nodes: + +So far, so good. Host `f0` would shut down itself when short on power. But what about the `f1` and `f2` nodes? They aren't connected directly to the UPS and, therefore, wouldn't know that their power is about to be cut off. For this, `apcupsd` running on the `f1` and `f2` nodes can be configured to retrieve UPS information via the network from the `apcupsd` server running on the `f0` node, which is connected directly to the APC via USB. + +Of course, this won't work when `f0` is down. In this case, no operational node would be connected to the UPS via USB; therefore, the current power status would not be known. However, I consider this a rare circumstance. Furthermore, in case of an `f0` system crash, sudden power outages on the two other nodes would occur at different times, making real data loss (the main concern here) effectively impossible. + +### Installation on partners + +To do this, I installed `apcupsd` via `doas pkg install apcupsd` on `f1` and `f2`, and then I could connect to it this way: + +```sh +paul@f1:~ % apcaccess -h f0.lan.buetow.org | grep Percent +LOADPCT : 12.0 Percent +BCHARGE : 94.0 Percent +MBATTCHG : 5 Percent +``` + +But I want the daemon to be configured and enabled in such a way that it connects to the master UPS node (the one with the UPS connected via USB) so that it can also initiate a system shutdown when the UPS battery reaches low levels. For that, `apcupsd` itself needs to be aware of the UPS status. + +On `f1` and `f2`, I changed the configuration to use `f0` (where `apcupsd` is listening) as a remote device. I also changed the `MINUTES` setting from 3 to 6 and the `BATTERYLEVEL` setting from 5 to 10 to ensure that the `f1` and `f2` nodes could still connect to the `f0` node for UPS information before `f0` decides to shut down itself. So `f1` and `f2` must shut down earlier than `f0`: + +```sh +paul@f2:/usr/local/etc/apcupsd % diff -u apcupsd.conf.sample apcupsd.conf +--- apcupsd.conf.sample 2024-11-01 16:40:42.000000000 +0200 ++++ apcupsd.conf 2025-01-26 15:52:45.108469000 +0200 +@@ -31,7 +31,7 @@ + # 940-1524C, 940-0024G, 940-0095A, 940-0095B, + # 940-0095C, 940-0625A, M-04-02-2000 + # +-UPSCABLE smart ++UPSCABLE ether + + # To get apcupsd to work, in addition to defining the cable + # above, you must also define a UPSTYPE, which corresponds to +@@ -52,7 +52,6 @@ + # Network Information Server. This is used if the + # UPS powering your computer is connected to a + # different computer for monitoring. +-# + # snmp hostname:port:vendor:community + # SNMP network link to an SNMP-enabled UPS device. + # Hostname is the ip address or hostname of the UPS +@@ -88,8 +87,8 @@ + # that apcupsd binds to that particular unit + # (helpful if you have more than one USB UPS). + # +-UPSTYPE apcsmart +-DEVICE /dev/usv ++UPSTYPE net ++DEVICE f0.lan.buetow.org:3551 + + # POLLTIME + # Interval (in seconds) at which apcupsd polls the UPS for status. This +@@ -147,12 +146,12 @@ + # If during a power failure, the remaining battery percentage + # (as reported by the UPS) is below or equal to BATTERYLEVEL, + # apcupsd will initiate a system shutdown. +-BATTERYLEVEL 5 ++BATTERYLEVEL 10 + + # If during a power failure, the remaining runtime in minutes + # (as calculated internally by the UPS) is below or equal to MINUTES, + # apcupsd, will initiate a system shutdown. +-MINUTES 3 ++MINUTES 6 + + # If during a power failure, the UPS has run on batteries for TIMEOUT + # many seconds or longer, apcupsd will initiate a system shutdown. + +``` +So I also ran the following commands on `f1` and `f2`: + +```sh +paul@f1:/usr/local/etc/apcupsd % doas sysrc apcupsd_enable=YES +apcupsd_enable: -> YES +paul@f1:/usr/local/etc/apcupsd % doas service apcupsd start +Starting apcupsd. +``` + +And then I was able to connect to localhost via the `apcaccess` command: + +```sh +paul@f1:~ % doas apcaccess | grep Percent +LOADPCT : 5.0 Percent +BCHARGE : 95.0 Percent +MBATTCHG : 5 Percent +``` + +## Power outage simulation + +### Pulling the plug + +I simulated a power outage by removing the power input from the APC. Immediately, the following message appeared on all the nodes: + +```` +Broadcast Message from root@f0.lan.buetow.org + (no tty) at 15:03 EET... + +Power failure. Running on UPS batteries. +``` + +I ran the following command to confirm the available battery time: + +```sh +paul@f0:/usr/local/etc/apcupsd % apcaccess -p TIMELEFT +63.9 Minutes +``` + +And after almost 60 minutes (`f1` and `f2` a bit earlier, `f0` a bit later due to the different `BATTERYLEVEL` and `MINUTES` settings outlined earlier), the following broadcast was sent out: + +``` +Broadcast Message from root@f0.lan.buetow.org + (no tty) at 15:08 EET... + + *** FINAL System shutdown message from paul@f1.lan.buetow.org *** + +System going down IMMEDIATELY + +apcupsd initiated shutdown +``` + +And all the nodes shut down safely before the UPS ran out of battery! + +### Restoring power + +After restoring power, I checked the logs in `/var/log/daemon.log` and found the following on all 3 nodes: + +``` +Jan 26 17:36:24 f2 apcupsd[2159]: Power failure. +Jan 26 17:36:30 f2 apcupsd[2159]: Running on UPS batteries. +Jan 26 17:36:30 f2 apcupsd[2159]: Battery charge below low limit. +Jan 26 17:36:30 f2 apcupsd[2159]: Initiating system shutdown! +Jan 26 17:36:30 f2 apcupsd[2159]: User logins prohibited +Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd exiting, signal 15 +Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded +``` + +All good :-) + +Other BSD related posts are: + +=> ./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi 2016-04-09 Jails and ZFS with Puppet on FreeBSD +=> ./2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi 2022-07-30 Let's Encrypt with OpenBSD and Rex +=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2024-01-13-one-reason-why-i-love-openbsd.gmi 2024-01-13 One reason why I love OpenBSD +=> ./2024-04-01-KISS-high-availability-with-OpenBSD.gmi 2024-04-01 KISS high-availability with OpenBSD +=> ./2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi 2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage +=> ./2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi 2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +=> ../ Back to the main site diff --git a/gemfeed/f3s-kubernetes-with-freebsd-part-4.gmi.tpl b/gemfeed/f3s-kubernetes-with-freebsd-part-4.gmi.tpl new file mode 100644 index 00000000..f8348c7e --- /dev/null +++ b/gemfeed/f3s-kubernetes-with-freebsd-part-4.gmi.tpl @@ -0,0 +1,239 @@ +# f3s: Kubernetes with FreeBSD - Rocky Linux Bhyve VMs - Part 4 + +This is the thourth blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution we will use on FreeBSD-based physical machines. + +<< template::inline::index f3s-kubernetes-with-freebsd-part + +=> ./f3s-kubernetes-with-frhyveeebsd-part-1/f3slogo.png f3s logo + +<< template::inline::toc + +## Introduction + +In this blog post, we are going to install the Bhyve hypervisor. + +The FreeBSD Bhyve hypervisor is a lightweight, modern hypervisor that enables virtualization on FreeBSD systems. Bhyve's strengths include its minimal overhead, which allows it to achieve near-native performance for virtual machines. It is designed to be efficient and lightweight, leveraging the capabilities of the FreeBSD operating system for performance and network management. + +Bhyve supports running a variety of guest operating systems, including FreeBSD, Linux, and Windows, on hardware platforms that support hardware virtualization extensions (such as Intel VT-x or AMD-V). In our case, we are going to virtualize Rocky Linux, which later on in this series will be used to run k3s. + +## Basic Bhyve setup + +For the management of the Bhyve VMs, we are using `vm-bhyve`, a tool not part of the FreeBSD operating system but available as a ready-to-use package. It eases VM management and reduces a lot of the overhead. We also install the required package to make Bhyve work with the UEFI firmware. + +=> https://github.com/churchers/vm-bhyve + +The following commands are executed on all three hosts `f0`, `f1`, and `f2`, where `re0` is the name of the Ethernet interface (which may need to be adjusted if your hardware is different): + +```sh +paul@f0:~ % doas pkg install vm-bhyve bhyve-firmware +paul@f0:~ % doas sysrc vm_enable=YES +vm_enable: -> YES +paul@f0:~ % doas sysrc vm_dir=zfs:zroot/bhyve +vm_dir: -> zfs:zroot/bhyve +paul@f0:~ % doas zfs create zroot/bhyve +paul@f0:~ % doas vm init +paul@f0:~ % doas vm switch create public +paul@f0:~ % doas vm switch add public re0 +``` + +Bhyve stores all it's data in the `/bhyve` of the `zroot` ZFS pool: + +```sh +paul@f0:~ % zfs list | grep bhyve +zroot/bhyve 1.74M 453G 1.74M /zroot/bhyve +``` + +For convenience, we also create this symlink: + +```sh +paul@f0:~ % doas ln -s /zroot/bhyve/ /bhyve + +``` + +Now, Bhyve is ready to rumble, but no VMs are there yet: + +```sh +paul@f0:~ % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +``` + +## Rocky Linux VMs + +### ISO download + +We're going to install the Rocky Linux from the latest minimal iso: + +```sh +paul@f0:~ % doas vm iso \ + https://download.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.5-x86_64-minimal.iso +/zroot/bhyve/.iso/Rocky-9.5-x86_64-minimal.iso 1808 MB 4780 kBps 06m28s +paul@f0:/bhyve % doas vm create rocky +``` +### VM configuration + +The default configuration looks like this now: + +```sh +paul@f0:/bhyve/rocky % cat rocky.conf +loader="bhyveload" +cpu=1 +memory=256M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +uuid="1c4655ac-c828-11ef-a920-e8ff1ed71ca0" +network0_mac="58:9c:fc:0d:13:3f" +``` + +Whereas the `uuid` and the `network0_mac` differ on each of the 3 hosts. + +but in order to make Rocky Linux boot it (plus some other adjustments, e.g. as I am intending to run the majority of the workload in the k3s cluster running on those linux VMs, I give them beefy specs like 4 CPU cores and 14GB RAM), I run `doas vm configure rocky` and modified it to: + +``` +guest="linux" +loader="uefi" +uefi_vars="yes" +cpu=4 +memory=14G +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +graphics="yes" +graphics_vga=io +uuid="1c45400b-c828-11ef-8871-e8ff1ed71cac" +network0_mac="58:9c:fc:0d:13:3f" +``` + +### VM installation + +To start the installer from the downloaded ISO, I run: + +```sh +paul@f0:~ % doas vm install rocky Rocky-9.5-x86_64-minimal.iso +Starting rocky + * found guest in /zroot/bhyve/rocky + * booting... + +paul@f0:/bhyve/rocky % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +rocky default uefi 4 14G 0.0.0.0:5900 No Locked (f0.lan.buetow.org) + +paul@f0:/bhyve/rocky % doas sockstat -4 | grep 5900 +root bhyve 6079 8 tcp4 *:5900 *:* +``` + +Port 5900 now also opened for VNC connections, so I connected to it with a VNC client and run through the installation dialogs. I'm sure this could be done unattended or more automated, there are only 3 VMs to install, and the automation doesn't seem worth it as we are doing it only once in a year or less often. + +### Increase of the disk image + +By default the VMs disk image is only 20G, which is a bit small for my purposes, so I stopped the VMs again and run `truncate` on the image file to enlarge them to 100G, and re-started the installation: + +```sh +paul@f0:/bhyve/rocky % doas vm stop rocky +paul@f0:/bhyve/rocky % doas truncate -s 100G disk0.img +paul@f0:/bhyve/rocky % doas vm install rocky Rocky-9.5-x86_64-minimal.iso +``` + +### Connect to VPN + +For the installation, I opened the VPN client on my Fedora laptop (GNOME comes with a simple VPN client) and ran through the base installation for each of the VMs manually. Again, I am sure this could have been automated a bit more, but there were just 3 VMs, and it wasn't worth the effort. The three VNC addresses of the VMs were: `vnc://f0:5900`, `vnc://f1:5900`, and `vnc://f0:5900`. + +I mostly selected the default settings (auto partitioning on the 100GB drive and a root user password). After the installation, the VMs were rebooted. + +## After install + +I performed the following steps for all 3 VMs. In the following, the examples are all executed on `f0` (bzw the VM `r0` running on `f0`): + +### VM auto-start after host reboot + +To automatically start the VM on the servers I added the following to the `rc.conf` on the FreeBSD hosts: + +```sh + +paul@f0:/bhyve/rocky % cat <>/etc/hosts +192.168.1.120 r0 r0.lan r0.lan.buetow.org +192.168.1.121 r1 r1.lan r1.lan.buetow.org +192.168.1.122 r2 r2.lan r2.lan.buetow.org +END +```` + +Whereas: + +* `192.168.1.120` is the IP of the VM itself (here: `r0.lan.buetow.org`) +* `192.168.1.1` is the address of my home router, which also does DNS. + +### Permitting root login + +As these VMs arent directly reachable via SSH from the internet, I enabled `root` login by adding a line with `PermitRootLogin yes` to `/etc/sshd/sshd_config`. + +Once done, I rebooted the VM by running `reboot` inside of the vm to test whether everything was configured and persisted correctly. + +After reboot, I copied my public key from my Laptop to the 3 VMs: + +```sh +% for i in 0 1 2; do ssh-copy-id root@r$i.lan.buetow.org; done +``` + +And then I edited the `/etc/ssh/sshd_config` file again on all 3 VMs and configured `PasswordAuthentication no`, to only allow SSH key authentication from now on. + +### Install latest updates + +```sh +[root@r0 ~] % dnf update +[root@r0 ~] % dreboot +``` + +CPU STRESS TESTER VM VS NOT VM + +Other *BSD-related posts: + +<< template::inline::index bsd + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +=> ../ Back to the main site -- cgit v1.2.3