1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>f3s: Kubernetes with FreeBSD - Rocky Linux Bhyve VMs - Part 3</title>
<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
<link rel="stylesheet" href="../style.css" />
<link rel="stylesheet" href="style-override.css" />
</head>
<body>
<p class="header">
View this page as <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-bhyve.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-bhyve.gmi">Gemini</a>
</p>
<h1 style='display: inline' id='f3s-kubernetes-with-freebsd---rocky-linux-bhyve-vms---part-3'>f3s: Kubernetes with FreeBSD - Rocky Linux Bhyve VMs - Part 3</h1><br />
<br />
<span>This is the third blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution we will use on FreeBSD-based physical machines.</span><br />
<br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<br />
<a href='./f3s-kubernetes-with-frhyveeebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-frhyveeebsd-part-1/f3slogo.png' /></a><br />
<br />
<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
<br />
<ul>
<li><a href='#f3s-kubernetes-with-freebsd---rocky-linux-bhyve-vms---part-3'>f3s: Kubernetes with FreeBSD - Rocky Linux Bhyve VMs - Part 3</a></li>
<li>⇢ <a href='#introduction'>Introduction</a></li>
<li>⇢ <a href='#basic-bhyve-setup'>Basic Bhyve setup</a></li>
<li>⇢ <a href='#rocky-linux-vms'>Rocky Linux VMs</a></li>
<li>⇢ ⇢ <a href='#iso-download'>ISO download</a></li>
<li>⇢ ⇢ <a href='#vm-configuration'>VM configuration</a></li>
<li>⇢ ⇢ <a href='#vm-installation'>VM installation</a></li>
<li>⇢ ⇢ <a href='#increase-of-the-disk-image'>Increase of the disk image</a></li>
<li>⇢ ⇢ <a href='#connect-to-vpn'>Connect to VPN</a></li>
<li>⇢ <a href='#after-install'>After install</a></li>
</ul><br />
<h2 style='display: inline' id='introduction'>Introduction</h2><br />
<br />
<span>In this blog post, we are going to install the Bhyve hypervisor.</span><br />
<br />
<span>The FreeBSD Bhyve hypervisor is a lightweight, modern hypervisor that enables virtualization on FreeBSD systems. Bhyve's strengths include its minimal overhead, which allows it to achieve near-native performance for virtual machines. It is designed to be efficient and lightweight, leveraging the capabilities of the FreeBSD operating system for performance and network management.</span><br />
<br />
<span>Bhyve supports running a variety of guest operating systems, including FreeBSD, Linux, and Windows, on hardware platforms that support hardware virtualization extensions (such as Intel VT-x or AMD-V). In our case, we are going to virtualize Rocky Linux, which later on in this series will be used to run k3s.</span><br />
<br />
<h2 style='display: inline' id='basic-bhyve-setup'>Basic Bhyve setup</h2><br />
<br />
<span>For the management of the Bhyve VMs, we are using <span class='inlinecode'>vm-bhyve</span>, a tool not part of the FreeBSD operating system but available as a ready-to-use package. It eases VM management and reduces a lot of the overhead. We also install the required package to make Bhyve work with the UEFI firmware.</span><br />
<br />
<a class='textlink' href='https://github.com/churchers/vm-bhyve'>https://github.com/churchers/vm-bhyve</a><br />
<br />
<span>The following commands are executed on all three hosts <span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span>, and <span class='inlinecode'>f2</span>, where <span class='inlinecode'>re0</span> is the name of the Ethernet interface (which may need to be adjusted if your hardware is different):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas pkg install vm-bhyve bhyve-firmware
paul@f0:~ % doas sysrc vm_enable=YES
vm_enable: -> YES
paul@f0:~ % doas sysrc vm_dir=zfs:zroot/bhyve
vm_dir: -> zfs:zroot/bhyve
paul@f0:~ % doas zfs create zroot/bhyve
paul@f0:~ % doas vm init
paul@f0:~ % doas vm switch create public
paul@f0:~ % doas vm switch add public re0
</pre>
<br />
<span>Bhyve stores all it's data in the <span class='inlinecode'>/bhyve</span> of the <span class='inlinecode'>zroot</span> ZFS pool:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % zfs list | grep bhyve
zroot/bhyve <font color="#000000">1</font>.74M 453G <font color="#000000">1</font>.74M /zroot/bhyve
</pre>
<br />
<span>For convenience, we also create this symlink:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas ln -s /zroot/bhyve/ /bhyve
</pre>
<br />
<span>Now, Bhyve is ready to rumble, but no VMs are there yet:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
</pre>
<br />
<h2 style='display: inline' id='rocky-linux-vms'>Rocky Linux VMs</h2><br />
<br />
<h3 style='display: inline' id='iso-download'>ISO download</h3><br />
<br />
<span>We're going to install the Rocky Linux from the latest minimal iso:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas vm iso \
https://download.rockylinux.org/pub/rocky/<font color="#000000">9</font>/isos/x86_64/Rocky-<font color="#000000">9.5</font>-x86_64-minimal.iso
/zroot/bhyve/.iso/Rocky-<font color="#000000">9.5</font>-x86_64-minimal.iso <font color="#000000">1808</font> MB <font color="#000000">4780</font> kBps 06m28s
paul@f0:/bhyve % doas vm create rocky
</pre>
<h3 style='display: inline' id='vm-configuration'>VM configuration</h3><br />
<br />
<span>The default configuration looks like this now:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:/bhyve/rocky % cat rocky.conf
loader=<font color="#808080">"bhyveload"</font>
cpu=<font color="#000000">1</font>
memory=256M
network0_type=<font color="#808080">"virtio-net"</font>
network0_switch=<font color="#808080">"public"</font>
disk0_type=<font color="#808080">"virtio-blk"</font>
disk0_name=<font color="#808080">"disk0.img"</font>
uuid=<font color="#808080">"1c4655ac-c828-11ef-a920-e8ff1ed71ca0"</font>
network0_mac=<font color="#808080">"58:9c:fc:0d:13:3f"</font>
</pre>
<br />
<span>Whereas the <span class='inlinecode'>uuid</span> and the <span class='inlinecode'>network0_mac</span> differ on each of the 3 hosts.</span><br />
<br />
<span>but in order to make Rocky Linux boot it (plus some other adjustments, e.g. as I am intending to run the majority of the workload in the k3s cluster running on those linux VMs, I give them beefy specs like 4 CPU cores and 14GB RAM), I modified it to:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>guest=<font color="#808080">"linux"</font>
loader=<font color="#808080">"uefi"</font>
uefi_vars=<font color="#808080">"yes"</font>
cpu=<font color="#000000">4</font>
memory=14G
network0_type=<font color="#808080">"virtio-net"</font>
network0_switch=<font color="#808080">"public"</font>
disk0_type=<font color="#808080">"virtio-blk"</font>
disk0_name=<font color="#808080">"disk0.img"</font>
graphics=<font color="#808080">"yes"</font>
graphics_vga=io
uuid=<font color="#808080">"1c45400b-c828-11ef-8871-e8ff1ed71cac"</font>
network0_mac=<font color="#808080">"58:9c:fc:0d:13:3f"</font>
</pre>
<br />
<h3 style='display: inline' id='vm-installation'>VM installation</h3><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas vm install rocky Rocky-<font color="#000000">9.5</font>-x86_64-minimal.iso
Starting rocky
* found guest <b><u><font color="#000000">in</font></u></b> /zroot/bhyve/rocky
* booting...
paul@f0:/bhyve/rocky % doas vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
rocky default uefi <font color="#000000">4</font> 14G <font color="#000000">0.0</font>.<font color="#000000">0.0</font>:<font color="#000000">5900</font> No Locked (f0.lan.buetow.org)
paul@f0:/bhyve/rocky % doas sockstat -<font color="#000000">4</font> | grep <font color="#000000">5900</font>
root bhyve <font color="#000000">6079</font> <font color="#000000">8</font> tcp4 *:<font color="#000000">5900</font> *:*
</pre>
<br />
<span>Port 5900 is now also open for VNC connections, so we connect to it with a VNC client and run through the installation dialogs. I'm sure this could be done unattended or more automated, but we have only 3 VMs to install, and the automation doesn't seem worth it as we are doing it only once.</span><br />
<br />
<h3 style='display: inline' id='increase-of-the-disk-image'>Increase of the disk image</h3><br />
<br />
<span>By default the VMs disk image is only 20G, which is a bit small for my purposes, so I stopped the VMs again and run <span class='inlinecode'>truncate</span> on the image file to enlarge them to 100G, and re-started the installation:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:/bhyve/rocky % doas vm stop rocky
paul@f0:/bhyve/rocky % doas truncate -s 100G disk0.img
paul@f0:/bhyve/rocky % doas vm install rocky Rocky-<font color="#000000">9.5</font>-x86_64-minimal.iso
</pre>
<br />
<h3 style='display: inline' id='connect-to-vpn'>Connect to VPN</h3><br />
<br />
<span>For the installation, I opened the VPN client on my Fedora laptop (GNOME comes with a simple VPN client) and ran through the base installation for each of the VMs manually. I am sure this could have been automated a bit more, but there were just 3 VMs, and it wasn't worth the effort. The three VNC addresses of the VMs were: <span class='inlinecode'>vnc://f0:5900</span>, <span class='inlinecode'>vnc://f1:5900</span>, and <span class='inlinecode'>vnc://f0:5900</span>.</span><br />
<br />
<span>I mostly selected the default settings (auto partitioning on the 100GB drive and a root user password). After the installation, the VMs were rebooted.</span><br />
<br />
<h2 style='display: inline' id='after-install'>After install</h2><br />
<br />
<span>After that, I changed the network configuration to be static here as well.</span><br />
<br />
<span>As per previous post of this series, the 3 FreeBSD hosts were already in my <span class='inlinecode'>/etc/hosts</span> file:</span><br />
<br />
<pre>
192.168.1.130 f0 f0.lan f0.lan.buetow.org
192.168.1.131 f1 f1.lan f1.lan.buetow.org
192.168.1.132 f2 f2.lan f2.lan.buetow.org
</pre>
<br />
<span>For the Rocky VMs I added those:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>cat <<END >>/etc/hosts
<font color="#000000">192.168</font>.<font color="#000000">1.120</font> r0 r0.lan r0.lan.buetow.org
<font color="#000000">192.168</font>.<font color="#000000">1.121</font> r1 r1.lan r1.lan.buetow.org
<font color="#000000">192.168</font>.<font color="#000000">1.122</font> r2 r2.lan r2.lan.buetow.org
END
</pre>
<span>and configured the IPs accordingly on the VMs themselves.</span><br />
<br />
<br />
<span>Other *BSD-related posts:</span><br />
<br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
<p class="footer">
Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
</p>
</body>
</html>
|
