1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</title>
<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
<link rel="stylesheet" href="../style.css" />
<link rel="stylesheet" href="style-override.css" />
</head>
<body>
<div class="rfx-overlay-grid"></div>
<div class="rfx-overlay-scanlines"></div>
<div id="rfx-stars"></div>
<div class="rfx-vignette"></div>
<p class="header">
<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi">Gemini</a>
</p>
<h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-4-rocky-linux-bhyve-vms'>f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</h1><br />
<br />
<span class='quote'>Published at 2025-04-04T23:21:01+03:00, last updated Fri 26 Dec 08:51:06 EET 2025</span><br />
<br />
<span>This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.</span><br />
<br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)</a><br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br />
<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br />
<br />
<a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br />
<br />
<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
<br />
<ul>
<li><a href='#f3s-kubernetes-with-freebsd---part-4-rocky-linux-bhyve-vms'>f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a></li>
<li>⇢ <a href='#introduction'>Introduction</a></li>
<li>⇢ <a href='#check-for-popcnt-cpu-support'>Check for <span class='inlinecode'>POPCNT</span> CPU support</a></li>
<li>⇢ <a href='#basic-bhyve-setup'>Basic Bhyve setup</a></li>
<li>⇢ <a href='#rocky-linux-vms'>Rocky Linux VMs</a></li>
<li>⇢ ⇢ <a href='#iso-download'>ISO download</a></li>
<li>⇢ ⇢ <a href='#vm-configuration'>VM configuration</a></li>
<li>⇢ ⇢ <a href='#vm-installation'>VM installation</a></li>
<li>⇢ ⇢ <a href='#increase-of-the-disk-image'>Increase of the disk image</a></li>
<li>⇢ ⇢ <a href='#connect-to-vnc'>Connect to VNC</a></li>
<li>⇢ <a href='#after-install'>After install</a></li>
<li>⇢ ⇢ <a href='#vm-auto-start-after-host-reboot'>VM auto-start after host reboot</a></li>
<li>⇢ ⇢ <a href='#static-ip-configuration'>Static IP configuration</a></li>
<li>⇢ ⇢ <a href='#permitting-root-login'>Permitting root login</a></li>
<li>⇢ ⇢ <a href='#install-latest-updates'>Install latest updates</a></li>
<li>⇢ <a href='#stress-testing-cpu'>Stress testing CPU</a></li>
<li>⇢ ⇢ <a href='#silly-freebsd-host-benchmark'>Silly FreeBSD host benchmark</a></li>
<li>⇢ ⇢ <a href='#silly-rocky-linux-vm--bhyve-benchmark'>Silly Rocky Linux VM @ Bhyve benchmark</a></li>
<li>⇢ ⇢ <a href='#silly-freebsd-vm--bhyve-benchmark'>Silly FreeBSD VM @ Bhyve benchmark</a></li>
<li>⇢ <a href='#benchmarking-with-ubench'>Benchmarking with <span class='inlinecode'>ubench</span></a></li>
<li>⇢ ⇢ <a href='#freebsd-host-ubench-benchmark'>FreeBSD host <span class='inlinecode'>ubench</span> benchmark</a></li>
<li>⇢ ⇢ <a href='#freebsd-vm--bhyve-ubench-benchmark'>FreeBSD VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</a></li>
<li>⇢ ⇢ <a href='#rocky-linux-vm--bhyve-ubench-benchmark'>Rocky Linux VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</a></li>
<li>⇢ <a href='#update-improving-disk-io-performance-for-etcd'>Update: Improving Disk I/O Performance for etcd</a></li>
<li>⇢ ⇢ <a href='#the-problem'>The Problem</a></li>
<li>⇢ ⇢ <a href='#the-solution-switch-to-nvme-emulation'>The Solution: Switch to NVMe Emulation</a></li>
<li>⇢ ⇢ <a href='#step-1-prepare-the-guest-os'>Step 1: Prepare the Guest OS</a></li>
<li>⇢ ⇢ <a href='#step-2-update-the-bhyve-configuration'>Step 2: Update the Bhyve Configuration</a></li>
<li>⇢ ⇢ <a href='#benchmark-results'>Benchmark Results</a></li>
<li>⇢ ⇢ <a href='#important-notes'>Important Notes</a></li>
<li>⇢ <a href='#conclusion'>Conclusion</a></li>
</ul><br />
<h2 style='display: inline' id='introduction'>Introduction</h2><br />
<br />
<span>In this blog post, we are going to install the Bhyve hypervisor.</span><br />
<br />
<span>The FreeBSD Bhyve hypervisor is a lightweight, modern hypervisor that enables virtualization on FreeBSD systems. Bhyve's strengths include its minimal overhead, which allows it to achieve near-native performance for virtual machines. It's efficient and lightweight, leveraging the capabilities of the FreeBSD operating system for performance and network management.</span><br />
<br />
<a class='textlink' href='https://wiki.freebsd.org/bhyve'>https://wiki.freebsd.org/bhyve</a><br />
<br />
<span>Bhyve supports running various guest operating systems, including FreeBSD, Linux, and Windows, on hardware platforms that support hardware virtualization extensions (such as Intel VT-x or AMD-V). In our case, we are going to virtualize Rocky Linux, which will later in this series be used to run k3s.</span><br />
<br />
<h2 style='display: inline' id='check-for-popcnt-cpu-support'>Check for <span class='inlinecode'>POPCNT</span> CPU support</h2><br />
<br />
<span>POPCNT is a CPU instruction that counts the number of set bits (ones) in a binary number. CPU virtualization and Bhyve support for the POPCNT instruction are important because guest operating systems utilize this instruction to perform various tasks more efficiently. If the host CPU supports POPCNT, Bhyve can pass this capability to virtual machines for better performance. Without POPCNT support, some applications might not run or perform sub-optimally in virtualized environments.</span><br />
<br />
<span>To check for <span class='inlinecode'>POPCNT</span> support, run:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> dmesg </font><font color="#F3E651">|</font><font color="#ff0000"> grep </font><font color="#bb00ff">'Features2=.*POPCNT'</font>
<font color="#ff0000"> </font><font color="#ff0000">Features2</font><font color="#F3E651">=</font><font color="#bb00ff">0x7ffafbbf</font><font color="#F3E651"><</font><font color="#ff0000">SSE3</font><font color="#F3E651">,</font><font color="#ff0000">PCLMULQDQ</font><font color="#F3E651">,</font><font color="#ff0000">DTES64</font><font color="#F3E651">,</font><font color="#ff0000">MON</font><font color="#F3E651">,</font><font color="#ff0000">DS_CPL</font><font color="#F3E651">,</font><font color="#ff0000">VMX</font><font color="#F3E651">,</font><font color="#ff0000">EST</font><font color="#F3E651">,</font><font color="#ff0000">TM2</font><font color="#F3E651">,</font><font color="#ff0000">SSSE3</font><font color="#F3E651">,</font><font color="#ff0000">SDBG</font><font color="#F3E651">,</font>
<font color="#ff0000"> FMA</font><font color="#F3E651">,</font><font color="#ff0000">CX16</font><font color="#F3E651">,</font><font color="#ff0000">xTPR</font><font color="#F3E651">,</font><font color="#ff0000">PDCM</font><font color="#F3E651">,</font><font color="#ff0000">PCID</font><font color="#F3E651">,</font><font color="#ff0000">SSE4</font><font color="#F3E651">.</font><font color="#bb00ff">1</font><font color="#F3E651">,</font><font color="#ff0000">SSE4</font><font color="#F3E651">.</font><font color="#bb00ff">2</font><font color="#F3E651">,</font><font color="#ff0000">x2APIC</font><font color="#F3E651">,</font><font color="#ff0000">MOVBE</font><font color="#F3E651">,</font><font color="#ff0000">POPCNT</font><font color="#F3E651">,</font><font color="#ff0000">TSCDLT</font><font color="#F3E651">,</font><font color="#ff0000">AESNI</font><font color="#F3E651">,</font><font color="#ff0000">XSAVE</font><font color="#F3E651">,</font>
<font color="#ff0000"> OSXSAVE</font><font color="#F3E651">,</font><font color="#ff0000">AVX</font><font color="#F3E651">,</font><font color="#ff0000">F16C</font><font color="#F3E651">,</font><font color="#ff0000">RDRAND</font><font color="#F3E651">></font>
</pre>
<br />
<span>So it's there! All good.</span><br />
<br />
<h2 style='display: inline' id='basic-bhyve-setup'>Basic Bhyve setup</h2><br />
<br />
<span>For managing the Bhyve VMs, we are using <span class='inlinecode'>vm-bhyve</span>, a tool not part of the FreeBSD operating system but available as a ready-to-use package. It eases VM management and reduces a lot of overhead. We also install the required package to make Bhyve work with the UEFI firmware.</span><br />
<br />
<a class='textlink' href='https://github.com/churchers/vm-bhyve'>https://github.com/churchers/vm-bhyve</a><br />
<br />
<span>The following commands are executed on all three hosts <span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span>, and <span class='inlinecode'>f2</span>, where <span class='inlinecode'>re0</span> is the name of the Ethernet interface (which may need to be adjusted if your hardware is different):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas pkg install vm-bhyve bhyve-firmware</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas sysrc </font><font color="#ff0000">vm_enable</font><font color="#F3E651">=</font><font color="#ff0000">YES</font>
<font color="#ff0000">vm_enable</font><font color="#F3E651">:</font><font color="#ff0000"> -</font><font color="#F3E651">></font><font color="#ff0000"> YES</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas sysrc </font><font color="#ff0000">vm_dir</font><font color="#F3E651">=</font><font color="#ff0000">zfs</font><font color="#F3E651">:</font><font color="#ff0000">zroot/bhyve</font>
<font color="#ff0000">vm_dir</font><font color="#F3E651">:</font><font color="#ff0000"> -</font><font color="#F3E651">></font><font color="#ff0000"> zfs</font><font color="#F3E651">:</font><font color="#ff0000">zroot/bhyve</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas zfs create zroot/bhyve</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm init</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm switch create public</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm switch add public re0</font>
</pre>
<br />
<span>Bhyve stores all its data in the <span class='inlinecode'>/bhyve</span> of the <span class='inlinecode'>zroot</span> ZFS pool:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> zfs list </font><font color="#F3E651">|</font><font color="#ff0000"> grep bhyve</font>
<font color="#ff0000">zroot/bhyve </font><font color="#bb00ff">1</font><font color="#F3E651">.</font><font color="#ff0000">74M 453G </font><font color="#bb00ff">1</font><font color="#F3E651">.</font><font color="#ff0000">74M /zroot/bhyve</font>
</pre>
<br />
<span>For convenience, we also create this symlink:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas ln -s /zroot/bhyve</font><font color="#F3E651">/</font><font color="#ff0000"> /bhyve</font>
</pre>
<br />
<span>Now, Bhyve is ready to rumble, but no VMs are there yet:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm list</font>
<font color="#ff0000">NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE</font>
</pre>
<br />
<h2 style='display: inline' id='rocky-linux-vms'>Rocky Linux VMs</h2><br />
<br />
<span>As guest VMs I decided to use Rocky Linux.</span><br />
<br />
<span>Using Rocky Linux 9 as a VM-based OS is beneficial primarily because of its long-term support and stable release cycle. This ensures a reliable environment that receives security updates and bug fixes for an extended period, reducing the need for frequent upgrades.</span><br />
<br />
<span>Rocky Linux is community-driven and aims to be fully compatible with enterprise Linux, making it a solid choice for consistency and performance in various deployment scenarios.</span><br />
<br />
<a class='textlink' href='https://rockylinux.org/'>https://rockylinux.org/</a><br />
<br />
<h3 style='display: inline' id='iso-download'>ISO download</h3><br />
<br />
<span>We're going to install the Rocky Linux from the latest minimal iso:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm iso </font><font color="#F3E651">\</font>
<font color="#ff0000"> https</font><font color="#F3E651">:</font><font color="#ff0000">//download</font><font color="#F3E651">.</font><font color="#ff0000">rockylinux</font><font color="#F3E651">.</font><font color="#ff0000">org/pub/rocky</font><font color="#F3E651">/</font><font color="#bb00ff">9</font><font color="#ff0000">/isos/x86_64/Rocky-</font><font color="#bb00ff">9.5</font><font color="#ff0000">-x86_64-minimal</font><font color="#F3E651">.</font><font color="#ff0000">iso</font>
<font color="#ff0000">/zroot/bhyve</font><font color="#F3E651">/.</font><font color="#ff0000">iso/Rocky-</font><font color="#bb00ff">9.5</font><font color="#ff0000">-x86_64-minimal</font><font color="#F3E651">.</font><font color="#ff0000">iso </font><font color="#bb00ff">1808</font><font color="#ff0000"> MB </font><font color="#bb00ff">4780</font><font color="#ff0000"> kBps 06m28s</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm create rocky</font>
</pre>
<br />
<h3 style='display: inline' id='vm-configuration'>VM configuration</h3><br />
<br />
<span>The default Bhyve VM configuration looks like this now:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> cat rocky</font><font color="#F3E651">.</font><font color="#ff0000">conf</font>
<font color="#ff0000">loader</font><font color="#F3E651">=</font><font color="#bb00ff">"bhyveload"</font>
<font color="#ff0000">cpu</font><font color="#F3E651">=</font><font color="#bb00ff">1</font>
<font color="#ff0000">memory</font><font color="#F3E651">=</font><font color="#ff0000">256M</font>
<font color="#ff0000">network0_type</font><font color="#F3E651">=</font><font color="#bb00ff">"virtio-net"</font>
<font color="#ff0000">network0_switch</font><font color="#F3E651">=</font><font color="#bb00ff">"public"</font>
<font color="#ff0000">disk0_type</font><font color="#F3E651">=</font><font color="#bb00ff">"virtio-blk"</font>
<font color="#ff0000">disk0_name</font><font color="#F3E651">=</font><font color="#bb00ff">"disk0.img"</font>
<font color="#ff0000">uuid</font><font color="#F3E651">=</font><font color="#bb00ff">"1c4655ac-c828-11ef-a920-e8ff1ed71ca0"</font>
<font color="#ff0000">network0_mac</font><font color="#F3E651">=</font><font color="#bb00ff">"58:9c:fc:0d:13:3f"</font>
</pre>
<br />
<span>The <span class='inlinecode'>uuid</span> and the <span class='inlinecode'>network0_mac</span> differ for each of the three VMs (the ones being installed on <span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span> and <span class='inlinecode'>f2</span>).</span><br />
<br />
<span>But to make Rocky Linux boot it (plus some other adjustments, e.g. as we intend to run the majority of the workload in the k3s cluster running on those Linux VMs, we give them beefy specs like 4 CPU cores and 14GB RAM). So we run <span class='inlinecode'>doas vm configure rocky</span> and modified it to:</span><br />
<br />
<pre>
guest="linux"
loader="uefi"
uefi_vars="yes"
cpu=4
memory=14G
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"
graphics="yes"
graphics_vga=io
uuid="1c45400b-c828-11ef-8871-e8ff1ed71cac"
network0_mac="58:9c:fc:0d:13:3f"
</pre>
<br />
<h3 style='display: inline' id='vm-installation'>VM installation</h3><br />
<br />
<span>To start the installer from the downloaded ISO, we run:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm install rocky Rocky-</font><font color="#bb00ff">9.5</font><font color="#ff0000">-x86_64-minimal</font><font color="#F3E651">.</font><font color="#ff0000">iso</font>
<font color="#ff0000">Starting rocky</font>
<font color="#ff0000"> </font><font color="#F3E651">*</font><font color="#ff0000"> found guest </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> /zroot/bhyve/rocky</font>
<font color="#ff0000"> </font><font color="#F3E651">*</font><font color="#ff0000"> booting</font><font color="#F3E651">...</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm list</font>
<font color="#ff0000">NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE</font>
<font color="#ff0000">rocky default uefi </font><font color="#bb00ff">4</font><font color="#ff0000"> 14G </font><font color="#bb00ff">0.0</font><font color="#F3E651">.</font><font color="#bb00ff">0.0</font><font color="#F3E651">:</font><font color="#bb00ff">5900</font><font color="#ff0000"> No Locked </font><font color="#F3E651">(</font><font color="#ff0000">f0</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font><font color="#F3E651">)</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> doas sockstat -</font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#F3E651">|</font><font color="#ff0000"> grep </font><font color="#bb00ff">5900</font>
<font color="#ff0000">root bhyve </font><font color="#bb00ff">6079</font><font color="#ff0000"> </font><font color="#bb00ff">8</font><font color="#ff0000"> tcp4 </font><font color="#F3E651">*:</font><font color="#bb00ff">5900</font><font color="#ff0000"> </font><font color="#F3E651">*:*</font>
</pre>
<br />
<span>Port 5900 now also opens for VNC connections, so I connected it with a VNC client and ran through the installation dialogues. This could be done unattended or more automated, but there are only three VMs to install, and the automation doesn't seem worth it as we do it only once a year or less often.</span><br />
<br />
<h3 style='display: inline' id='increase-of-the-disk-image'>Increase of the disk image</h3><br />
<br />
<span>By default, the VM disk image is only 20G, which is a bit small for our purposes, so we have to stop the VMs again, run <span class='inlinecode'>truncate</span> on the image file to enlarge them to 100G, and restart the installation:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm stop rocky</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> doas truncate -s 100G disk0</font><font color="#F3E651">.</font><font color="#ff0000">img</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm install rocky Rocky-</font><font color="#bb00ff">9.5</font><font color="#ff0000">-x86_64-minimal</font><font color="#F3E651">.</font><font color="#ff0000">iso</font>
</pre>
<br />
<h3 style='display: inline' id='connect-to-vnc'>Connect to VNC</h3><br />
<br />
<span>For the installation, I opened the VNC client on my Fedora laptop (GNOME comes with a simple VNC client) and manually ran through the base installation for each of the VMs. Again, I am sure this could have been automated a bit more, but there were just three VMs, and it wasn't worth the effort. The three VNC addresses of the VMs were <span class='inlinecode'>vnc://f0:5900</span>, <span class='inlinecode'>vnc://f1:5900</span>, and <span class='inlinecode'>vnc://f2:5900</span>.</span><br />
<br />
<a href='./f3s-kubernetes-with-freebsd-part-4/1.png'><img src='./f3s-kubernetes-with-freebsd-part-4/1.png' /></a><br />
<br />
<a href='./f3s-kubernetes-with-freebsd-part-4/2.png'><img src='./f3s-kubernetes-with-freebsd-part-4/2.png' /></a><br />
<br />
<span>I primarily selected the default settings (auto partitioning on the 100GB drive and a root user password). After the installation, the VMs were rebooted.</span><br />
<br />
<a href='./f3s-kubernetes-with-freebsd-part-4/3.png'><img src='./f3s-kubernetes-with-freebsd-part-4/3.png' /></a><br />
<br />
<a href='./f3s-kubernetes-with-freebsd-part-4/4.png'><img src='./f3s-kubernetes-with-freebsd-part-4/4.png' /></a><br />
<br />
<h2 style='display: inline' id='after-install'>After install</h2><br />
<br />
<span>We perform the following steps for all three VMs. In the following, the examples are all executed on <span class='inlinecode'>f0</span> (the VM <span class='inlinecode'>r0</span> running on <span class='inlinecode'>f0</span>):</span><br />
<br />
<h3 style='display: inline' id='vm-auto-start-after-host-reboot'>VM auto-start after host reboot</h3><br />
<br />
<span>To automatically start the VM on the servers, we add the following to the <span class='inlinecode'>rc.conf</span> on the FreeBSD hosts:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> cat </font><font color="#F3E651"><<</font><font color="#ff0000">END </font><font color="#F3E651">|</font><font color="#ff0000"> doas tee -a /etc/rc</font><font color="#F3E651">.</font><font color="#ff0000">conf</font>
<font color="#ff0000">vm_list</font><font color="#F3E651">=</font><font color="#bb00ff">"rocky"</font>
<font color="#ff0000">vm_delay</font><font color="#F3E651">=</font><font color="#bb00ff">"5"</font>
</pre>
<br />
<span>The <span class='inlinecode'>vm_delay</span> isn't really required. It is used to wait 5 seconds before starting each VM, but there is currently only one VM per host. Maybe later, when there are more, this will be useful. After adding, there's now a <span class='inlinecode'>Yes</span> indicator in the <span class='inlinecode'>AUTO</span> column.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm list</font>
<font color="#ff0000">NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE</font>
<font color="#ff0000">rocky default uefi </font><font color="#bb00ff">4</font><font color="#ff0000"> 14G </font><font color="#bb00ff">0.0</font><font color="#F3E651">.</font><font color="#bb00ff">0.0</font><font color="#F3E651">:</font><font color="#bb00ff">5900</font><font color="#ff0000"> Yes </font><font color="#F3E651">[</font><font color="#bb00ff">1</font><font color="#F3E651">]</font><font color="#ff0000"> Running </font><font color="#F3E651">(</font><font color="#bb00ff">2063</font><font color="#F3E651">)</font>
</pre>
<br />
<h3 style='display: inline' id='static-ip-configuration'>Static IP configuration</h3><br />
<br />
<span>After that, we change the network configuration of the VMs to be static (from DHCP) here. As per the previous post of this series, the three FreeBSD hosts were already in my <span class='inlinecode'>/etc/hosts</span> file:</span><br />
<br />
<pre>
192.168.1.130 f0 f0.lan f0.lan.buetow.org
192.168.1.131 f1 f1.lan f1.lan.buetow.org
192.168.1.132 f2 f2.lan f2.lan.buetow.org
</pre>
<br />
<span>For the Rocky VMs, we add those to the FreeBSD host systems as well:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:</font><font color="#ff0000">/bhyve/rocky </font><font color="#F3E651">%</font><font color="#ff0000"> cat </font><font color="#F3E651"><<</font><font color="#ff0000">END </font><font color="#F3E651">|</font><font color="#ff0000"> doas tee -a /etc/hosts</font>
<font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.120</font><font color="#ff0000"> r0 r0</font><font color="#F3E651">.</font><font color="#ff0000">lan r0</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.121</font><font color="#ff0000"> r1 r1</font><font color="#F3E651">.</font><font color="#ff0000">lan r1</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.122</font><font color="#ff0000"> r2 r2</font><font color="#F3E651">.</font><font color="#ff0000">lan r2</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#ff0000">END</font>
</pre>
<br />
<span>And we configure the IPs accordingly on the VMs themselves by opening a root shell via SSH to the VMs and entering the following commands on each of the VMs:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> nmcli connection modify enp0s5 ipv4</font><font color="#F3E651">.</font><font color="#ff0000">address </font><font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.120</font><font color="#F3E651">/</font><font color="#bb00ff">24</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> nmcli connection modify enp0s5 ipv4</font><font color="#F3E651">.</font><font color="#ff0000">gateway </font><font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.1</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> nmcli connection modify enp0s5 ipv4</font><font color="#F3E651">.</font><font color="#ff0000">DNS </font><font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.1</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> nmcli connection modify enp0s5 ipv4</font><font color="#F3E651">.</font><font color="#ff0000">method manual</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> nmcli connection down enp0s5</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> nmcli connection up enp0s5</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> hostnamectl set-hostname r0</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> cat </font><font color="#F3E651"><<</font><font color="#ff0000">END </font><font color="#F3E651">>></font><font color="#ff0000">/etc/hosts</font>
<font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.120</font><font color="#ff0000"> r0 r0</font><font color="#F3E651">.</font><font color="#ff0000">lan r0</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.121</font><font color="#ff0000"> r1 r1</font><font color="#F3E651">.</font><font color="#ff0000">lan r1</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">1.122</font><font color="#ff0000"> r2 r2</font><font color="#F3E651">.</font><font color="#ff0000">lan r2</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#ff0000">END</font>
</pre>
<br />
<span>Whereas:</span><br />
<br />
<ul>
<li><span class='inlinecode'>192.168.1.120</span> is the IP of the VM itself (here: <span class='inlinecode'>r0.lan.buetow.org</span>)</li>
<li><span class='inlinecode'>192.168.1.1</span> is the address of my home router, which also does DNS.</li>
</ul><br />
<h3 style='display: inline' id='permitting-root-login'>Permitting root login</h3><br />
<br />
<span>As these VMs aren't directly reachable via SSH from the internet, we enable <span class='inlinecode'>root</span> login by adding a line with <span class='inlinecode'>PermitRootLogin yes</span> to <span class='inlinecode'>/etc/sshd/sshd_config</span>.</span><br />
<br />
<span>Once done, we reboot the VM by running <span class='inlinecode'>reboot</span> inside the VM to test whether everything was configured and persisted correctly.</span><br />
<br />
<span>After reboot, we copy a public key over. E.g. I did this from my Laptop as follows:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#bb00ff">1</font><font color="#ff0000"> </font><font color="#bb00ff">2</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">do</font></b><font color="#ff0000"> ssh-copy-id root@r</font><font color="#ff0000">$i</font><font color="#F3E651">.</font><font color="#ff0000">lan</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">done</font></b>
</pre>
<br />
<span>Then, we edit the <span class='inlinecode'>/etc/ssh/sshd_config</span> file again on all three VMs and configure <span class='inlinecode'>PasswordAuthentication no</span> to only allow SSH key authentication from now on.</span><br />
<br />
<h3 style='display: inline' id='install-latest-updates'>Install latest updates</h3><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> dnf update</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> reboot</font>
</pre>
<br />
<h2 style='display: inline' id='stress-testing-cpu'>Stress testing CPU</h2><br />
<br />
<span>The aim is to prove that bhyve VMs are CPU efficient. As I could not find an off-the-shelf benchmarking tool available in the same version for FreeBSD as well as for Rocky Linux 9, I wrote my own silly CPU benchmarking tool in Go:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">package</font></b><font color="#ff0000"> main</font>
<b><font color="#ffffff">import</font></b><font color="#ff0000"> </font><font color="#bb00ff">"testing"</font>
<b><font color="#ffffff">func</font></b><font color="#ff0000"> </font><font color="#7bc710">BenchmarkCPUSilly1</font><font color="#F3E651">(</font><font color="#ff0000">b </font><font color="#F3E651">*</font><font color="#ff0000">testing</font><font color="#F3E651">.</font><font color="#ff0000">B</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#F3E651">;</font><font color="#ff0000"> i </font><font color="#F3E651"><</font><font color="#ff0000"> b</font><font color="#F3E651">.</font><font color="#ff0000">N</font><font color="#F3E651">;</font><font color="#ff0000"> i</font><font color="#F3E651">++</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> _ </font><font color="#F3E651">=</font><font color="#ff0000"> i </font><font color="#F3E651">*</font><font color="#ff0000"> i</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#F3E651">}</font>
<b><font color="#ffffff">func</font></b><font color="#ff0000"> </font><font color="#7bc710">BenchmarkCPUSilly2</font><font color="#F3E651">(</font><font color="#ff0000">b </font><font color="#F3E651">*</font><font color="#ff0000">testing</font><font color="#F3E651">.</font><font color="#ff0000">B</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">var</font></b><font color="#ff0000"> sillyResult </font><b><font color="#F35E1E">float64</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#F3E651">;</font><font color="#ff0000"> i </font><font color="#F3E651"><</font><font color="#ff0000"> b</font><font color="#F3E651">.</font><font color="#ff0000">N</font><font color="#F3E651">;</font><font color="#ff0000"> i</font><font color="#F3E651">++</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> sillyResult </font><font color="#F3E651">+=</font><font color="#ff0000"> </font><b><font color="#F35E1E">float64</font></b><font color="#F3E651">(</font><font color="#ff0000">i</font><font color="#F3E651">)</font>
<font color="#ff0000"> sillyResult </font><font color="#F3E651">*=</font><font color="#ff0000"> </font><b><font color="#F35E1E">float64</font></b><font color="#F3E651">(</font><font color="#ff0000">i</font><font color="#F3E651">)</font>
<font color="#ff0000"> divisor </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><b><font color="#F35E1E">float64</font></b><font color="#F3E651">(</font><font color="#ff0000">i</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">+</font><font color="#ff0000"> </font><font color="#bb00ff">1</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> divisor </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> sillyResult </font><font color="#F3E651">/=</font><font color="#ff0000"> divisor</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> _ </font><font color="#F3E651">=</font><font color="#ff0000"> sillyResult </font><i><font color="#ababab">// to avoid compiler optimization</font></i>
<font color="#F3E651">}</font>
</pre>
<br />
<span>You can find the repository here:</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/sillybench'>https://codeberg.org/snonux/sillybench</a><br />
<br />
<h3 style='display: inline' id='silly-freebsd-host-benchmark'>Silly FreeBSD host benchmark</h3><br />
<br />
<span>To install it on FreeBSD, we run:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas pkg install git go</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> mkdir </font><font color="#F3E651">~</font><font color="#ff0000">/git </font><font color="#F3E651">&&</font><font color="#ff0000"> cd </font><font color="#F3E651">~</font><font color="#ff0000">/git </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> git clone https</font><font color="#F3E651">:</font><font color="#ff0000">//codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/sillybench </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> cd sillybench</font>
</pre>
<br />
<span>And to run it:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000">/git/sillybench </font><font color="#F3E651">%</font><font color="#ff0000"> go version</font>
<font color="#ff0000">go version go1</font><font color="#F3E651">.</font><font color="#bb00ff">24.1</font><font color="#ff0000"> freebsd/amd</font><font color="#bb00ff">64</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000">/git/sillybench </font><font color="#F3E651">%</font><font color="#ff0000"> go </font><b><font color="#ffffff">test</font></b><font color="#ff0000"> -bench</font><font color="#F3E651">=.</font>
<font color="#ff0000">goos</font><font color="#F3E651">:</font><font color="#ff0000"> freebsd</font>
<font color="#ff0000">goarch</font><font color="#F3E651">:</font><font color="#ff0000"> amd64</font>
<font color="#ff0000">pkg</font><font color="#F3E651">:</font><font color="#ff0000"> codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/sillybench</font>
<font color="#ff0000">cpu</font><font color="#F3E651">:</font><font color="#ff0000"> Intel</font><font color="#F3E651">(</font><font color="#ff0000">R</font><font color="#F3E651">)</font><font color="#ff0000"> N100</font>
<font color="#ff0000">BenchmarkCPUSilly1-</font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">1000000000</font><font color="#ff0000"> </font><font color="#bb00ff">0.4022</font><font color="#ff0000"> ns/op</font>
<font color="#ff0000">BenchmarkCPUSilly2-</font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">1000000000</font><font color="#ff0000"> </font><font color="#bb00ff">0.4027</font><font color="#ff0000"> ns/op</font>
<font color="#ff0000">PASS</font>
<font color="#ff0000">ok codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/sillybench </font><font color="#bb00ff">0</font><font color="#F3E651">.</font><font color="#ff0000">891s</font>
</pre>
<br />
<h3 style='display: inline' id='silly-rocky-linux-vm--bhyve-benchmark'>Silly Rocky Linux VM @ Bhyve benchmark</h3><br />
<br />
<span>OK, let's compare this with the Rocky Linux VM running on Bhyve:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># dnf install golang git</font></i>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># mkdir ~/git && cd ~/git && \</font></i>
<font color="#ff0000"> git clone https</font><font color="#F3E651">:</font><font color="#ff0000">//codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/sillybench </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> cd sillybench</font>
</pre>
<br />
<span>And to run it:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 sillybench</font><font color="#F3E651">]</font><i><font color="#ababab"># go version</font></i>
<font color="#ff0000">go version go1</font><font color="#F3E651">.</font><font color="#bb00ff">22.9</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">Red Hat </font><font color="#bb00ff">1.22</font><font color="#F3E651">.</font><font color="#bb00ff">9</font><font color="#ff0000">-</font><font color="#bb00ff">2</font><font color="#F3E651">.</font><font color="#ff0000">el9_5</font><font color="#F3E651">)</font><font color="#ff0000"> linux/amd</font><font color="#bb00ff">64</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 sillybench</font><font color="#F3E651">]</font><i><font color="#ababab"># go test -bench=.</font></i>
<font color="#ff0000">goos</font><font color="#F3E651">:</font><font color="#ff0000"> linux</font>
<font color="#ff0000">goarch</font><font color="#F3E651">:</font><font color="#ff0000"> amd64</font>
<font color="#ff0000">pkg</font><font color="#F3E651">:</font><font color="#ff0000"> codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/sillybench</font>
<font color="#ff0000">cpu</font><font color="#F3E651">:</font><font color="#ff0000"> Intel</font><font color="#F3E651">(</font><font color="#ff0000">R</font><font color="#F3E651">)</font><font color="#ff0000"> N100</font>
<font color="#ff0000">BenchmarkCPUSilly1-</font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">1000000000</font><font color="#ff0000"> </font><font color="#bb00ff">0.4347</font><font color="#ff0000"> ns/op</font>
<font color="#ff0000">BenchmarkCPUSilly2-</font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">1000000000</font><font color="#ff0000"> </font><font color="#bb00ff">0.4345</font><font color="#ff0000"> ns/op</font>
</pre>
<br />
<span>The Linux benchmark is slightly slower than the FreeBSD one. The Go version is also a bit older. I tried the same with the up-to-date version of Go (1.24.x) with similar results. There could be a slight Bhyve overhead, or FreeBSD is just slightly more efficient in this benchmark. Overall, this shows that Bhyve performs excellently.</span><br />
<br />
<h3 style='display: inline' id='silly-freebsd-vm--bhyve-benchmark'>Silly FreeBSD VM @ Bhyve benchmark</h3><br />
<br />
<span>But as I am curious and don't want to compare apples with bananas, I decided to install a FreeBSD Bhyve VM to run the same silly benchmark in it. I am not going through the details of how to install a FreeBSD Bhyve VM here; you can easily look it up in the documentation.</span><br />
<br />
<span>But here are the results running the same silly benchmark in a FreeBSD Bhyve VM with the same FreeBSD and Go versions as the host system (I have the VM 4 vCPUs and 14GB of RAM; the benchmark won't use as many CPUs (and memory) anyway):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">root@freebsd</font><font color="#F3E651">:~</font><font color="#ff0000">/git/sillybench </font><i><font color="#ababab"># go test -bench=.</font></i>
<font color="#ff0000">goos</font><font color="#F3E651">:</font><font color="#ff0000"> freebsd</font>
<font color="#ff0000">goarch</font><font color="#F3E651">:</font><font color="#ff0000"> amd64</font>
<font color="#ff0000">pkg</font><font color="#F3E651">:</font><font color="#ff0000"> codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/sillybench</font>
<font color="#ff0000">cpu</font><font color="#F3E651">:</font><font color="#ff0000"> Intel</font><font color="#F3E651">(</font><font color="#ff0000">R</font><font color="#F3E651">)</font><font color="#ff0000"> N100</font>
<font color="#ff0000">BenchmarkCPUSilly1 </font><font color="#bb00ff">1000000000</font><font color="#ff0000"> </font><font color="#bb00ff">0.4273</font><font color="#ff0000"> ns/op</font>
<font color="#ff0000">BenchmarkCPUSilly2 </font><font color="#bb00ff">1000000000</font><font color="#ff0000"> </font><font color="#bb00ff">0.4286</font><font color="#ff0000"> ns/op</font>
<font color="#ff0000">PASS</font>
<font color="#ff0000">ok codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/sillybench </font><font color="#bb00ff">0</font><font color="#F3E651">.</font><font color="#ff0000">949s</font>
</pre>
<br />
<span>It's a bit better than Linux! I am sure that this is not really a scientific benchmark, so take the results with a grain of salt!</span><br />
<br />
<h2 style='display: inline' id='benchmarking-with-ubench'>Benchmarking with <span class='inlinecode'>ubench</span></h2><br />
<br />
<span>Let's run another, more sophisticated benchmark using <span class='inlinecode'>ubench</span>, the Unix Benchmark Utility available for FreeBSD. It was installed by simply running <span class='inlinecode'>doas pkg install ubench</span>. It can benchmark CPU and memory performance. Here, we limit it to one CPU for the first run with <span class='inlinecode'>-s</span>, and then let it run at full speed (using all available CPUs in parallel) in the second run.</span><br />
<br />
<h3 style='display: inline' id='freebsd-host-ubench-benchmark'>FreeBSD host <span class='inlinecode'>ubench</span> benchmark</h3><br />
<br />
<span>Single CPU:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas ubench -s </font><font color="#bb00ff">1</font>
<font color="#ff0000">Unix Benchmark Utility v</font><font color="#F3E651">.</font><font color="#bb00ff">0.3</font>
<font color="#ff0000">Copyright </font><font color="#F3E651">(</font><font color="#ff0000">C</font><font color="#F3E651">)</font><font color="#ff0000"> July</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">1999</font><font color="#ff0000"> PhysTech</font><font color="#F3E651">,</font><font color="#ff0000"> Inc</font><font color="#F3E651">.</font>
<font color="#ff0000">Author</font><font color="#F3E651">:</font><font color="#ff0000"> Sergei Viznyuk </font><font color="#F3E651"><</font><font color="#ff0000">sv@phystech</font><font color="#F3E651">.</font><font color="#ff0000">com</font><font color="#F3E651">></font>
<font color="#ff0000">http</font><font color="#F3E651">:</font><font color="#ff0000">//www</font><font color="#F3E651">.</font><font color="#ff0000">phystech</font><font color="#F3E651">.</font><font color="#ff0000">com/download/ubench</font><font color="#F3E651">.</font><font color="#ff0000">html</font>
<font color="#ff0000">FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> GENERIC amd64</font>
<font color="#ff0000">Ubench Single CPU</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">671010</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#bb00ff">0</font><font color="#F3E651">.</font><font color="#ff0000">40s</font><font color="#F3E651">)</font>
<font color="#ff0000">Ubench Single MEM</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">1705237</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#bb00ff">0</font><font color="#F3E651">.</font><font color="#ff0000">48s</font><font color="#F3E651">)</font>
<font color="#ff0000">-----------------------------------</font>
<font color="#ff0000">Ubench Single AVG</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">1188123</font>
</pre>
<br />
<span>All CPUs (with all Bhyve VMs stopped):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas ubench</font>
<font color="#ff0000">Unix Benchmark Utility v</font><font color="#F3E651">.</font><font color="#bb00ff">0.3</font>
<font color="#ff0000">Copyright </font><font color="#F3E651">(</font><font color="#ff0000">C</font><font color="#F3E651">)</font><font color="#ff0000"> July</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">1999</font><font color="#ff0000"> PhysTech</font><font color="#F3E651">,</font><font color="#ff0000"> Inc</font><font color="#F3E651">.</font>
<font color="#ff0000">Author</font><font color="#F3E651">:</font><font color="#ff0000"> Sergei Viznyuk </font><font color="#F3E651"><</font><font color="#ff0000">sv@phystech</font><font color="#F3E651">.</font><font color="#ff0000">com</font><font color="#F3E651">></font>
<font color="#ff0000">http</font><font color="#F3E651">:</font><font color="#ff0000">//www</font><font color="#F3E651">.</font><font color="#ff0000">phystech</font><font color="#F3E651">.</font><font color="#ff0000">com/download/ubench</font><font color="#F3E651">.</font><font color="#ff0000">html</font>
<font color="#ff0000">FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> GENERIC amd64</font>
<font color="#ff0000">Ubench CPU</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">2660220</font>
<font color="#ff0000">Ubench MEM</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">3095182</font>
<font color="#ff0000">--------------------</font>
<font color="#ff0000">Ubench AVG</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">2877701</font>
</pre>
<br />
<h3 style='display: inline' id='freebsd-vm--bhyve-ubench-benchmark'>FreeBSD VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</h3><br />
<br />
<span>Single CPU:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">root@freebsd</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><i><font color="#ababab"># ubench -s 1</font></i>
<font color="#ff0000">Unix Benchmark Utility v</font><font color="#F3E651">.</font><font color="#bb00ff">0.3</font>
<font color="#ff0000">Copyright </font><font color="#F3E651">(</font><font color="#ff0000">C</font><font color="#F3E651">)</font><font color="#ff0000"> July</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">1999</font><font color="#ff0000"> PhysTech</font><font color="#F3E651">,</font><font color="#ff0000"> Inc</font><font color="#F3E651">.</font>
<font color="#ff0000">Author</font><font color="#F3E651">:</font><font color="#ff0000"> Sergei Viznyuk </font><font color="#F3E651"><</font><font color="#ff0000">sv@phystech</font><font color="#F3E651">.</font><font color="#ff0000">com</font><font color="#F3E651">></font>
<font color="#ff0000">http</font><font color="#F3E651">:</font><font color="#ff0000">//www</font><font color="#F3E651">.</font><font color="#ff0000">phystech</font><font color="#F3E651">.</font><font color="#ff0000">com/download/ubench</font><font color="#F3E651">.</font><font color="#ff0000">html</font>
<font color="#ff0000">FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> GENERIC amd64</font>
<font color="#ff0000">Ubench Single CPU</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">672792</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#bb00ff">0</font><font color="#F3E651">.</font><font color="#ff0000">40s</font><font color="#F3E651">)</font>
<font color="#ff0000">Ubench Single MEM</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">852757</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#bb00ff">0</font><font color="#F3E651">.</font><font color="#ff0000">48s</font><font color="#F3E651">)</font>
<font color="#ff0000">-----------------------------------</font>
<font color="#ff0000">Ubench Single AVG</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">762774</font>
</pre>
<br />
<span>Wow, the CPU in the VM was a tiny bit faster than on the host! So this was probably just a glitch in the matrix. Memory seems slower, though.</span><br />
<br />
<span>All CPUs:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">root@freebsd</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><i><font color="#ababab"># ubench</font></i>
<font color="#ff0000">Unix Benchmark Utility v</font><font color="#F3E651">.</font><font color="#bb00ff">0.3</font>
<font color="#ff0000">Copyright </font><font color="#F3E651">(</font><font color="#ff0000">C</font><font color="#F3E651">)</font><font color="#ff0000"> July</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">1999</font><font color="#ff0000"> PhysTech</font><font color="#F3E651">,</font><font color="#ff0000"> Inc</font><font color="#F3E651">.</font>
<font color="#ff0000">Author</font><font color="#F3E651">:</font><font color="#ff0000"> Sergei Viznyuk </font><font color="#F3E651"><</font><font color="#ff0000">sv@phystech</font><font color="#F3E651">.</font><font color="#ff0000">com</font><font color="#F3E651">></font>
<font color="#ff0000">http</font><font color="#F3E651">:</font><font color="#ff0000">//www</font><font color="#F3E651">.</font><font color="#ff0000">phystech</font><font color="#F3E651">.</font><font color="#ff0000">com/download/ubench</font><font color="#F3E651">.</font><font color="#ff0000">html</font>
<font color="#ff0000">FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> FreeBSD </font><font color="#bb00ff">14.2</font><font color="#ff0000">-RELEASE-p</font><font color="#bb00ff">1</font><font color="#ff0000"> GENERIC amd64</font>
<font color="#ff0000">Ubench CPU</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">2652857</font>
<font color="#ff0000">swap_pager</font><font color="#F3E651">:</font><font color="#ff0000"> out of swap space</font>
<font color="#ff0000">swp_pager_getswapspace</font><font color="#F3E651">(</font><font color="#bb00ff">27</font><font color="#F3E651">):</font><font color="#ff0000"> failed</font>
<font color="#ff0000">swap_pager</font><font color="#F3E651">:</font><font color="#ff0000"> out of swap space</font>
<font color="#ff0000">swp_pager_getswapspace</font><font color="#F3E651">(</font><font color="#bb00ff">18</font><font color="#F3E651">):</font><font color="#ff0000"> failed</font>
<font color="#ff0000">Apr </font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">23</font><font color="#F3E651">:</font><font color="#bb00ff">02</font><font color="#F3E651">:</font><font color="#bb00ff">43</font><font color="#ff0000"> freebsd kernel</font><font color="#F3E651">:</font><font color="#ff0000"> pid </font><font color="#bb00ff">862</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">ubench</font><font color="#F3E651">),</font><font color="#ff0000"> jid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> uid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> was killed</font><font color="#F3E651">:</font><font color="#ff0000"> failed to reclaim memory</font>
<font color="#ff0000">swp_pager_getswapspace</font><font color="#F3E651">(</font><font color="#bb00ff">6</font><font color="#F3E651">):</font><font color="#ff0000"> failed</font>
<font color="#ff0000">Apr </font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">23</font><font color="#F3E651">:</font><font color="#bb00ff">02</font><font color="#F3E651">:</font><font color="#bb00ff">46</font><font color="#ff0000"> freebsd kernel</font><font color="#F3E651">:</font><font color="#ff0000"> pid </font><font color="#bb00ff">863</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">ubench</font><font color="#F3E651">),</font><font color="#ff0000"> jid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> uid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> was killed</font><font color="#F3E651">:</font><font color="#ff0000"> failed to reclaim memory</font>
<font color="#ff0000">Apr </font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">23</font><font color="#F3E651">:</font><font color="#bb00ff">02</font><font color="#F3E651">:</font><font color="#bb00ff">47</font><font color="#ff0000"> freebsd kernel</font><font color="#F3E651">:</font><font color="#ff0000"> pid </font><font color="#bb00ff">864</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">ubench</font><font color="#F3E651">),</font><font color="#ff0000"> jid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> uid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> was killed</font><font color="#F3E651">:</font><font color="#ff0000"> failed to reclaim memory</font>
<font color="#ff0000">Apr </font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">23</font><font color="#F3E651">:</font><font color="#bb00ff">02</font><font color="#F3E651">:</font><font color="#bb00ff">48</font><font color="#ff0000"> freebsd kernel</font><font color="#F3E651">:</font><font color="#ff0000"> pid </font><font color="#bb00ff">865</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">ubench</font><font color="#F3E651">),</font><font color="#ff0000"> jid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> uid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> was killed</font><font color="#F3E651">:</font><font color="#ff0000"> failed to reclaim memory</font>
<font color="#ff0000">Apr </font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">23</font><font color="#F3E651">:</font><font color="#bb00ff">02</font><font color="#F3E651">:</font><font color="#bb00ff">49</font><font color="#ff0000"> freebsd kernel</font><font color="#F3E651">:</font><font color="#ff0000"> pid </font><font color="#bb00ff">861</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">ubench</font><font color="#F3E651">),</font><font color="#ff0000"> jid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> uid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> was killed</font><font color="#F3E651">:</font><font color="#ff0000"> failed to reclaim memory</font>
<font color="#ff0000">Apr </font><font color="#bb00ff">4</font><font color="#ff0000"> </font><font color="#bb00ff">23</font><font color="#F3E651">:</font><font color="#bb00ff">02</font><font color="#F3E651">:</font><font color="#bb00ff">51</font><font color="#ff0000"> freebsd kernel</font><font color="#F3E651">:</font><font color="#ff0000"> pid </font><font color="#bb00ff">839</font><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">ubench</font><font color="#F3E651">),</font><font color="#ff0000"> jid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> uid </font><font color="#bb00ff">0</font><font color="#F3E651">,</font><font color="#ff0000"> was killed</font><font color="#F3E651">:</font><font color="#ff0000"> failed to reclaim memory</font>
</pre>
<br />
<span>The multi-CPU benchmark in the Bhyve VM ran with almost identical results to the FreeBSD host system. However, the memory benchmark failed with out-of-swap space errors. I am unsure why, as the VM has 14GB RAM, but I am not investigating further.</span><br />
<br />
<span>Also, during the benchmark, I noticed the <span class='inlinecode'>bhyve</span> process on the host was constantly using 399% of the CPU (all 4 CPUs).</span><br />
<br />
<pre>
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
7449 root 14 20 0 14G 78M kqread 2 2:12 399.81% bhyve
</pre>
<br />
<span>Overall, Bhyve has a small overhead, but the CPU performance difference is negligible. The FreeBSD host is slightly faster than the FreeBSD VM running on Bhyve, but the difference is small enough for our use cases. The memory benchmark seems slightly off, but I'm not sure whether to trust it, especially due to the swap errors. Does <span class='inlinecode'>ubench</span>'s memory benchmark use swap space for the memory test? That wouldn't make sense and might explain the difference to some degree, though. Do you have any ideas?</span><br />
<br />
<h3 style='display: inline' id='rocky-linux-vm--bhyve-ubench-benchmark'>Rocky Linux VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</h3><br />
<br />
<span>Unfortunately, I wasn't able to find <span class='inlinecode'>ubench</span> in any of the Rocky Linux repositories. So, I skipped this test.</span><br />
<br />
<h2 style='display: inline' id='update-improving-disk-io-performance-for-etcd'>Update: Improving Disk I/O Performance for etcd</h2><br />
<br />
<span class='quote'>Updated: Fri 26 Dec 08:51:23 EET 2025</span><br />
<br />
<span>After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default <span class='inlinecode'>virtio-blk</span> disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.</span><br />
<br />
<h3 style='display: inline' id='the-problem'>The Problem</h3><br />
<br />
<span>The k3s logs showed etcd struggling with disk I/O:</span><br />
<br />
<pre>
{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
</pre>
<br />
<span>A simple sync write benchmark confirmed the issue:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync</font></i>
<font color="#bb00ff">8192000</font><font color="#ff0000"> bytes copied</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">31.7058</font><font color="#ff0000"> s</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">258</font><font color="#ff0000"> kB/s</font>
</pre>
<br />
<h3 style='display: inline' id='the-solution-switch-to-nvme-emulation'>The Solution: Switch to NVMe Emulation</h3><br />
<br />
<span>Bhyve's NVMe emulation provides significantly better I/O performance than <span class='inlinecode'>virtio-blk</span>.</span><br />
<br />
<h3 style='display: inline' id='step-1-prepare-the-guest-os'>Step 1: Prepare the Guest OS</h3><br />
<br />
<span>Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># cat > /etc/dracut.conf.d/nvme.conf << EOF</font></i>
<font color="#ff0000">add_drivers</font><font color="#F3E651">+=</font><font color="#bb00ff">" nvme nvme_core "</font>
<font color="#ff0000">hostonly</font><font color="#F3E651">=</font><font color="#ff0000">no</font>
<font color="#ff0000">EOF</font>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf</font></i>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># dracut -f</font></i>
<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># shutdown -h now</font></i>
</pre>
<br />
<span>The <span class='inlinecode'>hostonly=no</span> setting ensures the initramfs includes drivers for hardware not currently present. The <span class='inlinecode'>use_devicesfile = 0</span> tells LVM to scan all block devices rather than only those recorded in <span class='inlinecode'>/etc/lvm/devices/system.devices</span> - this is important because the device path changes from <span class='inlinecode'>/dev/vda</span> to <span class='inlinecode'>/dev/nvme0n1</span>.</span><br />
<br />
<h3 style='display: inline' id='step-2-update-the-bhyve-configuration'>Step 2: Update the Bhyve Configuration</h3><br />
<br />
<span>On the FreeBSD host, update the VM configuration to use NVMe:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm stop rocky</font>
<font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm configure rocky</font>
</pre>
<br />
<span>Change <span class='inlinecode'>disk0_type</span> from <span class='inlinecode'>virtio-blk</span> to <span class='inlinecode'>nvme</span>:</span><br />
<br />
<pre>
disk0_type="nvme"
</pre>
<br />
<span>Then start the VM:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas vm start rocky</font>
</pre>
<br />
<h3 style='display: inline' id='benchmark-results'>Benchmark Results</h3><br />
<br />
<span>After switching to NVMe emulation, the sync write performance improved dramatically:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync</font></i>
<font color="#bb00ff">8192000</font><font color="#ff0000"> bytes copied</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">0.330718</font><font color="#ff0000"> s</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">24.8</font><font color="#ff0000"> MB/s</font>
</pre>
<br />
<span>That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).</span><br />
<br />
<span>The etcd metrics also showed healthy fsync latencies:</span><br />
<br />
<pre>
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
</pre>
<br />
<span>Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.</span><br />
<br />
<h3 style='display: inline' id='important-notes'>Important Notes</h3><br />
<br />
<ul>
<li>Do NOT use <span class='inlinecode'>disk0_opts="nocache,direct"</span> with NVMe emulation - in my testing this actually made performance worse.</li>
<li>The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot.</li>
<li>LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.</li>
</ul><br />
<h2 style='display: inline' id='conclusion'>Conclusion</h2><br />
<br />
<span>Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.</span><br />
<br />
<span>Future uses (out of scope for this blog series) would be additional VMs for different workloads. For example, how about a Windows or NetBSD VM to tinker with?</span><br />
<br />
<span>This flexibility is great for keeping options open and managing different workloads without overcomplicating things. Overall, it's a nice setup for getting the most out of my hardware and keeping things running smoothly.</span><br />
<br />
<span>Read the next post of this series:</span><br />
<br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<br />
<span>Other *BSD-related posts:</span><br />
<br />
<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br />
<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br />
<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)</a><br />
<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span></span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
<p class="footer">
Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
<br />
Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a>
</p>
</body>
</html>
|
