1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>f3s: Kubernetes with FreeBSD - Part 7: First pod deployments</title>
<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
<link rel="stylesheet" href="../style.css" />
<link rel="stylesheet" href="style-override.css" />
</head>
<body>
<p class="header">
<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-kubernetes-with-freebsd-part-7.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-kubernetes-with-freebsd-part-7.gmi">Gemini</a>
</p>
<h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-7-first-pod-deployments'>f3s: Kubernetes with FreeBSD - Part 7: First pod deployments</h1><br />
<br />
<span>This is the seventh blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.</span><br />
<br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
<br />
<a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br />
<br />
<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
<br />
<ul>
<li><a href='#f3s-kubernetes-with-freebsd---part-7-first-pod-deployments'>f3s: Kubernetes with FreeBSD - Part 7: First pod deployments</a></li>
<li>⇢ <a href='#introduction'>Introduction</a></li>
<li>⇢ <a href='#updating'>Updating</a></li>
<li>⇢ <a href='#installing-k3s'>Installing k3s</a></li>
<li>⇢ ⇢ <a href='#generating-k3stoken-and-starting-first-k3s-node'>Generating <span class='inlinecode'>K3S_TOKEN</span> and starting first k3s node</a></li>
<li>⇢ ⇢ <a href='#adding-the-remaining-nodes-to-the-cluster'>Adding the remaining nodes to the cluster</a></li>
<li>⇢ <a href='#test-deployments'>Test deployments</a></li>
<li>⇢ ⇢ <a href='#test-deployment-to-kubernetes'>Test deployment to Kubernetes</a></li>
<li>⇢ ⇢ <a href='#test-deployment-with-persistent-volume-claim'>Test deployment with persistent volume claim</a></li>
<li>⇢ <a href='#make-it-accessible-from-the-public-internet'>Make it accessible from the public internet</a></li>
<li>⇢ <a href='#failure-test'>Failure test</a></li>
</ul><br />
<h2 style='display: inline' id='introduction'>Introduction</h2><br />
<br />
<h2 style='display: inline' id='updating'>Updating</h2><br />
<br />
<span>On all three Rocky Linux 9 boxes <span class='inlinecode'>r0</span>, <span class='inlinecode'>r1</span>, and <span class='inlinecode'>r2</span>:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>dnf update -y
reboot
</pre>
<br />
<span>On the FreeBSD hosts, upgrading from FreeBSD 14.2 to 14.3-RELEASE, running this on all three hosts <span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span> and <span class='inlinecode'>f2</span>:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas freebsd-update fetch
paul@f0:~ % doas freebsd-update install
paul@f0:~ % doas reboot
.
.
.
paul@f0:~ % doas freebsd-update -r <font color="#000000">14.3</font>-RELEASE upgrade
paul@f0:~ % doas freebsd-update install
paul@f0:~ % doas freebsd-update install
paul@f0:~ % doas reboot
.
.
.
paul@f0:~ % doas freebsd-update install
paul@f0:~ % doas pkg update
paul@f0:~ % doas pkg upgrade
paul@f0:~ % doas reboot
.
.
.
paul@f0:~ % uname -a
FreeBSD f0.lan.buetow.org <font color="#000000">14.3</font>-RELEASE FreeBSD <font color="#000000">14.3</font>-RELEASE
releng/<font color="#000000">14.3</font>-n<font color="#000000">271432</font>-8c9ce319fef7 GENERIC amd64
</pre>
<br />
<h2 style='display: inline' id='installing-k3s'>Installing k3s</h2><br />
<br />
<h3 style='display: inline' id='generating-k3stoken-and-starting-first-k3s-node'>Generating <span class='inlinecode'>K3S_TOKEN</span> and starting first k3s node</h3><br />
<br />
<span>Generating the k3s token on my Fedora Laptop with <span class='inlinecode'>pwgen -n 32</span> and selected one. And then on all 3 <span class='inlinecode'>r</span> hosts (replace SECRET_TOKEN with the actual secret!! before running the following command) run:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>[root@r0 ~]<i><font color="silver"># echo -n SECRET_TOKEN > ~/.k3s_token</font></i>
</pre>
<br />
<span>The following steps are also documented on the k3s website:</span><br />
<br />
<a class='textlink' href='https://docs.k3s.io/datastore/ha-embedded'>https://docs.k3s.io/datastore/ha-embedded</a><br />
<br />
<span>So on <span class='inlinecode'>r0</span> we run:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>[root@r0 ~]<i><font color="silver"># curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \</font></i>
sh -s - server --cluster-init --tls-san=r0.wg0.wan.buetow.org
[INFO] Finding release <b><u><font color="#000000">for</font></u></b> channel stable
[INFO] Using v1.<font color="#000000">32.6</font>+k3s1 as release
.
.
.
[INFO] systemd: Starting k3s
</pre>
<br />
<h3 style='display: inline' id='adding-the-remaining-nodes-to-the-cluster'>Adding the remaining nodes to the cluster</h3><br />
<br />
<span>And we run on the other two nodes <span class='inlinecode'>r1</span> and <span class='inlinecode'>r2</span>:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>[root@r1 ~]<i><font color="silver"># curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \</font></i>
sh -s - server --server https://r<font color="#000000">0</font>.wg0.wan.buetow.org:<font color="#000000">6443</font> \
--tls-san=r1.wg0.wan.buetow.org
[root@r2 ~]<i><font color="silver"># curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \</font></i>
sh -s - server --server https://r<font color="#000000">0</font>.wg0.wan.buetow.org:<font color="#000000">6443</font> \
--tls-san=r2.wg0.wan.buetow.org
.
.
.
</pre>
<br />
<span>Once done, we've got a 3 node Kubernetes cluster control plane:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>[root@r0 ~]<i><font color="silver"># kubectl get nodes</font></i>
NAME STATUS ROLES AGE VERSION
r0.lan.buetow.org Ready control-plane,etcd,master 4m44s v1.<font color="#000000">32.6</font>+k3s1
r1.lan.buetow.org Ready control-plane,etcd,master 3m13s v1.<font color="#000000">32.6</font>+k3s1
r2.lan.buetow.org Ready control-plane,etcd,master 30s v1.<font color="#000000">32.6</font>+k3s1
[root@r0 ~]<i><font color="silver"># kubectl get pods --all-namespaces</font></i>
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5688667fd4-fs2jj <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 5m27s
kube-system helm-install-traefik-crd-f9hgd <font color="#000000">0</font>/<font color="#000000">1</font> Completed <font color="#000000">0</font> 5m27s
kube-system helm-install-traefik-zqqqk <font color="#000000">0</font>/<font color="#000000">1</font> Completed <font color="#000000">2</font> 5m27s
kube-system local-path-provisioner-774c6665dc-jqlnc <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 5m27s
kube-system metrics-server-6f4c6675d5-5xpmp <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 5m27s
kube-system svclb-traefik-411cec5b-cdp2l <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 78s
kube-system svclb-traefik-411cec5b-f625r <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 4m58s
kube-system svclb-traefik-411cec5b-twrd<font color="#000000">7</font> <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 4m2s
kube-system traefik-c98fdf6fb-lt6fx <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 4m58s
</pre>
<br />
<span>In order to connect with <span class='inlinecode'>kubect</span> from my Fedora Laptop, I had to copy <span class='inlinecode'>/etc/rancher/k3s/k3s.yaml</span> from <span class='inlinecode'>r0</span> to <span class='inlinecode'>~/.kube/config</span> and then replace the value of the server field with <span class='inlinecode'>r0.lan.buetow.org</span>. kubectl can now manage the cluster. Note this step has to be repeated when we want to connect to another node of the cluster (e.g. when <span class='inlinecode'>r0</span> is down).</span><br />
<br />
<h2 style='display: inline' id='test-deployments'>Test deployments</h2><br />
<br />
<h3 style='display: inline' id='test-deployment-to-kubernetes'>Test deployment to Kubernetes</h3><br />
<br />
<span>Let's create a test namespace:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ kubectl create namespace <b><u><font color="#000000">test</font></u></b>
namespace/test created
> ~ kubectl get namespaces
NAME STATUS AGE
default Active 6h11m
kube-node-lease Active 6h11m
kube-public Active 6h11m
kube-system Active 6h11m
<b><u><font color="#000000">test</font></u></b> Active 5s
> ~ kubectl config set-context --current --namespace=<b><u><font color="#000000">test</font></u></b>
Context <font color="#808080">"default"</font> modified.
</pre>
<br />
<span>And let's also create an apache test pod:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ cat <<END > apache-deployment.yaml
<i><font color="silver"># Apache HTTP Server Deployment</font></i>
apiVersion: apps/v<font color="#000000">1</font>
kind: Deployment
metadata:
name: apache-deployment
spec:
replicas: <font color="#000000">1</font>
selector:
matchLabels:
app: apache
template:
metadata:
labels:
app: apache
spec:
containers:
- name: apache
image: httpd:latest
ports:
<i><font color="silver"># Container port where Apache listens</font></i>
- containerPort: <font color="#000000">80</font>
END
> ~ kubectl apply -f apache-deployment.yaml
deployment.apps/apache-deployment created
> ~ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/apache-deployment-5fd955856f-4pjmf <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 7s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/apache-deployment <font color="#000000">1</font>/<font color="#000000">1</font> <font color="#000000">1</font> <font color="#000000">1</font> 7s
NAME DESIRED CURRENT READY AGE
replicaset.apps/apache-deployment-5fd955856f <font color="#000000">1</font> <font color="#000000">1</font> <font color="#000000">1</font> 7s
</pre>
<br />
<span>Let's also create a service: </span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ cat <<END > apache-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: apache
name: apache-service
spec:
ports:
- name: web
port: <font color="#000000">80</font>
protocol: TCP
<i><font color="silver"># Expose port 80 on the service</font></i>
targetPort: <font color="#000000">80</font>
selector:
<i><font color="silver"># Link this service to pods with the label app=apache</font></i>
app: apache
END
> ~ kubectl apply -f apache-service.yaml
service/apache-service created
> ~ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
apache-service ClusterIP <font color="#000000">10.43</font>.<font color="#000000">249.165</font> <none> <font color="#000000">80</font>/TCP 4s
</pre>
<br />
<span>And also an ingress:</span><br />
<br />
<span class='quote'>Note: I've modified the hosts listed in this example after I've published this blog post. This is to ensure that there aren't any bots scarping it.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ cat <<END > apache-ingress.yaml
apiVersion: networking.k8s.io/v<font color="#000000">1</font>
kind: Ingress
metadata:
name: apache-ingress
namespace: <b><u><font color="#000000">test</font></u></b>
annotations:
spec.ingressClassName: traefik
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: f3s.foo.zone
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-service
port:
number: <font color="#000000">80</font>
- host: standby.f3s.foo.zone
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-service
port:
number: <font color="#000000">80</font>
- host: www.f3s.foo.zone
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-service
port:
number: <font color="#000000">80</font>
END
> ~ kubectl apply -f apache-ingress.yaml
ingress.networking.k8s.io/apache-ingress created
> ~ kubectl describe ingress
Name: apache-ingress
Labels: <none>
Namespace: <b><u><font color="#000000">test</font></u></b>
Address: <font color="#000000">192.168</font>.<font color="#000000">1.120</font>,<font color="#000000">192.168</font>.<font color="#000000">1.121</font>,<font color="#000000">192.168</font>.<font color="#000000">1.122</font>
Ingress Class: traefik
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
f3s.foo.zone
/ apache-service:<font color="#000000">80</font> (<font color="#000000">10.42</font>.<font color="#000000">1.11</font>:<font color="#000000">80</font>)
standby.f3s.foo.zone
/ apache-service:<font color="#000000">80</font> (<font color="#000000">10.42</font>.<font color="#000000">1.11</font>:<font color="#000000">80</font>)
www.f3s.foo.zone
/ apache-service:<font color="#000000">80</font> (<font color="#000000">10.42</font>.<font color="#000000">1.11</font>:<font color="#000000">80</font>)
Annotations: spec.ingressClassName: traefik
traefik.ingress.kubernetes.io/router.entrypoints: web
Events: <none>
</pre>
<br />
<span>Notes: </span><br />
<br />
<ul>
<li>I've modified the ingress hosts after I'd published this blog post. This is to ensure that there aren't any bots scarping it.</li>
<li>In the ingress we use plain http (web) for the traefik rule, as all the "production" traefic will routed through a WireGuard tunnel anyway as we will see later.</li>
</ul><br />
<span>So let's test the Apache webserver through the ingress rule:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ curl -H <font color="#808080">"Host: www.f3s.foo.zone"</font> http://r<font color="#000000">0</font>.lan.buetow.org:<font color="#000000">80</font>
<html><body><h1>It works!</h<font color="#000000">1</font>></body></html>
</pre>
<br />
<h3 style='display: inline' id='test-deployment-with-persistent-volume-claim'>Test deployment with persistent volume claim</h3><br />
<br />
<span>So let's modify the Apache example to serve the <span class='inlinecode'>htdocs</span> directory from the NFS share we created in the previous blog post. We are using the following manifests. The majority of the manifests are the same as before, except for the persistent volume claim and the volume mount in the Apache deployment.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ cat <<END > apache-deployment.yaml
<i><font color="silver"># Apache HTTP Server Deployment</font></i>
apiVersion: apps/v<font color="#000000">1</font>
kind: Deployment
metadata:
name: apache-deployment
namespace: <b><u><font color="#000000">test</font></u></b>
spec:
replicas: <font color="#000000">2</font>
selector:
matchLabels:
app: apache
template:
metadata:
labels:
app: apache
spec:
containers:
- name: apache
image: httpd:latest
ports:
<i><font color="silver"># Container port where Apache listens</font></i>
- containerPort: <font color="#000000">80</font>
readinessProbe:
httpGet:
path: /
port: <font color="#000000">80</font>
initialDelaySeconds: <font color="#000000">5</font>
periodSeconds: <font color="#000000">10</font>
livenessProbe:
httpGet:
path: /
port: <font color="#000000">80</font>
initialDelaySeconds: <font color="#000000">15</font>
periodSeconds: <font color="#000000">10</font>
volumeMounts:
- name: apache-htdocs
mountPath: /usr/local/apache<font color="#000000">2</font>/htdocs/
volumes:
- name: apache-htdocs
persistentVolumeClaim:
claimName: example-apache-pvc
END
> ~ cat <<END > apache-ingress.yaml
apiVersion: networking.k8s.io/v<font color="#000000">1</font>
kind: Ingress
metadata:
name: apache-ingress
namespace: <b><u><font color="#000000">test</font></u></b>
annotations:
spec.ingressClassName: traefik
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: f3s.buetow.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-service
port:
number: <font color="#000000">80</font>
- host: standby.f3s.buetow.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-service
port:
number: <font color="#000000">80</font>
- host: www.f3s.buetow.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-service
port:
number: <font color="#000000">80</font>
END
> ~ cat <<END > apache-persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-apache-pv
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/nfs/k3svolumes/example-apache-volume-claim
<b><u><font color="#000000">type</font></u></b>: Directory
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: example-apache-pvc
namespace: <b><u><font color="#000000">test</font></u></b>
spec:
storageClassName: <font color="#808080">""</font>
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
END
> ~ cat <<END > apache-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: apache
name: apache-service
namespace: <b><u><font color="#000000">test</font></u></b>
spec:
ports:
- name: web
port: <font color="#000000">80</font>
protocol: TCP
<i><font color="silver"># Expose port 80 on the service</font></i>
targetPort: <font color="#000000">80</font>
selector:
<i><font color="silver"># Link this service to pods with the label app=apache</font></i>
app: apache
END
</pre>
<br />
<span>And let's apply the manifests:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ kubectl apply -f apache-persistent-volume.yaml
kubectl apply -f apache-service.yaml
kubectl apply -f apache-deployment.yaml
kubectl apply -f apache-ingress.yaml
</pre>
<br />
<span>So looking at the deployment, it failed now, as the directory doesn't exist yet on the NFS share (note, we also increased the replica count to 2, so in case one node goes down, that there is already a replica running on another node for faster failover):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
apache-deployment-5b96bd6b6b-fv2jx <font color="#000000">0</font>/<font color="#000000">1</font> ContainerCreating <font color="#000000">0</font> 9m15s
apache-deployment-5b96bd6b6b-ax2ji <font color="#000000">0</font>/<font color="#000000">1</font> ContainerCreating <font color="#000000">0</font> 9m15s
> ~ kubectl describe pod apache-deployment-5b96bd6b6b-fv2jx | tail -n <font color="#000000">5</font>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m34s default-scheduler Successfully
assigned test/apache-deployment-5b96bd6b6b-fv2jx to r2.lan.buetow.org
Warning FailedMount 80s (x12 over 9m34s) kubelet MountVolume.SetUp
failed <b><u><font color="#000000">for</font></u></b> volume <font color="#808080">"example-apache-pv"</font> : hostPath <b><u><font color="#000000">type</font></u></b> check failed:
/data/nfs/k3svolumes/example-apache is not a directory
</pre>
<br />
<span>This is on purpose! We need to create the directory on the NFS share first, so let's do that (e.g. on <span class='inlinecode'>r0</span>):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>[root@r0 ~]<i><font color="silver"># mkdir /data/nfs/k3svolumes/example-apache-volume-claim/</font></i>
[root@r0 ~ ] cat <<END > /data/nfs/k3svolumes/example-apache-volume-claim/index.html
<!DOCTYPE html>
<html>
<head>
<title>Hello, it works</title>
</head>
<body>
<h1>Hello, it works!</h<font color="#000000">1</font>>
<p>This site is served via a PVC!</p>
</body>
</html>
END
</pre>
<br />
<span>The <span class='inlinecode'>index.html</span> file was also created to serve content along the way. After deleting the pod, it recreates itself, and the volume mounts correctly:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ kubectl delete pod apache-deployment-5b96bd6b6b-fv2jx
> ~ curl -H <font color="#808080">"Host: www.f3s.buetow.org"</font> http://r<font color="#000000">0</font>.lan.buetow.org:<font color="#000000">80</font>
<!DOCTYPE html>
<html>
<head>
<title>Hello, it works</title>
</head>
<body>
<h1>Hello, it works!</h<font color="#000000">1</font>>
<p>This site is served via a PVC!</p>
</body>
</html>
</pre>
<br />
<h2 style='display: inline' id='make-it-accessible-from-the-public-internet'>Make it accessible from the public internet</h2><br />
<br />
<span>Next, this should be made accessible through the public internet via the <span class='inlinecode'>www.f3s.foo.zone</span> hosts. As a reminder, refer back to part 1 of this series and review the section titled "OpenBSD/relayd to the rescue for external connectivity":</span><br />
<br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<br />
<span class='quote'>All apps should be reachable through the internet (e.g., from my phone or computer when travelling). For external connectivity and TLS management, I've got two OpenBSD VMs (one hosted by OpenBSD Amsterdam and another hosted by Hetzner) handling public-facing services like DNS, relaying traffic, and automating Let's Encrypt certificates.</span><br />
<br />
<span class='quote'>All of this (every Linux VM to every OpenBSD box) will be connected via WireGuard tunnels, keeping everything private and secure. There will be 6 WireGuard tunnels (3 k3s nodes times two OpenBSD VMs).</span><br />
<br />
<span class='quote'>So, when I want to access a service running in k3s, I will hit an external DNS endpoint (with the authoritative DNS servers being the OpenBSD boxes). The DNS will resolve to the master OpenBSD VM (see my KISS highly-available with OpenBSD blog post), and from there, the relayd process (with a Let's Encrypt certificate—see my Let's Encrypt with OpenBSD and Rex blog post) will accept the TCP connection and forward it through the WireGuard tunnel to a reachable node port of one of the k3s nodes, thus serving the traffic.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>> ~ curl https://f3s.foo.zone
<html><body><h1>It works!</h<font color="#000000">1</font>></body></html>
> ~ curl https://www.f3s.foo.zone
<html><body><h1>It works!</h<font color="#000000">1</font>></body></html>
> ~ curl https://standby.f3s.foo.zone
<html><body><h1>It works!</h<font color="#000000">1</font>></body></html>
</pre>
<br />
<h2 style='display: inline' id='failure-test'>Failure test</h2><br />
<br />
<span>Shutting down <span class='inlinecode'>f0</span> and let NFS failing over for the Apache content.</span><br />
<br />
<br />
<span>TODO: openbsd relayd config</span><br />
<span>TODO: registry howto</span><br />
<span>TODO: anki-droid deployment</span><br />
<span>TODO: include k9s screenshot</span><br />
<span>TODO: include a diagram again?</span><br />
<span>TODO: increase replica of traefik to 2, persist config surviving reboots</span><br />
<span>TODO: fix check-mounts script (mountpoint command and stale mounts... differentiate better)</span><br />
<span>TODO: remove traefic metal lb pods? persist the change? </span><br />
<br />
<span>Other *BSD-related posts:</span><br />
<br />
<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span></span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
<br />
<br />
<span>Note, that I've modified the hosts after I'd published this blog post. This is to ensure that there aren't any bots scarping it.</span><br />
<p class="footer">
Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
</p>
</body>
</html>
|
