From e8dcdd5de24a27e25e46f017f4a6e8861e90bfb5 Mon Sep 17 00:00:00 2001
From: Paul Buetow <paul@buetow.org>
Date: Thu, 5 Feb 2026 17:39:15 +0200
Subject: Update content for html

---
 ...5-12-07-f3s-kubernetes-with-freebsd-part-8.html | 281 ++++++++++++++++++++
 gemfeed/atom.xml                                   | 283 ++++++++++++++++++++-
 2 files changed, 563 insertions(+), 1 deletion(-)

(limited to 'gemfeed')

diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
index c38ac8fa..6d617b16 100644
--- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
+++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
@@ -33,6 +33,7 @@
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-8-observability'>f3s: Kubernetes with FreeBSD - Part 8: Observability</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#update-lan-ingress-support-february-2026'>Update: LAN Ingress Support (February 2026)</a></li>
 <li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#persistent-storage-recap'>Persistent storage recap</a></li>
 <li>⇢ <a href='#the-monitoring-namespace'>The monitoring namespace</a></li>
@@ -40,6 +41,7 @@
 <li>⇢ ⇢ <a href='#prerequisites'>Prerequisites</a></li>
 <li>⇢ ⇢ <a href='#deploying-with-the-justfile'>Deploying with the Justfile</a></li>
 <li>⇢ ⇢ <a href='#exposing-grafana-via-ingress'>Exposing Grafana via ingress</a></li>
+<li>⇢ ⇢ <a href='#exposing-services-via-lan-ingress'>Exposing services via LAN ingress</a></li>
 <li>⇢ <a href='#installing-loki-and-alloy'>Installing Loki and Alloy</a></li>
 <li>⇢ ⇢ <a href='#prerequisites'>Prerequisites</a></li>
 <li>⇢ ⇢ <a href='#deploying-loki-and-alloy'>Deploying Loki and Alloy</a></li>
@@ -77,6 +79,10 @@
 <br />
 <a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
 <br />
+<h2 style='display: inline' id='update-lan-ingress-support-february-2026'>Update: LAN Ingress Support (February 2026)</h2><br />
+<br />
+<span>Update (2026-02-05):** This blog post has been updated to include a new section on exposing services via LAN ingress. The original blog post focused on external access through OpenBSD edge relays. The new section documents how to:</span><br />
+<br />
 <h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
 <br />
 <span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
@@ -252,6 +258,281 @@ prometheus-kube-prometheus-prometheus   ClusterIP   <font color="#000000">10.43<
 <br />
 <a href='./f3s-kubernetes-with-freebsd-part-8/grafana-dashboard.png'><img alt='Grafana dashboard showing cluster metrics' title='Grafana dashboard showing cluster metrics' src='./f3s-kubernetes-with-freebsd-part-8/grafana-dashboard.png' /></a><br />
 <br />
+<h3 style='display: inline' id='exposing-services-via-lan-ingress'>Exposing services via LAN ingress</h3><br />
+<br />
+<span>In addition to external access through the OpenBSD relays, services can also be exposed on the local network using LAN-specific ingresses. This is useful for accessing services from within the home network without going through the internet, reducing latency and providing an alternative path if the external relays are unavailable.</span><br />
+<br />
+<span>The LAN ingress architecture leverages the existing FreeBSD CARP (Common Address Redundancy Protocol) failover infrastructure that&#39;s already in place for NFS-over-TLS (see Part 5). Instead of deploying MetalLB or another LoadBalancer implementation, we reuse the CARP virtual IP (<span class='inlinecode'>192.168.1.138</span>) by adding HTTP/HTTPS forwarding alongside the existing stunnel service on port 2323.</span><br />
+<br />
+<span>*Architecture overview*:</span><br />
+<br />
+<span>The LAN access path differs from external access:</span><br />
+<br />
+<span>**External access (*.f3s.foo.zone):**</span><br />
+<pre>
+Internet → OpenBSD relayd (TLS termination, Let&#39;s Encrypt)
+        → WireGuard tunnel
+        → k3s Traefik :80 (HTTP)
+        → Service
+</pre>
+<br />
+<span>**LAN access (*.f3s.lan.foo.zone):**</span><br />
+<pre>
+LAN → FreeBSD CARP VIP (192.168.1.138)
+    → FreeBSD relayd (TCP forwarding)
+    → k3s Traefik :443 (TLS termination, cert-manager)
+    → Service
+</pre>
+<br />
+<span>The key architectural decisions:</span><br />
+<br />
+<ul>
+<li>FreeBSD <span class='inlinecode'>relayd</span> performs pure TCP forwarding (Layer 4) for ports 80 and 443, not TLS termination</li>
+<li>Traefik inside k3s handles TLS offloading using certificates from cert-manager</li>
+<li>Self-signed CA for LAN domains (no external dependencies)</li>
+<li>CARP provides automatic failover between f0 and f1</li>
+<li>No code changes to applications—just add a LAN ingress resource</li>
+</ul><br />
+<span>*Installing cert-manager*:</span><br />
+<br />
+<span>First, install cert-manager to handle certificate lifecycle management for LAN services. The installation is automated with a Justfile:</span><br />
+<br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/cert-manager'>codeberg.org/snonux/conf/f3s/cert-manager</a><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ cd conf/f3s/cert-manager
+$ just install
+kubectl apply -f cert-manager.yaml
+<i><font color="silver"># ... cert-manager CRDs and resources created ...</font></i>
+kubectl apply -f self-signed-issuer.yaml
+clusterissuer.cert-manager.io/selfsigned-issuer created
+clusterissuer.cert-manager.io/selfsigned-ca-issuer created
+kubectl apply -f ca-certificate.yaml
+certificate.cert-manager.io/selfsigned-ca created
+kubectl apply -f wildcard-certificate.yaml
+certificate.cert-manager.io/f3s-lan-wildcard created
+</pre>
+<br />
+<span>This creates:</span><br />
+<br />
+<ul>
+<li>A self-signed ClusterIssuer</li>
+<li>A CA certificate (<span class='inlinecode'>f3s-lan-ca</span>) valid for 10 years</li>
+<li>A CA-signed ClusterIssuer</li>
+<li>A wildcard certificate (<span class='inlinecode'>*.f3s.lan.foo.zone</span>) valid for 90 days with automatic renewal</li>
+</ul><br />
+<span>Verify the certificates:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get certificate -n cert-manager
+NAME               READY   SECRET                 AGE
+f3s-lan-wildcard   True    f3s-lan-tls            5m
+selfsigned-ca      True    selfsigned-ca-secret   5m
+</pre>
+<br />
+<span>The wildcard certificate (<span class='inlinecode'>f3s-lan-tls</span>) needs to be copied to any namespace that uses it:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \
+    sed <font color="#808080">'s/namespace: cert-manager/namespace: services/'</font> | \
+    kubectl apply -f -
+</pre>
+<br />
+<span>*Configuring FreeBSD relayd for LAN access*:</span><br />
+<br />
+<span>On both FreeBSD hosts (f0, f1), install and configure <span class='inlinecode'>relayd</span> for TCP forwarding:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas pkg install -y relayd
+</pre>
+<br />
+<span>Create <span class='inlinecode'>/usr/local/etc/relayd.conf</span>:</span><br />
+<br />
+<pre>
+# k3s nodes backend table
+table &lt;k3s_nodes&gt; { 192.168.1.120 192.168.1.121 192.168.1.122 }
+
+# TCP forwarding to Traefik (no TLS termination)
+relay "lan_http" {
+    listen on 192.168.1.138 port 80
+    forward to &lt;k3s_nodes&gt; port 80 check tcp
+}
+
+relay "lan_https" {
+    listen on 192.168.1.138 port 443
+    forward to &lt;k3s_nodes&gt; port 443 check tcp
+}
+</pre>
+<br />
+<span>Note: The IP addresses <span class='inlinecode'>192.168.1.120-122</span> are the LAN IPs of the k3s nodes (r0, r1, r2), not their WireGuard IPs. FreeBSD <span class='inlinecode'>relayd</span> requires PF (Packet Filter) to be enabled. Create a minimal <span class='inlinecode'>/etc/pf.conf</span>:</span><br />
+<br />
+<pre>
+# Basic PF rules for relayd
+set skip on lo0
+pass in quick
+pass out quick
+</pre>
+<br />
+<span>Enable PF and relayd:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES
+paul@f0:~ % doas service pf start
+paul@f0:~ % doas service pflog start
+paul@f0:~ % doas service relayd start
+</pre>
+<br />
+<span>Verify <span class='inlinecode'>relayd</span> is listening on the CARP VIP:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas sockstat -<font color="#000000">4</font> -l | grep <font color="#000000">192.168</font>.<font color="#000000">1.138</font>
+_relayd  relayd   <font color="#000000">2903</font>  <font color="#000000">11</font>  tcp4   <font color="#000000">192.168</font>.<font color="#000000">1.138</font>:<font color="#000000">80</font>      *:*
+_relayd  relayd   <font color="#000000">2903</font>  <font color="#000000">12</font>  tcp4   <font color="#000000">192.168</font>.<font color="#000000">1.138</font>:<font color="#000000">443</font>     *:*
+</pre>
+<br />
+<span>Repeat the same configuration on f1. Both hosts will run <span class='inlinecode'>relayd</span> listening on the CARP VIP, but only the CARP MASTER will respond to traffic. When failover occurs, the new MASTER takes over seamlessly.</span><br />
+<br />
+<span>*Adding LAN ingress to services*:</span><br />
+<br />
+<span>To expose a service on the LAN, add a second Ingress resource to its Helm chart. Here&#39;s an example for Grafana:</span><br />
+<br />
+<pre>
+---
+# LAN Ingress for grafana.f3s.lan.foo.zone
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: grafana-ingress-lan
+  namespace: monitoring
+  annotations:
+    spec.ingressClassName: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
+spec:
+  tls:
+    - hosts:
+        - grafana.f3s.lan.foo.zone
+      secretName: f3s-lan-tls
+  rules:
+    - host: grafana.f3s.lan.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: prometheus-grafana
+                port:
+                  number: 80
+</pre>
+<br />
+<span>Key points:</span><br />
+<br />
+<ul>
+<li>Use <span class='inlinecode'>web,websecure</span> entrypoints (both HTTP and HTTPS)</li>
+<li>Reference the <span class='inlinecode'>f3s-lan-tls</span> secret in the <span class='inlinecode'>tls</span> section</li>
+<li>Use <span class='inlinecode'>.f3s.lan.foo.zone</span> subdomain pattern</li>
+<li>Same backend service as the external ingress</li>
+</ul><br />
+<span>Apply the ingress and test:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl apply -f grafana-ingress-lan.yaml
+ingress.networking.k8s.io/grafana-ingress-lan created
+
+$ curl -k https://grafana.f3s.lan.foo.zone
+HTTP/<font color="#000000">2</font> <font color="#000000">302</font> 
+location: /login
+</pre>
+<br />
+<span>*Client-side DNS and CA setup*:</span><br />
+<br />
+<span>To access LAN services, clients need DNS entries and must trust the self-signed CA.</span><br />
+<br />
+<span>Add DNS entries to <span class='inlinecode'>/etc/hosts</span> on your laptop:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ sudo tee -a /etc/hosts &lt;&lt; <font color="#808080">'EOF'</font>
+<i><font color="silver"># f3s LAN services</font></i>
+<font color="#000000">192.168</font>.<font color="#000000">1.138</font>  grafana.f3s.lan.foo.zone
+<font color="#000000">192.168</font>.<font color="#000000">1.138</font>  navidrome.f3s.lan.foo.zone
+EOF
+</pre>
+<br />
+<span>The CARP VIP <span class='inlinecode'>192.168.1.138</span> provides high availability—traffic automatically fails over to the backup host if the master goes down.</span><br />
+<br />
+<span>Export the self-signed CA certificate:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get secret selfsigned-ca-secret -n cert-manager -o jsonpath=<font color="#808080">'{.data.ca</font>\.<font color="#808080">crt}'</font> | \
+    base64 -d &gt; f3s-lan-ca.crt
+</pre>
+<br />
+<span>Install the CA certificate on Linux (Fedora/Rocky):</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ sudo cp f3s-lan-ca.crt /etc/pki/ca-trust/source/anchors/
+$ sudo update-ca-trust
+</pre>
+<br />
+<span>After trusting the CA, browsers will accept the LAN certificates without warnings.</span><br />
+<br />
+<span>*Scaling to other services*:</span><br />
+<br />
+<span>The same pattern can be applied to any service. To add LAN access:</span><br />
+<br />
+<span>1. Copy the <span class='inlinecode'>f3s-lan-tls</span> secret to the service&#39;s namespace (if not already there)</span><br />
+<span>2. Add a LAN Ingress resource using the pattern above</span><br />
+<span>3. Configure DNS: <span class='inlinecode'>192.168.1.138  service.f3s.lan.foo.zone</span></span><br />
+<span>4. Commit and push (ArgoCD will deploy automatically)</span><br />
+<br />
+<span>No changes needed to:</span><br />
+<br />
+<ul>
+<li>relayd configuration (forwards all traffic)</li>
+<li>cert-manager (wildcard cert covers all <span class='inlinecode'>*.f3s.lan.foo.zone</span>)</li>
+<li>CARP configuration (VIP shared by all services)</li>
+</ul><br />
+<span>*TLS offloaders summary*:</span><br />
+<br />
+<span>The f3s infrastructure now has three distinct TLS offloaders:</span><br />
+<br />
+<ul>
+<li>**OpenBSD relayd**: External internet traffic (<span class='inlinecode'>*.f3s.foo.zone</span>) using Let&#39;s Encrypt</li>
+<li>**Traefik (k3s)**: LAN HTTPS traffic (<span class='inlinecode'>*.f3s.lan.foo.zone</span>) using cert-manager</li>
+<li>**stunnel**: NFS-over-TLS (port 2323) using custom PKI</li>
+</ul><br />
+<span>Each serves a different purpose with appropriate certificate management for its use case.</span><br />
+<br />
 <h2 style='display: inline' id='installing-loki-and-alloy'>Installing Loki and Alloy</h2><br />
 <br />
 <span>While Prometheus handles metrics, Loki handles logs. It&#39;s designed to be cost-effective and easy to operate—it doesn&#39;t index the contents of logs, only the metadata (labels), making it very efficient for storage.</span><br />
diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml
index 6f849905..72abde8f 100644
--- a/gemfeed/atom.xml
+++ b/gemfeed/atom.xml
@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <feed xmlns="http://www.w3.org/2005/Atom">
-    <updated>2026-02-02T11:03:17+02:00</updated>
+    <updated>2026-02-05T17:38:26+02:00</updated>
     <title>foo.zone feed</title>
     <subtitle>To be in the .zone!</subtitle>
     <link href="https://foo.zone/gemfeed/atom.xml" rel="self" />
@@ -2706,6 +2706,7 @@ $ curl -s -G "http://localhost:3200/api/search" \
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-8-observability'>f3s: Kubernetes with FreeBSD - Part 8: Observability</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#update-lan-ingress-support-february-2026'>Update: LAN Ingress Support (February 2026)</a></li>
 <li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#persistent-storage-recap'>Persistent storage recap</a></li>
 <li>⇢ <a href='#the-monitoring-namespace'>The monitoring namespace</a></li>
@@ -2713,6 +2714,7 @@ $ curl -s -G "http://localhost:3200/api/search" \
 <li>⇢ ⇢ <a href='#prerequisites'>Prerequisites</a></li>
 <li>⇢ ⇢ <a href='#deploying-with-the-justfile'>Deploying with the Justfile</a></li>
 <li>⇢ ⇢ <a href='#exposing-grafana-via-ingress'>Exposing Grafana via ingress</a></li>
+<li>⇢ ⇢ <a href='#exposing-services-via-lan-ingress'>Exposing services via LAN ingress</a></li>
 <li>⇢ <a href='#installing-loki-and-alloy'>Installing Loki and Alloy</a></li>
 <li>⇢ ⇢ <a href='#prerequisites'>Prerequisites</a></li>
 <li>⇢ ⇢ <a href='#deploying-loki-and-alloy'>Deploying Loki and Alloy</a></li>
@@ -2750,6 +2752,10 @@ $ curl -s -G "http://localhost:3200/api/search" \
 <br />
 <a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
 <br />
+<h2 style='display: inline' id='update-lan-ingress-support-february-2026'>Update: LAN Ingress Support (February 2026)</h2><br />
+<br />
+<span>Update (2026-02-05):** This blog post has been updated to include a new section on exposing services via LAN ingress. The original blog post focused on external access through OpenBSD edge relays. The new section documents how to:</span><br />
+<br />
 <h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
 <br />
 <span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
@@ -2925,6 +2931,281 @@ prometheus-kube-prometheus-prometheus   ClusterIP   <font color="#000000">10.43<
 <br />
 <a href='./f3s-kubernetes-with-freebsd-part-8/grafana-dashboard.png'><img alt='Grafana dashboard showing cluster metrics' title='Grafana dashboard showing cluster metrics' src='./f3s-kubernetes-with-freebsd-part-8/grafana-dashboard.png' /></a><br />
 <br />
+<h3 style='display: inline' id='exposing-services-via-lan-ingress'>Exposing services via LAN ingress</h3><br />
+<br />
+<span>In addition to external access through the OpenBSD relays, services can also be exposed on the local network using LAN-specific ingresses. This is useful for accessing services from within the home network without going through the internet, reducing latency and providing an alternative path if the external relays are unavailable.</span><br />
+<br />
+<span>The LAN ingress architecture leverages the existing FreeBSD CARP (Common Address Redundancy Protocol) failover infrastructure that&#39;s already in place for NFS-over-TLS (see Part 5). Instead of deploying MetalLB or another LoadBalancer implementation, we reuse the CARP virtual IP (<span class='inlinecode'>192.168.1.138</span>) by adding HTTP/HTTPS forwarding alongside the existing stunnel service on port 2323.</span><br />
+<br />
+<span>*Architecture overview*:</span><br />
+<br />
+<span>The LAN access path differs from external access:</span><br />
+<br />
+<span>**External access (*.f3s.foo.zone):**</span><br />
+<pre>
+Internet → OpenBSD relayd (TLS termination, Let&#39;s Encrypt)
+        → WireGuard tunnel
+        → k3s Traefik :80 (HTTP)
+        → Service
+</pre>
+<br />
+<span>**LAN access (*.f3s.lan.foo.zone):**</span><br />
+<pre>
+LAN → FreeBSD CARP VIP (192.168.1.138)
+    → FreeBSD relayd (TCP forwarding)
+    → k3s Traefik :443 (TLS termination, cert-manager)
+    → Service
+</pre>
+<br />
+<span>The key architectural decisions:</span><br />
+<br />
+<ul>
+<li>FreeBSD <span class='inlinecode'>relayd</span> performs pure TCP forwarding (Layer 4) for ports 80 and 443, not TLS termination</li>
+<li>Traefik inside k3s handles TLS offloading using certificates from cert-manager</li>
+<li>Self-signed CA for LAN domains (no external dependencies)</li>
+<li>CARP provides automatic failover between f0 and f1</li>
+<li>No code changes to applications—just add a LAN ingress resource</li>
+</ul><br />
+<span>*Installing cert-manager*:</span><br />
+<br />
+<span>First, install cert-manager to handle certificate lifecycle management for LAN services. The installation is automated with a Justfile:</span><br />
+<br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/cert-manager'>codeberg.org/snonux/conf/f3s/cert-manager</a><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ cd conf/f3s/cert-manager
+$ just install
+kubectl apply -f cert-manager.yaml
+<i><font color="silver"># ... cert-manager CRDs and resources created ...</font></i>
+kubectl apply -f self-signed-issuer.yaml
+clusterissuer.cert-manager.io/selfsigned-issuer created
+clusterissuer.cert-manager.io/selfsigned-ca-issuer created
+kubectl apply -f ca-certificate.yaml
+certificate.cert-manager.io/selfsigned-ca created
+kubectl apply -f wildcard-certificate.yaml
+certificate.cert-manager.io/f3s-lan-wildcard created
+</pre>
+<br />
+<span>This creates:</span><br />
+<br />
+<ul>
+<li>A self-signed ClusterIssuer</li>
+<li>A CA certificate (<span class='inlinecode'>f3s-lan-ca</span>) valid for 10 years</li>
+<li>A CA-signed ClusterIssuer</li>
+<li>A wildcard certificate (<span class='inlinecode'>*.f3s.lan.foo.zone</span>) valid for 90 days with automatic renewal</li>
+</ul><br />
+<span>Verify the certificates:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get certificate -n cert-manager
+NAME               READY   SECRET                 AGE
+f3s-lan-wildcard   True    f3s-lan-tls            5m
+selfsigned-ca      True    selfsigned-ca-secret   5m
+</pre>
+<br />
+<span>The wildcard certificate (<span class='inlinecode'>f3s-lan-tls</span>) needs to be copied to any namespace that uses it:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \
+    sed <font color="#808080">'s/namespace: cert-manager/namespace: services/'</font> | \
+    kubectl apply -f -
+</pre>
+<br />
+<span>*Configuring FreeBSD relayd for LAN access*:</span><br />
+<br />
+<span>On both FreeBSD hosts (f0, f1), install and configure <span class='inlinecode'>relayd</span> for TCP forwarding:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas pkg install -y relayd
+</pre>
+<br />
+<span>Create <span class='inlinecode'>/usr/local/etc/relayd.conf</span>:</span><br />
+<br />
+<pre>
+# k3s nodes backend table
+table &lt;k3s_nodes&gt; { 192.168.1.120 192.168.1.121 192.168.1.122 }
+
+# TCP forwarding to Traefik (no TLS termination)
+relay "lan_http" {
+    listen on 192.168.1.138 port 80
+    forward to &lt;k3s_nodes&gt; port 80 check tcp
+}
+
+relay "lan_https" {
+    listen on 192.168.1.138 port 443
+    forward to &lt;k3s_nodes&gt; port 443 check tcp
+}
+</pre>
+<br />
+<span>Note: The IP addresses <span class='inlinecode'>192.168.1.120-122</span> are the LAN IPs of the k3s nodes (r0, r1, r2), not their WireGuard IPs. FreeBSD <span class='inlinecode'>relayd</span> requires PF (Packet Filter) to be enabled. Create a minimal <span class='inlinecode'>/etc/pf.conf</span>:</span><br />
+<br />
+<pre>
+# Basic PF rules for relayd
+set skip on lo0
+pass in quick
+pass out quick
+</pre>
+<br />
+<span>Enable PF and relayd:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES
+paul@f0:~ % doas service pf start
+paul@f0:~ % doas service pflog start
+paul@f0:~ % doas service relayd start
+</pre>
+<br />
+<span>Verify <span class='inlinecode'>relayd</span> is listening on the CARP VIP:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas sockstat -<font color="#000000">4</font> -l | grep <font color="#000000">192.168</font>.<font color="#000000">1.138</font>
+_relayd  relayd   <font color="#000000">2903</font>  <font color="#000000">11</font>  tcp4   <font color="#000000">192.168</font>.<font color="#000000">1.138</font>:<font color="#000000">80</font>      *:*
+_relayd  relayd   <font color="#000000">2903</font>  <font color="#000000">12</font>  tcp4   <font color="#000000">192.168</font>.<font color="#000000">1.138</font>:<font color="#000000">443</font>     *:*
+</pre>
+<br />
+<span>Repeat the same configuration on f1. Both hosts will run <span class='inlinecode'>relayd</span> listening on the CARP VIP, but only the CARP MASTER will respond to traffic. When failover occurs, the new MASTER takes over seamlessly.</span><br />
+<br />
+<span>*Adding LAN ingress to services*:</span><br />
+<br />
+<span>To expose a service on the LAN, add a second Ingress resource to its Helm chart. Here&#39;s an example for Grafana:</span><br />
+<br />
+<pre>
+---
+# LAN Ingress for grafana.f3s.lan.foo.zone
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: grafana-ingress-lan
+  namespace: monitoring
+  annotations:
+    spec.ingressClassName: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
+spec:
+  tls:
+    - hosts:
+        - grafana.f3s.lan.foo.zone
+      secretName: f3s-lan-tls
+  rules:
+    - host: grafana.f3s.lan.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: prometheus-grafana
+                port:
+                  number: 80
+</pre>
+<br />
+<span>Key points:</span><br />
+<br />
+<ul>
+<li>Use <span class='inlinecode'>web,websecure</span> entrypoints (both HTTP and HTTPS)</li>
+<li>Reference the <span class='inlinecode'>f3s-lan-tls</span> secret in the <span class='inlinecode'>tls</span> section</li>
+<li>Use <span class='inlinecode'>.f3s.lan.foo.zone</span> subdomain pattern</li>
+<li>Same backend service as the external ingress</li>
+</ul><br />
+<span>Apply the ingress and test:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl apply -f grafana-ingress-lan.yaml
+ingress.networking.k8s.io/grafana-ingress-lan created
+
+$ curl -k https://grafana.f3s.lan.foo.zone
+HTTP/<font color="#000000">2</font> <font color="#000000">302</font> 
+location: /login
+</pre>
+<br />
+<span>*Client-side DNS and CA setup*:</span><br />
+<br />
+<span>To access LAN services, clients need DNS entries and must trust the self-signed CA.</span><br />
+<br />
+<span>Add DNS entries to <span class='inlinecode'>/etc/hosts</span> on your laptop:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ sudo tee -a /etc/hosts &lt;&lt; <font color="#808080">'EOF'</font>
+<i><font color="silver"># f3s LAN services</font></i>
+<font color="#000000">192.168</font>.<font color="#000000">1.138</font>  grafana.f3s.lan.foo.zone
+<font color="#000000">192.168</font>.<font color="#000000">1.138</font>  navidrome.f3s.lan.foo.zone
+EOF
+</pre>
+<br />
+<span>The CARP VIP <span class='inlinecode'>192.168.1.138</span> provides high availability—traffic automatically fails over to the backup host if the master goes down.</span><br />
+<br />
+<span>Export the self-signed CA certificate:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get secret selfsigned-ca-secret -n cert-manager -o jsonpath=<font color="#808080">'{.data.ca</font>\.<font color="#808080">crt}'</font> | \
+    base64 -d &gt; f3s-lan-ca.crt
+</pre>
+<br />
+<span>Install the CA certificate on Linux (Fedora/Rocky):</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ sudo cp f3s-lan-ca.crt /etc/pki/ca-trust/source/anchors/
+$ sudo update-ca-trust
+</pre>
+<br />
+<span>After trusting the CA, browsers will accept the LAN certificates without warnings.</span><br />
+<br />
+<span>*Scaling to other services*:</span><br />
+<br />
+<span>The same pattern can be applied to any service. To add LAN access:</span><br />
+<br />
+<span>1. Copy the <span class='inlinecode'>f3s-lan-tls</span> secret to the service&#39;s namespace (if not already there)</span><br />
+<span>2. Add a LAN Ingress resource using the pattern above</span><br />
+<span>3. Configure DNS: <span class='inlinecode'>192.168.1.138  service.f3s.lan.foo.zone</span></span><br />
+<span>4. Commit and push (ArgoCD will deploy automatically)</span><br />
+<br />
+<span>No changes needed to:</span><br />
+<br />
+<ul>
+<li>relayd configuration (forwards all traffic)</li>
+<li>cert-manager (wildcard cert covers all <span class='inlinecode'>*.f3s.lan.foo.zone</span>)</li>
+<li>CARP configuration (VIP shared by all services)</li>
+</ul><br />
+<span>*TLS offloaders summary*:</span><br />
+<br />
+<span>The f3s infrastructure now has three distinct TLS offloaders:</span><br />
+<br />
+<ul>
+<li>**OpenBSD relayd**: External internet traffic (<span class='inlinecode'>*.f3s.foo.zone</span>) using Let&#39;s Encrypt</li>
+<li>**Traefik (k3s)**: LAN HTTPS traffic (<span class='inlinecode'>*.f3s.lan.foo.zone</span>) using cert-manager</li>
+<li>**stunnel**: NFS-over-TLS (port 2323) using custom PKI</li>
+</ul><br />
+<span>Each serves a different purpose with appropriate certificate management for its use case.</span><br />
+<br />
 <h2 style='display: inline' id='installing-loki-and-alloy'>Installing Loki and Alloy</h2><br />
 <br />
 <span>While Prometheus handles metrics, Loki handles logs. It&#39;s designed to be cost-effective and easy to operate—it doesn&#39;t index the contents of logs, only the metadata (labels), making it very efficient for storage.</span><br />
-- 
cgit v1.2.3