From 4359ae65896c573d3b50768bfa9dc216737e0858 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 11:50:41 +0200 Subject: Add LAN ingress section to f3s Part 8 blog post - Document CARP + relayd architecture for LAN access - Explain cert-manager setup with self-signed CA - Provide example LAN ingress configuration for Grafana - Include CARP failover testing results - Document three TLS offloaders (OpenBSD, Traefik, stunnel) - Show step-by-step setup for FreeBSD relayd - Add client-side CA trust instructions for Linux/macOS/Windows --- ...2-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 274 +++++++++++++++++++++ 1 file changed, 274 insertions(+) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index 008c512e..01d3b7f5 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -178,6 +178,280 @@ Grafana connects to Prometheus using the internal service URL `http://prometheus => ./f3s-kubernetes-with-freebsd-part-8/grafana-dashboard.png Grafana dashboard showing cluster metrics +### Exposing services via LAN ingress + +In addition to external access through the OpenBSD relays, services can also be exposed on the local network using LAN-specific ingresses. This is useful for accessing services from within the home network without going through the internet, reducing latency and providing an alternative path if the external relays are unavailable. + +The LAN ingress architecture leverages the existing FreeBSD CARP (Common Address Redundancy Protocol) failover infrastructure that's already in place for NFS-over-TLS (see Part 5). Instead of deploying MetalLB or another LoadBalancer implementation, we reuse the CARP virtual IP (`192.168.1.138`) by adding HTTP/HTTPS forwarding alongside the existing stunnel service on port 2323. + +#### Architecture overview + +The LAN access path differs from external access: + +**External access (*.f3s.foo.zone):** +``` +Internet → OpenBSD relayd (TLS termination, Let's Encrypt) + → WireGuard tunnel + → k3s Traefik :80 (HTTP) + → Service +``` + +**LAN access (*.f3s.lan.foo.zone):** +``` +LAN → FreeBSD CARP VIP (192.168.1.138) + → FreeBSD relayd (TCP forwarding) + → k3s Traefik :443 (TLS termination, cert-manager) + → Service +``` + +The key architectural decisions: + +* FreeBSD `relayd` performs pure TCP forwarding (Layer 4) for ports 80 and 443, not TLS termination +* Traefik inside k3s handles TLS offloading using certificates from cert-manager +* Self-signed CA for LAN domains (no external dependencies) +* CARP provides automatic failover between f0 and f1 +* No code changes to applications—just add a LAN ingress resource + +#### Installing cert-manager + +First, install cert-manager to handle certificate lifecycle management for LAN services. The installation is automated with a Justfile: + +=> https://codeberg.org/snonux/conf/src/branch/master/f3s/cert-manager codeberg.org/snonux/conf/f3s/cert-manager + +```sh +$ cd conf/f3s/cert-manager +$ just install +kubectl apply -f cert-manager.yaml +# ... cert-manager CRDs and resources created ... +kubectl apply -f self-signed-issuer.yaml +clusterissuer.cert-manager.io/selfsigned-issuer created +clusterissuer.cert-manager.io/selfsigned-ca-issuer created +kubectl apply -f ca-certificate.yaml +certificate.cert-manager.io/selfsigned-ca created +kubectl apply -f wildcard-certificate.yaml +certificate.cert-manager.io/f3s-lan-wildcard created +``` + +This creates: + +* A self-signed ClusterIssuer +* A CA certificate (`f3s-lan-ca`) valid for 10 years +* A CA-signed ClusterIssuer +* A wildcard certificate (`*.f3s.lan.foo.zone`) valid for 90 days with automatic renewal + +Verify the certificates: + +```sh +$ kubectl get certificate -n cert-manager +NAME READY SECRET AGE +f3s-lan-wildcard True f3s-lan-tls 5m +selfsigned-ca True selfsigned-ca-secret 5m +``` + +The wildcard certificate (`f3s-lan-tls`) needs to be copied to any namespace that uses it: + +```sh +$ kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \ + sed 's/namespace: cert-manager/namespace: services/' | \ + kubectl apply -f - +``` + +#### Configuring FreeBSD relayd for LAN access + +On both FreeBSD hosts (f0, f1), install and configure `relayd` for TCP forwarding: + +```sh +paul@f0:~ % doas pkg install -y relayd +``` + +Create `/usr/local/etc/relayd.conf`: + +``` +# k3s nodes backend table +table { 192.168.1.120 192.168.1.121 192.168.1.122 } + +# TCP forwarding to Traefik (no TLS termination) +relay "lan_http" { + listen on 192.168.1.138 port 80 + forward to port 80 check tcp +} + +relay "lan_https" { + listen on 192.168.1.138 port 443 + forward to port 443 check tcp +} +``` + +Note: The IP addresses `192.168.1.120-122` are the LAN IPs of the k3s nodes (r0, r1, r2), not their WireGuard IPs. FreeBSD `relayd` requires PF (Packet Filter) to be enabled. Create a minimal `/etc/pf.conf`: + +``` +# Basic PF rules for relayd +set skip on lo0 +pass in quick +pass out quick +``` + +Enable PF and relayd: + +```sh +paul@f0:~ % doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES +paul@f0:~ % doas service pf start +paul@f0:~ % doas service pflog start +paul@f0:~ % doas service relayd start +``` + +Verify `relayd` is listening on the CARP VIP: + +```sh +paul@f0:~ % doas sockstat -4 -l | grep 192.168.1.138 +_relayd relayd 2903 11 tcp4 192.168.1.138:80 *:* +_relayd relayd 2903 12 tcp4 192.168.1.138:443 *:* +``` + +Repeat the same configuration on f1. Both hosts will run `relayd` listening on the CARP VIP, but only the CARP MASTER will respond to traffic. When failover occurs, the new MASTER takes over seamlessly. + +#### Adding LAN ingress to services + +To expose a service on the LAN, add a second Ingress resource to its Helm chart. Here's an example for Grafana: + +```yaml +--- +# LAN Ingress for grafana.f3s.lan.foo.zone +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: grafana-ingress-lan + namespace: monitoring + annotations: + spec.ingressClassName: traefik + traefik.ingress.kubernetes.io/router.entrypoints: web,websecure +spec: + tls: + - hosts: + - grafana.f3s.lan.foo.zone + secretName: f3s-lan-tls + rules: + - host: grafana.f3s.lan.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: prometheus-grafana + port: + number: 80 +``` + +Key points: + +* Use `web,websecure` entrypoints (both HTTP and HTTPS) +* Reference the `f3s-lan-tls` secret in the `tls` section +* Use `.f3s.lan.foo.zone` subdomain pattern +* Same backend service as the external ingress + +Apply the ingress and test: + +```sh +$ kubectl apply -f grafana-ingress-lan.yaml +ingress.networking.k8s.io/grafana-ingress-lan created + +$ curl -k https://grafana.f3s.lan.foo.zone +HTTP/2 302 +location: /login +``` + +#### Client-side CA trust + +Since the LAN certificates are self-signed, clients need to trust the CA. Export the CA certificate: + +```sh +$ kubectl get secret selfsigned-ca-secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | \ + base64 -d > f3s-lan-ca.crt +``` + +**On Linux (Fedora/Rocky):** + +```sh +$ sudo cp f3s-lan-ca.crt /etc/pki/ca-trust/source/anchors/ +$ sudo update-ca-trust +``` + +**On macOS:** + +```sh +$ sudo security add-trusted-cert -d -r trustRoot \ + -k /Library/Keychains/System.keychain f3s-lan-ca.crt +``` + +**On Windows:** + +Import `f3s-lan-ca.crt` into "Trusted Root Certification Authorities" via `certmgr.msc`. + +After trusting the CA, browsers will accept the LAN certificates without warnings. + +#### CARP failover testing + +The CARP + relayd architecture provides high availability with automatic failover. Testing confirmed zero-downtime operation: + +**Failover test (f0 → f1):** +```sh +# Demote f0 to BACKUP, promote f1 to MASTER +$ ssh f0 'doas ifconfig re0 vhid 1 advskew 200' +$ ssh f1 'doas ifconfig re0 vhid 1 advskew 0 state master' + +# Service tests: 10/10 requests successful (HTTP 302) +$ for i in {1..10}; do curl -k https://grafana.f3s.lan.foo.zone \ + -s -o /dev/null -w "Test $i: %{http_code}\n"; sleep 0.5; done +Test 1: 302 ✓ +Test 2: 302 ✓ +... +Test 10: 302 ✓ +``` + +**Failback test (f1 → f0):** +```sh +# Restore f0 as MASTER +$ ssh f0 'doas ifconfig re0 vhid 1 advskew 0 state master' +$ ssh f1 'doas ifconfig re0 vhid 1 advskew 100' + +# Service tests: 10/10 requests successful (HTTP 302) +``` + +Results: + +* Failover time: ~3 seconds +* Packet loss: 0/20 requests +* Service availability: 100% +* TLS certificate: Maintained (Traefik handles TLS) + +The architecture successfully provides high availability for LAN services without requiring MetalLB or complex load balancer setups. + +#### Scaling to other services + +The same pattern can be applied to any service. To add LAN access: + +1. Copy the `f3s-lan-tls` secret to the service's namespace (if not already there) +2. Add a LAN Ingress resource using the pattern above +3. Configure DNS: `192.168.1.138 service.f3s.lan.foo.zone` +4. Commit and push (ArgoCD will deploy automatically) + +No changes needed to: + +* relayd configuration (forwards all traffic) +* cert-manager (wildcard cert covers all `*.f3s.lan.foo.zone`) +* CARP configuration (VIP shared by all services) + +#### TLS offloaders summary + +The f3s infrastructure now has three distinct TLS offloaders: + +* **OpenBSD relayd**: External internet traffic (`*.f3s.foo.zone`) using Let's Encrypt +* **Traefik (k3s)**: LAN HTTPS traffic (`*.f3s.lan.foo.zone`) using cert-manager +* **stunnel**: NFS-over-TLS (port 2323) using custom PKI + +Each serves a different purpose with appropriate certificate management for its use case. + ## Installing Loki and Alloy While Prometheus handles metrics, Loki handles logs. It's designed to be cost-effective and easy to operate—it doesn't index the contents of logs, only the metadata (labels), making it very efficient for storage. -- cgit v1.2.3 From 2abd85d2d78bc66c87a08490446e8ec70607139f Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 11:52:21 +0200 Subject: Add update notice for LAN ingress feature Added update notice at the beginning of the blog post to inform readers about the LAN ingress feature addition in February 2026. The notice references the new section and summarizes key features. --- .../2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index 01d3b7f5..4e280cdd 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -45,6 +45,20 @@ $ cd f3s/prometheus/ The deployment concepts and architecture remain the same—only the deployment method changed from imperative (`helm install/upgrade`) to declarative (GitOps with ArgoCD). +## Update: LAN Ingress Support (February 2026) + +**Update (2026-02-05):** This blog post has been updated to include a new section on exposing services via LAN ingress. The original blog post focused on external access through OpenBSD edge relays. The new section documents how to: + +* Expose services on the local network using LAN-specific ingresses (`*.f3s.lan.foo.zone`) +* Leverage the existing FreeBSD CARP failover infrastructure for high availability +* Configure FreeBSD `relayd` for TCP forwarding (Layer 4) instead of TLS termination +* Use cert-manager with self-signed certificates for LAN TLS offloading via Traefik +* Achieve zero-downtime CARP failover (tested and validated) + +This enhancement complements the existing external access method without requiring MetalLB or other LoadBalancer implementations. The f3s infrastructure now has three distinct TLS offloaders: OpenBSD relayd for external traffic, Traefik (k3s) for LAN HTTPS, and stunnel for NFS-over-TLS. + +See the new "### Exposing services via LAN ingress" section under "### Exposing Grafana via ingress" for complete details. + ## Persistent storage recap All observability components need persistent storage so that metrics and logs survive pod restarts. As covered in Part 6 of this series, the cluster uses NFS-backed persistent volumes: -- cgit v1.2.3 From bfa592ee211aab73ad4c898cbf389cdd6912bbae Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 11:59:18 +0200 Subject: Fix Gemini gemtext format: Replace #### headers with *Header*: Gemini gemtext only supports headers up to ###. Changed all #### headers to the *Header*: format for proper Gemini compatibility. --- ...2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index 4e280cdd..33f7827a 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -198,7 +198,7 @@ In addition to external access through the OpenBSD relays, services can also be The LAN ingress architecture leverages the existing FreeBSD CARP (Common Address Redundancy Protocol) failover infrastructure that's already in place for NFS-over-TLS (see Part 5). Instead of deploying MetalLB or another LoadBalancer implementation, we reuse the CARP virtual IP (`192.168.1.138`) by adding HTTP/HTTPS forwarding alongside the existing stunnel service on port 2323. -#### Architecture overview +*Architecture overview*: The LAN access path differs from external access: @@ -226,7 +226,7 @@ The key architectural decisions: * CARP provides automatic failover between f0 and f1 * No code changes to applications—just add a LAN ingress resource -#### Installing cert-manager +*Installing cert-manager*: First, install cert-manager to handle certificate lifecycle management for LAN services. The installation is automated with a Justfile: @@ -270,7 +270,7 @@ $ kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \ kubectl apply -f - ``` -#### Configuring FreeBSD relayd for LAN access +*Configuring FreeBSD relayd for LAN access*: On both FreeBSD hosts (f0, f1), install and configure `relayd` for TCP forwarding: @@ -324,7 +324,7 @@ _relayd relayd 2903 12 tcp4 192.168.1.138:443 *:* Repeat the same configuration on f1. Both hosts will run `relayd` listening on the CARP VIP, but only the CARP MASTER will respond to traffic. When failover occurs, the new MASTER takes over seamlessly. -#### Adding LAN ingress to services +*Adding LAN ingress to services*: To expose a service on the LAN, add a second Ingress resource to its Helm chart. Here's an example for Grafana: @@ -375,7 +375,7 @@ HTTP/2 302 location: /login ``` -#### Client-side CA trust +*Client-side CA trust*: Since the LAN certificates are self-signed, clients need to trust the CA. Export the CA certificate: @@ -404,7 +404,7 @@ Import `f3s-lan-ca.crt` into "Trusted Root Certification Authorities" via `certm After trusting the CA, browsers will accept the LAN certificates without warnings. -#### CARP failover testing +*CARP failover testing*: The CARP + relayd architecture provides high availability with automatic failover. Testing confirmed zero-downtime operation: @@ -441,7 +441,7 @@ Results: The architecture successfully provides high availability for LAN services without requiring MetalLB or complex load balancer setups. -#### Scaling to other services +*Scaling to other services*: The same pattern can be applied to any service. To add LAN access: @@ -456,7 +456,7 @@ No changes needed to: * cert-manager (wildcard cert covers all `*.f3s.lan.foo.zone`) * CARP configuration (VIP shared by all services) -#### TLS offloaders summary +*TLS offloaders summary*: The f3s infrastructure now has three distinct TLS offloaders: -- cgit v1.2.3 From 23780f846fe27983dcd4e713b679103505f0eff8 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 12:01:16 +0200 Subject: Remove macOS and Windows CA trust instructions Keep only Linux/FreeBSD setup instructions that match the actual infrastructure. Removed unnecessary platform-specific instructions for macOS and Windows from the Client-side CA trust section. --- .../2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index 33f7827a..7a054ed6 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -384,24 +384,13 @@ $ kubectl get secret selfsigned-ca-secret -n cert-manager -o jsonpath='{.data.ca base64 -d > f3s-lan-ca.crt ``` -**On Linux (Fedora/Rocky):** +Install the CA certificate on Linux (Fedora/Rocky): ```sh $ sudo cp f3s-lan-ca.crt /etc/pki/ca-trust/source/anchors/ $ sudo update-ca-trust ``` -**On macOS:** - -```sh -$ sudo security add-trusted-cert -d -r trustRoot \ - -k /Library/Keychains/System.keychain f3s-lan-ca.crt -``` - -**On Windows:** - -Import `f3s-lan-ca.crt` into "Trusted Root Certification Authorities" via `certmgr.msc`. - After trusting the CA, browsers will accept the LAN certificates without warnings. *CARP failover testing*: -- cgit v1.2.3 From e7665c384557096338ccbb58caba4022f6f74946 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 12:03:07 +0200 Subject: Add /etc/hosts DNS configuration for LAN services Updated client-side setup section to include DNS configuration. Added instructions for adding /etc/hosts entries pointing LAN service domains to the CARP VIP (192.168.1.138). Renamed section from 'Client-side CA trust' to 'Client-side DNS and CA setup' to reflect both DNS and certificate configuration. --- ...25-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index 7a054ed6..4011e017 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -375,9 +375,23 @@ HTTP/2 302 location: /login ``` -*Client-side CA trust*: +*Client-side DNS and CA setup*: -Since the LAN certificates are self-signed, clients need to trust the CA. Export the CA certificate: +To access LAN services, clients need DNS entries and must trust the self-signed CA. + +Add DNS entries to `/etc/hosts` on your laptop: + +```sh +$ sudo tee -a /etc/hosts << 'EOF' +# f3s LAN services +192.168.1.138 grafana.f3s.lan.foo.zone +192.168.1.138 navidrome.f3s.lan.foo.zone +EOF +``` + +The CARP VIP `192.168.1.138` provides high availability—traffic automatically fails over to the backup host if the master goes down. + +Export the self-signed CA certificate: ```sh $ kubectl get secret selfsigned-ca-secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | \ -- cgit v1.2.3 From c50b338cda40cfcd400db5279a3564a8804edd2e Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 12:04:06 +0200 Subject: Remove CARP failover testing section from LAN ingress Removed the detailed CARP failover testing section including failover/failback test commands and results. The section was too detailed for the blog post scope. --- ...2-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 37 ---------------------- 1 file changed, 37 deletions(-) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index 4011e017..c69e6d39 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -407,43 +407,6 @@ $ sudo update-ca-trust After trusting the CA, browsers will accept the LAN certificates without warnings. -*CARP failover testing*: - -The CARP + relayd architecture provides high availability with automatic failover. Testing confirmed zero-downtime operation: - -**Failover test (f0 → f1):** -```sh -# Demote f0 to BACKUP, promote f1 to MASTER -$ ssh f0 'doas ifconfig re0 vhid 1 advskew 200' -$ ssh f1 'doas ifconfig re0 vhid 1 advskew 0 state master' - -# Service tests: 10/10 requests successful (HTTP 302) -$ for i in {1..10}; do curl -k https://grafana.f3s.lan.foo.zone \ - -s -o /dev/null -w "Test $i: %{http_code}\n"; sleep 0.5; done -Test 1: 302 ✓ -Test 2: 302 ✓ -... -Test 10: 302 ✓ -``` - -**Failback test (f1 → f0):** -```sh -# Restore f0 as MASTER -$ ssh f0 'doas ifconfig re0 vhid 1 advskew 0 state master' -$ ssh f1 'doas ifconfig re0 vhid 1 advskew 100' - -# Service tests: 10/10 requests successful (HTTP 302) -``` - -Results: - -* Failover time: ~3 seconds -* Packet loss: 0/20 requests -* Service availability: 100% -* TLS certificate: Maintained (Traefik handles TLS) - -The architecture successfully provides high availability for LAN services without requiring MetalLB or complex load balancer setups. - *Scaling to other services*: The same pattern can be applied to any service. To add LAN access: -- cgit v1.2.3 From 87196de6fdbadd046d550706911332df4b9f61b7 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 17:05:15 +0200 Subject: Move LAN ingress content from Part 8 to Part 7 Relocated the "Exposing services via LAN ingress" section from Part 8 (Observability) to Part 7 (k3s and first pod deployments), as Part 7 is where ingresses were originally introduced. Updated the Part 8 update notice to reference Part 7 for LAN ingress implementation details. --- ...0-02-f3s-kubernetes-with-freebsd-part-7.gmi.tpl | 239 ++++++++++++++++++++ ...2-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 246 +-------------------- 2 files changed, 243 insertions(+), 242 deletions(-) diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi.tpl b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi.tpl index 5b5073c7..9c715877 100644 --- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi.tpl +++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi.tpl @@ -779,6 +779,245 @@ This approach provides several benefits: This fallback mechanism has proven invaluable during maintenance windows and unexpected outages, ensuring that users always get a response even when the home lab is offline. +## Exposing services via LAN ingress + +In addition to external access through the OpenBSD relays, services can also be exposed on the local network using LAN-specific ingresses. This is useful for accessing services from within the home network without going through the internet, reducing latency and providing an alternative path if the external relays are unavailable. + +The LAN ingress architecture leverages the existing FreeBSD CARP (Common Address Redundancy Protocol) failover infrastructure that's already in place for NFS-over-TLS (see Part 5). Instead of deploying MetalLB or another LoadBalancer implementation, we reuse the CARP virtual IP (`192.168.1.138`) by adding HTTP/HTTPS forwarding alongside the existing stunnel service on port 2323. + +*Architecture overview*: + +The LAN access path differs from external access: + +**External access (*.f3s.foo.zone):** +``` +Internet → OpenBSD relayd (TLS termination, Let's Encrypt) + → WireGuard tunnel + → k3s Traefik :80 (HTTP) + → Service +``` + +**LAN access (*.f3s.lan.foo.zone):** +``` +LAN → FreeBSD CARP VIP (192.168.1.138) + → FreeBSD relayd (TCP forwarding) + → k3s Traefik :443 (TLS termination, cert-manager) + → Service +``` + +The key architectural decisions: + +* FreeBSD `relayd` performs pure TCP forwarding (Layer 4) for ports 80 and 443, not TLS termination +* Traefik inside k3s handles TLS offloading using certificates from cert-manager +* Self-signed CA for LAN domains (no external dependencies) +* CARP provides automatic failover between f0 and f1 +* No code changes to applications—just add a LAN ingress resource + +*Installing cert-manager*: + +First, install cert-manager to handle certificate lifecycle management for LAN services. The installation is automated with a Justfile: + +=> https://codeberg.org/snonux/conf/src/branch/master/f3s/cert-manager codeberg.org/snonux/conf/f3s/cert-manager + +```sh +$ cd conf/f3s/cert-manager +$ just install +kubectl apply -f cert-manager.yaml +# ... cert-manager CRDs and resources created ... +kubectl apply -f self-signed-issuer.yaml +clusterissuer.cert-manager.io/selfsigned-issuer created +clusterissuer.cert-manager.io/selfsigned-ca-issuer created +kubectl apply -f ca-certificate.yaml +certificate.cert-manager.io/selfsigned-ca created +kubectl apply -f wildcard-certificate.yaml +certificate.cert-manager.io/f3s-lan-wildcard created +``` + +This creates: + +* A self-signed ClusterIssuer +* A CA certificate (`f3s-lan-ca`) valid for 10 years +* A CA-signed ClusterIssuer +* A wildcard certificate (`*.f3s.lan.foo.zone`) valid for 90 days with automatic renewal + +Verify the certificates: + +```sh +$ kubectl get certificate -n cert-manager +NAME READY SECRET AGE +f3s-lan-wildcard True f3s-lan-tls 5m +selfsigned-ca True selfsigned-ca-secret 5m +``` + +The wildcard certificate (`f3s-lan-tls`) needs to be copied to any namespace that uses it: + +```sh +$ kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \ + sed 's/namespace: cert-manager/namespace: services/' | \ + kubectl apply -f - +``` + +*Configuring FreeBSD relayd for LAN access*: + +On both FreeBSD hosts (f0, f1), install and configure `relayd` for TCP forwarding: + +```sh +paul@f0:~ % doas pkg install -y relayd +``` + +Create `/usr/local/etc/relayd.conf`: + +``` +# k3s nodes backend table +table { 192.168.1.120 192.168.1.121 192.168.1.122 } + +# TCP forwarding to Traefik (no TLS termination) +relay "lan_http" { + listen on 192.168.1.138 port 80 + forward to port 80 check tcp +} + +relay "lan_https" { + listen on 192.168.1.138 port 443 + forward to port 443 check tcp +} +``` + +Note: The IP addresses `192.168.1.120-122` are the LAN IPs of the k3s nodes (r0, r1, r2), not their WireGuard IPs. FreeBSD `relayd` requires PF (Packet Filter) to be enabled. Create a minimal `/etc/pf.conf`: + +``` +# Basic PF rules for relayd +set skip on lo0 +pass in quick +pass out quick +``` + +Enable PF and relayd: + +```sh +paul@f0:~ % doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES +paul@f0:~ % doas service pf start +paul@f0:~ % doas service pflog start +paul@f0:~ % doas service relayd start +``` + +Verify `relayd` is listening on the CARP VIP: + +```sh +paul@f0:~ % doas sockstat -4 -l | grep 192.168.1.138 +_relayd relayd 2903 11 tcp4 192.168.1.138:80 *:* +_relayd relayd 2903 12 tcp4 192.168.1.138:443 *:* +``` + +Repeat the same configuration on f1. Both hosts will run `relayd` listening on the CARP VIP, but only the CARP MASTER will respond to traffic. When failover occurs, the new MASTER takes over seamlessly. + +*Adding LAN ingress to services*: + +To expose a service on the LAN, add a second Ingress resource to its Helm chart. Here's an example for Navidrome: + +```yaml +--- +# LAN Ingress for navidrome.f3s.lan.foo.zone +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: navidrome-ingress-lan + namespace: services + annotations: + spec.ingressClassName: traefik + traefik.ingress.kubernetes.io/router.entrypoints: web,websecure +spec: + tls: + - hosts: + - navidrome.f3s.lan.foo.zone + secretName: f3s-lan-tls + rules: + - host: navidrome.f3s.lan.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: navidrome-service + port: + number: 4533 +``` + +Key points: + +* Use `web,websecure` entrypoints (both HTTP and HTTPS) +* Reference the `f3s-lan-tls` secret in the `tls` section +* Use `.f3s.lan.foo.zone` subdomain pattern +* Same backend service as the external ingress + +Apply the ingress and test: + +```sh +$ kubectl apply -f navidrome-ingress-lan.yaml +ingress.networking.k8s.io/navidrome-ingress-lan created + +$ curl -k https://navidrome.f3s.lan.foo.zone +HTTP/2 302 +location: /app/ +``` + +*Client-side DNS and CA setup*: + +To access LAN services, clients need DNS entries and must trust the self-signed CA. + +Add DNS entries to `/etc/hosts` on your laptop: + +```sh +$ sudo tee -a /etc/hosts << 'EOF' +# f3s LAN services +192.168.1.138 navidrome.f3s.lan.foo.zone +EOF +``` + +The CARP VIP `192.168.1.138` provides high availability—traffic automatically fails over to the backup host if the master goes down. + +Export the self-signed CA certificate: + +```sh +$ kubectl get secret selfsigned-ca-secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | \ + base64 -d > f3s-lan-ca.crt +``` + +Install the CA certificate on Linux (Fedora/Rocky): + +```sh +$ sudo cp f3s-lan-ca.crt /etc/pki/ca-trust/source/anchors/ +$ sudo update-ca-trust +``` + +After trusting the CA, browsers will accept the LAN certificates without warnings. + +*Scaling to other services*: + +The same pattern can be applied to any service. To add LAN access: + +1. Copy the `f3s-lan-tls` secret to the service's namespace (if not already there) +2. Add a LAN Ingress resource using the pattern above +3. Configure DNS: `192.168.1.138 service.f3s.lan.foo.zone` +4. Commit and push (ArgoCD will deploy automatically) + +No changes needed to: + +* relayd configuration (forwards all traffic) +* cert-manager (wildcard cert covers all `*.f3s.lan.foo.zone`) +* CARP configuration (VIP shared by all services) + +*TLS offloaders summary*: + +The f3s infrastructure now has three distinct TLS offloaders: + +* **OpenBSD relayd**: External internet traffic (`*.f3s.foo.zone`) using Let's Encrypt +* **Traefik (k3s)**: LAN HTTPS traffic (`*.f3s.lan.foo.zone`) using cert-manager +* **stunnel**: NFS-over-TLS (port 2323) using custom PKI + +Each serves a different purpose with appropriate certificate management for its use case. + ## Deploying the private Docker image registry As not all Docker images I want to deploy are available on public Docker registries and as I also build some of them by myself, there is the need of a private registry. diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index c69e6d39..56e3323b 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -47,7 +47,7 @@ The deployment concepts and architecture remain the same—only the deployment m ## Update: LAN Ingress Support (February 2026) -**Update (2026-02-05):** This blog post has been updated to include a new section on exposing services via LAN ingress. The original blog post focused on external access through OpenBSD edge relays. The new section documents how to: +**Update (2026-02-05):** This blog post series has been updated to include a new section on exposing services via LAN ingress. The implementation details have been added to Part 7 (where ingresses were originally introduced). The LAN ingress section documents how to: * Expose services on the local network using LAN-specific ingresses (`*.f3s.lan.foo.zone`) * Leverage the existing FreeBSD CARP failover infrastructure for high availability @@ -57,7 +57,9 @@ The deployment concepts and architecture remain the same—only the deployment m This enhancement complements the existing external access method without requiring MetalLB or other LoadBalancer implementations. The f3s infrastructure now has three distinct TLS offloaders: OpenBSD relayd for external traffic, Traefik (k3s) for LAN HTTPS, and stunnel for NFS-over-TLS. -See the new "### Exposing services via LAN ingress" section under "### Exposing Grafana via ingress" for complete details. +For the full implementation details, see: + +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments ## Persistent storage recap @@ -192,246 +194,6 @@ Grafana connects to Prometheus using the internal service URL `http://prometheus => ./f3s-kubernetes-with-freebsd-part-8/grafana-dashboard.png Grafana dashboard showing cluster metrics -### Exposing services via LAN ingress - -In addition to external access through the OpenBSD relays, services can also be exposed on the local network using LAN-specific ingresses. This is useful for accessing services from within the home network without going through the internet, reducing latency and providing an alternative path if the external relays are unavailable. - -The LAN ingress architecture leverages the existing FreeBSD CARP (Common Address Redundancy Protocol) failover infrastructure that's already in place for NFS-over-TLS (see Part 5). Instead of deploying MetalLB or another LoadBalancer implementation, we reuse the CARP virtual IP (`192.168.1.138`) by adding HTTP/HTTPS forwarding alongside the existing stunnel service on port 2323. - -*Architecture overview*: - -The LAN access path differs from external access: - -**External access (*.f3s.foo.zone):** -``` -Internet → OpenBSD relayd (TLS termination, Let's Encrypt) - → WireGuard tunnel - → k3s Traefik :80 (HTTP) - → Service -``` - -**LAN access (*.f3s.lan.foo.zone):** -``` -LAN → FreeBSD CARP VIP (192.168.1.138) - → FreeBSD relayd (TCP forwarding) - → k3s Traefik :443 (TLS termination, cert-manager) - → Service -``` - -The key architectural decisions: - -* FreeBSD `relayd` performs pure TCP forwarding (Layer 4) for ports 80 and 443, not TLS termination -* Traefik inside k3s handles TLS offloading using certificates from cert-manager -* Self-signed CA for LAN domains (no external dependencies) -* CARP provides automatic failover between f0 and f1 -* No code changes to applications—just add a LAN ingress resource - -*Installing cert-manager*: - -First, install cert-manager to handle certificate lifecycle management for LAN services. The installation is automated with a Justfile: - -=> https://codeberg.org/snonux/conf/src/branch/master/f3s/cert-manager codeberg.org/snonux/conf/f3s/cert-manager - -```sh -$ cd conf/f3s/cert-manager -$ just install -kubectl apply -f cert-manager.yaml -# ... cert-manager CRDs and resources created ... -kubectl apply -f self-signed-issuer.yaml -clusterissuer.cert-manager.io/selfsigned-issuer created -clusterissuer.cert-manager.io/selfsigned-ca-issuer created -kubectl apply -f ca-certificate.yaml -certificate.cert-manager.io/selfsigned-ca created -kubectl apply -f wildcard-certificate.yaml -certificate.cert-manager.io/f3s-lan-wildcard created -``` - -This creates: - -* A self-signed ClusterIssuer -* A CA certificate (`f3s-lan-ca`) valid for 10 years -* A CA-signed ClusterIssuer -* A wildcard certificate (`*.f3s.lan.foo.zone`) valid for 90 days with automatic renewal - -Verify the certificates: - -```sh -$ kubectl get certificate -n cert-manager -NAME READY SECRET AGE -f3s-lan-wildcard True f3s-lan-tls 5m -selfsigned-ca True selfsigned-ca-secret 5m -``` - -The wildcard certificate (`f3s-lan-tls`) needs to be copied to any namespace that uses it: - -```sh -$ kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \ - sed 's/namespace: cert-manager/namespace: services/' | \ - kubectl apply -f - -``` - -*Configuring FreeBSD relayd for LAN access*: - -On both FreeBSD hosts (f0, f1), install and configure `relayd` for TCP forwarding: - -```sh -paul@f0:~ % doas pkg install -y relayd -``` - -Create `/usr/local/etc/relayd.conf`: - -``` -# k3s nodes backend table -table { 192.168.1.120 192.168.1.121 192.168.1.122 } - -# TCP forwarding to Traefik (no TLS termination) -relay "lan_http" { - listen on 192.168.1.138 port 80 - forward to port 80 check tcp -} - -relay "lan_https" { - listen on 192.168.1.138 port 443 - forward to port 443 check tcp -} -``` - -Note: The IP addresses `192.168.1.120-122` are the LAN IPs of the k3s nodes (r0, r1, r2), not their WireGuard IPs. FreeBSD `relayd` requires PF (Packet Filter) to be enabled. Create a minimal `/etc/pf.conf`: - -``` -# Basic PF rules for relayd -set skip on lo0 -pass in quick -pass out quick -``` - -Enable PF and relayd: - -```sh -paul@f0:~ % doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES -paul@f0:~ % doas service pf start -paul@f0:~ % doas service pflog start -paul@f0:~ % doas service relayd start -``` - -Verify `relayd` is listening on the CARP VIP: - -```sh -paul@f0:~ % doas sockstat -4 -l | grep 192.168.1.138 -_relayd relayd 2903 11 tcp4 192.168.1.138:80 *:* -_relayd relayd 2903 12 tcp4 192.168.1.138:443 *:* -``` - -Repeat the same configuration on f1. Both hosts will run `relayd` listening on the CARP VIP, but only the CARP MASTER will respond to traffic. When failover occurs, the new MASTER takes over seamlessly. - -*Adding LAN ingress to services*: - -To expose a service on the LAN, add a second Ingress resource to its Helm chart. Here's an example for Grafana: - -```yaml ---- -# LAN Ingress for grafana.f3s.lan.foo.zone -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: grafana-ingress-lan - namespace: monitoring - annotations: - spec.ingressClassName: traefik - traefik.ingress.kubernetes.io/router.entrypoints: web,websecure -spec: - tls: - - hosts: - - grafana.f3s.lan.foo.zone - secretName: f3s-lan-tls - rules: - - host: grafana.f3s.lan.foo.zone - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: prometheus-grafana - port: - number: 80 -``` - -Key points: - -* Use `web,websecure` entrypoints (both HTTP and HTTPS) -* Reference the `f3s-lan-tls` secret in the `tls` section -* Use `.f3s.lan.foo.zone` subdomain pattern -* Same backend service as the external ingress - -Apply the ingress and test: - -```sh -$ kubectl apply -f grafana-ingress-lan.yaml -ingress.networking.k8s.io/grafana-ingress-lan created - -$ curl -k https://grafana.f3s.lan.foo.zone -HTTP/2 302 -location: /login -``` - -*Client-side DNS and CA setup*: - -To access LAN services, clients need DNS entries and must trust the self-signed CA. - -Add DNS entries to `/etc/hosts` on your laptop: - -```sh -$ sudo tee -a /etc/hosts << 'EOF' -# f3s LAN services -192.168.1.138 grafana.f3s.lan.foo.zone -192.168.1.138 navidrome.f3s.lan.foo.zone -EOF -``` - -The CARP VIP `192.168.1.138` provides high availability—traffic automatically fails over to the backup host if the master goes down. - -Export the self-signed CA certificate: - -```sh -$ kubectl get secret selfsigned-ca-secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | \ - base64 -d > f3s-lan-ca.crt -``` - -Install the CA certificate on Linux (Fedora/Rocky): - -```sh -$ sudo cp f3s-lan-ca.crt /etc/pki/ca-trust/source/anchors/ -$ sudo update-ca-trust -``` - -After trusting the CA, browsers will accept the LAN certificates without warnings. - -*Scaling to other services*: - -The same pattern can be applied to any service. To add LAN access: - -1. Copy the `f3s-lan-tls` secret to the service's namespace (if not already there) -2. Add a LAN Ingress resource using the pattern above -3. Configure DNS: `192.168.1.138 service.f3s.lan.foo.zone` -4. Commit and push (ArgoCD will deploy automatically) - -No changes needed to: - -* relayd configuration (forwards all traffic) -* cert-manager (wildcard cert covers all `*.f3s.lan.foo.zone`) -* CARP configuration (VIP shared by all services) - -*TLS offloaders summary*: - -The f3s infrastructure now has three distinct TLS offloaders: - -* **OpenBSD relayd**: External internet traffic (`*.f3s.foo.zone`) using Let's Encrypt -* **Traefik (k3s)**: LAN HTTPS traffic (`*.f3s.lan.foo.zone`) using cert-manager -* **stunnel**: NFS-over-TLS (port 2323) using custom PKI - -Each serves a different purpose with appropriate certificate management for its use case. - ## Installing Loki and Alloy While Prometheus handles metrics, Loki handles logs. It's designed to be cost-effective and easy to operate—it doesn't index the contents of logs, only the metadata (labels), making it very efficient for storage. -- cgit v1.2.3 From 8831e5201b512b623e980a0367db5f848f4af2da Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 5 Feb 2026 17:31:24 +0200 Subject: Revert Part 8 to original state without LAN ingress content Removed the LAN ingress update notice from Part 8 entirely, as the LAN ingress content has been moved to Part 7 where ingresses were originally introduced. --- ...2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl index 56e3323b..008c512e 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi.tpl @@ -45,22 +45,6 @@ $ cd f3s/prometheus/ The deployment concepts and architecture remain the same—only the deployment method changed from imperative (`helm install/upgrade`) to declarative (GitOps with ArgoCD). -## Update: LAN Ingress Support (February 2026) - -**Update (2026-02-05):** This blog post series has been updated to include a new section on exposing services via LAN ingress. The implementation details have been added to Part 7 (where ingresses were originally introduced). The LAN ingress section documents how to: - -* Expose services on the local network using LAN-specific ingresses (`*.f3s.lan.foo.zone`) -* Leverage the existing FreeBSD CARP failover infrastructure for high availability -* Configure FreeBSD `relayd` for TCP forwarding (Layer 4) instead of TLS termination -* Use cert-manager with self-signed certificates for LAN TLS offloading via Traefik -* Achieve zero-downtime CARP failover (tested and validated) - -This enhancement complements the existing external access method without requiring MetalLB or other LoadBalancer implementations. The f3s infrastructure now has three distinct TLS offloaders: OpenBSD relayd for external traffic, Traefik (k3s) for LAN HTTPS, and stunnel for NFS-over-TLS. - -For the full implementation details, see: - -=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments - ## Persistent storage recap All observability components need persistent storage so that metrics and logs survive pod restarts. As covered in Part 6 of this series, the cluster uses NFS-backed persistent volumes: -- cgit v1.2.3