From eda47468cf837c0d58b5fa1053a65496c60bd31b Mon Sep 17 00:00:00 2001
From: Paul Buetow
-
Technical references
I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order:
-
Self-development and soft-skills books
In random order:
-
Here are notes of mine for some of the books
@@ -164,31 +164,31 @@
Some of these were in-person with exams; others were online learning lectures only. In random order:
-
Technical guides
These are not whole books, but guides (smaller or larger) which I found very useful. in random order:
+
Podcasts
@@ -197,61 +197,61 @@
In random order:
-
Podcasts I liked
I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests.
-
Newsletters I like
This is a mix of tech and non-tech newsletters I am subscribed to. In random order:
-
Magazines I like(d)
This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order:
-
Formal education
diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
index 70a8e44f..e17fdf7d 100644
--- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
+++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
@@ -33,6 +33,7 @@
https://k3s.io
+Important Note: GitOps Migration
+
+**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests and Helm charts in the repository have been reorganized for ArgoCD-based continuous deployment.
+
+**To view the exact manifests and charts as they existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:
+
+
+$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3 # Last commit before ArgoCD migration
+$ cd f3s/
+
+
+**Current master branch** contains the ArgoCD-managed versions with:
+- Application manifests organized under argocd-apps/{monitoring,services,infra,test}/
+- Additional resources under */manifests/ directories (e.g., prometheus/manifests/)
+- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands
+
+The deployment concepts and architecture remain the same—only the deployment method changed from imperative (helm install/upgrade) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series.
+
Updating
Before proceeding, I bring all systems involved up-to-date. On all three Rocky Linux 9 boxes r0, r1, and r2:
@@ -878,21 +902,7 @@ http://www.gnu.org/software/src-highlite -->
codeberg.org/snonux/conf/f3s
-**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests and Helm charts in the repository have been reorganized for declarative deployment. To view the exact manifests and charts as they existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:
-
-
-$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3 # Last commit before ArgoCD migration
-$ cd f3s/
-
-
-The current master branch contains the ArgoCD-managed versions with manifests organized under argocd-apps/ and */manifests/ directories.
-
-Within that repo, the examples/conf/f3s/registry/ directory contains the Helm chart, a Justfile, and a detailed README. Here's the condensed walkthrough I used to roll out the registry with Helm.
+Within that repo, the f3s/registry/ directory contains the Helm chart, a Justfile, and a detailed README. Here's the condensed walkthrough I used to roll out the registry with Helm.
Prepare the NFS-backed storage
diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
index 00fcadc9..0295f7dc 100644
--- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
+++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
@@ -33,6 +33,7 @@
All manifests for the f3s stack live in my configuration repository:
-codeberg.org/snonux/conf/f3s
+codeberg.org/snonux/conf/f3s
+
+Important Note: GitOps Migration
+
+**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for ArgoCD-based continuous deployment.
+
+**To view the exact configuration as it existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:
+
+
+$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3 # Last commit before ArgoCD migration
+$ cd f3s/prometheus/
+
+
+**Current master branch** contains the ArgoCD-managed versions with:
+- Application manifests organized under argocd-apps/{monitoring,services,infra,test}/
+- Resources organized under prometheus/manifests/, loki/, etc.
+- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands
+
+The deployment concepts and architecture remain the same—only the deployment method changed from imperative (helm install/upgrade) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series.
Persistent storage recap
@@ -107,20 +131,6 @@ namespace/monitoring created
Installing Prometheus and Grafana
-**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for declarative deployment. To view the exact configuration as it existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:
-
-
-$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3 # Last commit before ArgoCD migration
-$ cd f3s/prometheus/
-
-
-The current master branch contains the ArgoCD-managed versions with Application manifests under argocd-apps/ and resources organized under prometheus/manifests/, loki/, etc. The Justfiles have been updated to trigger ArgoCD syncs instead of direct Helm commands.
-
Prometheus and Grafana are deployed together using the kube-prometheus-stack Helm chart from the Prometheus community. This chart bundles Prometheus, Grafana, Alertmanager, and various exporters (Node Exporter, Kube State Metrics) into a single deployment. Ill explain what each component does in detail later when we look at the running pods.
Prerequisites
diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
new file mode 100644
index 00000000..5d58d4c5
--- /dev/null
+++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
@@ -0,0 +1,1227 @@
+
+
+
+
+f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD
+
+DRAFT - Not yet published
+
+This is part X of the f3s series for my self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.
+
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
+2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
+2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
+2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+
+
+
+Table of Contents
+
+
+
+Introduction
+
+In the previous posts, I deployed applications to the k3s cluster using Helm charts and Justfiles—running just install or just upgrade to imperatively push changes to the cluster. While this approach works, it has several drawbacks:
+
+
+
+This blog post covers the migration from imperative Helm deployments to declarative GitOps using ArgoCD. After this migration, the Git repository becomes the single source of truth, and ArgoCD automatically ensures the cluster matches what's defined in Git.
+
+What is GitOps?
+
+GitOps is an operational framework that applies DevOps best practices—like version control, collaboration, and CI/CD—to infrastructure automation. The core idea is simple: the entire desired state of your infrastructure is stored in Git, and automated processes ensure the actual state matches the desired state.
+
+Key principles:
+
+
+
+For Kubernetes, this means:
+
+1. All manifests, Helm charts, and configuration live in a Git repository
+2. A tool (ArgoCD in our case) watches the repository
+3. When changes are pushed to Git, ArgoCD automatically applies them to the cluster
+4. If someone manually changes resources in the cluster, ArgoCD detects the drift and can automatically revert it
+
+What is ArgoCD?
+
+ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It's implemented as a Kubernetes controller that continuously monitors running applications and compares the current, live state against the desired target state defined in Git.
+
+ArgoCD Documentation
+
+Key features:
+
+
+
+Why ArgoCD for f3s?
+
+For a home lab cluster, ArgoCD provides several benefits:
+
+**Disaster recovery**: If the entire cluster is lost, I can rebuild it by:
+1. Bootstrapping a new k3s cluster
+2. Installing ArgoCD
+3. Pointing ArgoCD at the Git repository
+4. All applications automatically deploy to the desired state
+
+**Experimentation safety**: I can test changes in a separate Git branch without affecting the running cluster. Once validated, merge to master and ArgoCD applies the changes.
+
+**Drift detection**: If I manually change something in the cluster (for debugging), ArgoCD shows the difference and can automatically revert it.
+
+**Declarative configuration**: The Git repository documents the entire cluster configuration. No need to remember which just commands to run or in which order.
+
+**Automatic sync**: Push to Git, and changes deploy automatically. No need to SSH to a workstation and run Helm commands.
+
+Deploying ArgoCD
+
+ArgoCD itself runs as a set of Kubernetes resources in the cluster. The official installation method uses kubectl apply, which is fitting—ArgoCD manages everything else via GitOps, but ArgoCD itself needs a bootstrap.
+
+Prerequisites
+
+Create the cicd namespace where ArgoCD will run:
+
+
+$ kubectl create namespace cicd
+namespace/cicd created
+
+
+Installing ArgoCD
+
+The ArgoCD installation lives in the configuration repository:
+
+codeberg.org/snonux/conf/f3s/argocd
+
+I deployed ArgoCD using Helm instead of the raw manifests. This provides easier upgrades and customization. The installation is managed via a Justfile:
+
+
+$ cd conf/f3s/argocd
+$ just install
+helm repo add argo https://argoproj.github.io/argo-helm
+helm repo update
+helm install argocd argo/argo-cd \
+ --namespace cicd \
+ --version 7.7.12 \
+ -f values.yaml
+NAME: argocd
+LAST DEPLOYED: ...
+NAMESPACE: cicd
+STATUS: deployed
+
+
+The values.yaml file configures several important aspects:
+
+**Persistent storage for the repo-server**: ArgoCD clones Git repositories to cache them locally. I configured a persistent volume so the cache survives pod restarts:
+
+
+repoServer:
+ volumes:
+ - name: repo-cache
+ persistentVolumeClaim:
+ claimName: argocd-repo-cache-pvc
+ volumeMounts:
+ - name: repo-cache
+ mountPath: /tmp
+
+
+**Admin password preservation**: By default, the admin password is auto-generated and stored in a secret. To ensure it persists across Helm upgrades:
+
+
+configs:
+ secret:
+ createSecret: false
+
+
+I manually created the secret before installation:
+
+
+$ ARGOCD_ADMIN_PASSWORD=$(pwgen -s 32 1)
+$ BCRYPT_HASH=$(htpasswd -nbBC 10 "" "$ARGOCD_ADMIN_PASSWORD" | tr -d ':\n' | sed 's/$2y/$2a/')
+$ kubectl create secret generic argocd-secret \
+ --from-literal=admin.password="$BCRYPT_HASH" \
+ -n cicd
+$ echo "ArgoCD admin password: $ARGOCD_ADMIN_PASSWORD"
+
+
+**Server configuration**: Enabled insecure mode since TLS is handled by the OpenBSD edge relays:
+
+
+server:
+ insecure: true
+
+
+Accessing ArgoCD
+
+After deployment, ArgoCD runs several pods in the cicd namespace:
+
+
+$ kubectl get pods -n cicd
+NAME READY STATUS RESTARTS AGE
+argocd-application-controller-0 1/1 Running 0 45d
+argocd-applicationset-controller-66d6b9b8f4-vhm9k 1/1 Running 0 45d
+argocd-dex-server-7fb556b7dd-xjr2l 1/1 Running 0 45d
+argocd-notifications-controller-6d8dd4c5f5-b8vwl 1/1 Running 0 45d
+argocd-redis-77b8d6c6d4-mz9hg 1/1 Running 0 45d
+argocd-repo-server-5f98f77b97-8xtcq 1/1 Running 0 45d
+argocd-server-6b9c4b4f8d-kxw7p 1/1 Running 0 45d
+
+
+I created an ingress to expose the ArgoCD web UI:
+
+
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: argocd-server-ingress
+ namespace: cicd
+ annotations:
+ spec.ingressClassName: traefik
+ traefik.ingress.kubernetes.io/router.entrypoints: web
+spec:
+ rules:
+ - host: argocd.f3s.buetow.org
+ http:
+ paths:
+ - path: /
+ pathType: Prefix
+ backend:
+ service:
+ name: argocd-server
+ port:
+ number: 80
+
+
+Following the same pattern as other services, the OpenBSD edge relays terminate TLS and forward traffic through WireGuard to the cluster. ArgoCD is now accessible at:
+
+ArgoCD Web UI
+
+The ArgoCD CLI can also be used for operations:
+
+
+$ argocd login argocd.f3s.buetow.org
+$ argocd app list
+
+
+ArgoCD Application Structure
+
+ArgoCD uses a CRD called Application to define what should be deployed. Each application specifies:
+
+
+
+Here's a simple example for the miniflux application:
+
+
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+ name: miniflux
+ namespace: cicd
+ finalizers:
+ - resources-finalizer.argocd.argoproj.io
+spec:
+ project: default
+ source:
+ repoURL: https://codeberg.org/snonux/conf.git
+ targetRevision: master
+ path: f3s/miniflux/helm-chart
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: services
+ syncPolicy:
+ automated:
+ prune: true
+ selfHeal: true
+ syncOptions:
+ - CreateNamespace=false
+ retry:
+ limit: 3
+ backoff:
+ duration: 5s
+ factor: 2
+ maxDuration: 1m
+
+
+Key fields:
+
+
+
+Repository Organization
+
+I reorganized the configuration repository to support GitOps:
+
+
+/home/paul/git/conf/f3s/
+├── argocd-apps/ # ArgoCD Application manifests (organized by namespace)
+│ ├── README.md # Documentation of structure
+│ ├── monitoring/ # Observability stack (6 apps)
+│ │ ├── alloy.yaml
+│ │ ├── grafana-ingress.yaml
+│ │ ├── loki.yaml
+│ │ ├── prometheus.yaml
+│ │ ├── pushgateway.yaml
+│ │ └── tempo.yaml
+│ ├── services/ # User-facing applications (13 apps)
+│ │ ├── anki-sync-server.yaml
+│ │ ├── audiobookshelf.yaml
+│ │ ├── filebrowser.yaml
+│ │ ├── immich.yaml
+│ │ ├── keybr.yaml
+│ │ ├── kobo-sync-server.yaml
+│ │ ├── miniflux.yaml
+│ │ ├── opodsync.yaml
+│ │ ├── radicale.yaml
+│ │ ├── syncthing.yaml
+│ │ ├── tracing-demo.yaml
+│ │ ├── wallabag.yaml
+│ │ └── webdav.yaml
+│ ├── infra/ # Infrastructure services (1 app)
+│ │ └── registry.yaml
+│ └── test/ # Test/example applications (1 app)
+│ └── example-apache-volume-claim.yaml
+├── miniflux/ # Application directories (unchanged)
+│ ├── helm-chart/
+│ │ ├── Chart.yaml
+│ │ ├── values.yaml
+│ │ └── templates/
+│ └── Justfile # Updated for ArgoCD
+├── prometheus/
+│ ├── manifests/ # NEW: Additional manifests
+│ │ ├── persistent-volumes.yaml
+│ │ ├── grafana-restart-hook.yaml
+│ │ ├── freebsd-recording-rules.yaml
+│ │ └── ...
+│ └── Justfile # Updated for ArgoCD
+└── ...
+
+
+The application directories (miniflux, prometheus, etc.) remained mostly unchanged—ArgoCD references the same Helm charts. The main additions:
+
+1. **argocd-apps/**: Application manifests organized by Kubernetes namespace for better clarity
+ - monitoring/: 6 observability applications
+ - services/: 13 user-facing applications
+ - infra/: 1 infrastructure application (registry)
+ - test/: 1 test application
+2. ***/manifests/**: Additional Kubernetes manifests for complex apps (like Prometheus)
+3. **Justfiles updated**: Changed from helm install/upgrade to argocd app sync
+
+This organization makes it easy to apply all applications in a specific namespace or manage them independently.
+
+Migration Strategy: Incremental, One App at a Time
+
+Rather than attempting a "big bang" migration of all 21 applications at once, I migrated them incrementally:
+
+1. **Start with a simple app**: Validate the pattern with a low-risk application
+2. **Migrate in waves**: Group similar applications and migrate together
+3. **Validate thoroughly**: Ensure each app is healthy before moving to the next
+4. **Learn and iterate**: Apply lessons from earlier migrations to later ones
+
+This approach reduced risk and allowed me to refine the migration process.
+
+Migration Phases
+
+**Phase 1: Simple services** (13 apps)
+
+
+These apps have straightforward Helm charts with no complex dependencies. Pattern established:
+1. Create Application manifest in argocd-apps/
+2. Apply with kubectl apply -f argocd-apps/<app>.yaml
+3. Verify sync status: argocd app get <app>
+4. Update Justfile to use ArgoCD commands
+
+**Phase 2: Infrastructure apps** (3 apps)
+
+
+**Phase 3: Monitoring stack** (4 apps)
+
+
+**Phase 4: Monitoring addons** (1 app)
+
+
+Example Migration: Miniflux
+
+Let me walk through the migration of miniflux as a concrete example.
+
+Before: Imperative Helm deployment
+
+Original Justfile:
+
+
+NAMESPACE := "services"
+APP_NAME := "miniflux"
+
+install:
+ kubectl apply -f helm-chart/persistent-volumes.yaml
+ helm install {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}
+
+upgrade:
+ helm upgrade {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}
+
+uninstall:
+ helm uninstall {{APP_NAME}} --namespace {{NAMESPACE}}
+ kubectl delete -f helm-chart/persistent-volumes.yaml
+
+status:
+ @kubectl get all -n {{NAMESPACE}} -l app={{APP_NAME}}
+
+
+Workflow:
+1. Make changes to helm-chart/
+2. Run just upgrade
+3. Helm pushes changes to cluster
+
+After: Declarative GitOps with ArgoCD
+
+Created argocd-apps/services/miniflux.yaml:
+
+
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+ name: miniflux
+ namespace: cicd
+ finalizers:
+ - resources-finalizer.argocd.argoproj.io
+spec:
+ project: default
+ source:
+ repoURL: https://codeberg.org/snonux/conf.git
+ targetRevision: master
+ path: f3s/miniflux/helm-chart
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: services
+ syncPolicy:
+ automated:
+ prune: true
+ selfHeal: true
+ syncOptions:
+ - CreateNamespace=false
+ retry:
+ limit: 3
+ backoff:
+ duration: 5s
+ factor: 2
+ maxDuration: 1m
+
+
+Updated Justfile:
+
+
+NAMESPACE := "services"
+APP_NAME := "miniflux"
+
+status:
+ @echo "=== Pods ==="
+ @kubectl get pods -n {{NAMESPACE}} -l app={{APP_NAME}}
+ @echo ""
+ @echo "=== Services ==="
+ @kubectl get svc -n {{NAMESPACE}} -l app={{APP_NAME}}
+ @echo ""
+ @echo "=== ArgoCD Status ==="
+ @kubectl get application {{APP_NAME}} -n cicd -o jsonpath='Sync: {.status.sync.status}, Health: {.status.health.status}' 2>/dev/null && echo ""
+
+sync:
+ @echo "Triggering ArgoCD sync..."
+ @kubectl annotate application {{APP_NAME}} -n cicd argocd.argoproj.io/refresh=normal --overwrite
+ @sleep 2
+ @kubectl get application {{APP_NAME}} -n cicd -o jsonpath='Sync: {.status.sync.status}, Health: {.status.health.status}' && echo ""
+
+argocd-status:
+ argocd app get {{APP_NAME}} --core
+
+logs:
+ kubectl logs -n {{NAMESPACE}} -l app={{APP_NAME}} --tail=100 -f
+
+
+New workflow:
+1. Make changes to helm-chart/
+2. Commit and push to Git
+3. ArgoCD automatically detects and syncs changes
+4. (Optional) Run just sync to force immediate sync
+
+Migration procedure
+
+1. **Backup current state**:
+
+$ helm get values miniflux -n services > /tmp/miniflux-backup-values.yaml
+$ kubectl get all,ingress -n services -o yaml > /tmp/miniflux-backup.yaml
+
+
+2. **Create Application manifest**:
+
+$ kubectl apply -f argocd-apps/services/miniflux.yaml
+application.argoproj.io/miniflux created
+
+
+3. **Verify ArgoCD adopted the resources**:
+
+$ argocd app get miniflux
+Name: miniflux
+Project: default
+Server: https://kubernetes.default.svc
+Namespace: services
+URL: https://argocd.f3s.buetow.org/applications/miniflux
+Repo: https://codeberg.org/snonux/conf.git
+Target: master
+Path: f3s/miniflux/helm-chart
+SyncWindow: Sync Allowed
+Sync Policy: Automated (Prune)
+Sync Status: Synced to master (4e3c216)
+Health Status: Healthy
+
+
+4. **Monitor for issues**:
+
+$ kubectl get pods -n services -l app=miniflux -w
+NAME READY STATUS RESTARTS AGE
+miniflux-postgres-556444cb8d-xvv2p 1/1 Running 0 54d
+miniflux-server-85d7c64664-stmt9 1/1 Running 0 54d
+
+
+5. **Test the application**:
+
+$ curl -I https://flux.f3s.buetow.org
+HTTP/2 200
+
+
+6. **Update Justfile** and commit changes
+
+Total time: 10 minutes. Zero downtime.
+
+Complex Migration: Prometheus with Multi-Source
+
+The Prometheus migration was more complex because it combines:
+
+
+ArgoCD supports "multi-source" Applications that combine multiple sources:
+
+
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+ name: prometheus
+ namespace: cicd
+ finalizers:
+ - resources-finalizer.argocd.argoproj.io
+spec:
+ project: default
+ sources:
+ # Source 1: Upstream Helm chart from prometheus-community
+ - repoURL: https://prometheus-community.github.io/helm-charts
+ chart: kube-prometheus-stack
+ targetRevision: 55.5.0
+ helm:
+ releaseName: prometheus
+ valuesObject:
+ # Full Prometheus configuration embedded here
+ kubeEtcd:
+ enabled: true
+ endpoints:
+ - 192.168.2.120
+ - 192.168.2.121
+ - 192.168.2.122
+ # ... (hundreds of lines of configuration)
+
+ # Source 2: Additional manifests from Git repository
+ - repoURL: https://codeberg.org/snonux/conf.git
+ targetRevision: master
+ path: f3s/prometheus/manifests
+
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: monitoring
+
+ syncPolicy:
+ automated:
+ prune: false # Manual pruning for safety on complex stack
+ selfHeal: true
+ syncOptions:
+ - CreateNamespace=false
+ - ServerSideApply=true
+ retry:
+ limit: 3
+ backoff:
+ duration: 10s
+ factor: 2
+ maxDuration: 3m
+
+
+The prometheus/manifests/ directory contains:
+
+
+f3s/prometheus/manifests/
+├── persistent-volumes.yaml # Sync wave 0
+├── additional-scrape-configs-secret.yaml # Sync wave 1
+├── grafana-datasources-configmap.yaml # Sync wave 1
+├── freebsd-recording-rules.yaml # Sync wave 3
+├── openbsd-recording-rules.yaml # Sync wave 3
+├── zfs-recording-rules.yaml # Sync wave 3
+├── epimetheus-dashboard.yaml # Sync wave 4
+├── zfs-dashboards.yaml # Sync wave 4
+├── grafana-restart-hook.yaml # Sync wave 10 (PostSync)
+└── grafana-restart-rbac.yaml # Sync wave 0
+
+
+Sync Waves and Hooks
+
+ArgoCD allows controlling the order of resource deployment using sync waves (the argocd.argoproj.io/sync-wave annotation):
+
+
+
+The Grafana restart hook ensures Grafana reloads datasources after they're updated:
+
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: grafana-restart-hook
+ namespace: monitoring
+ annotations:
+ argocd.argoproj.io/hook: PostSync
+ argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
+ argocd.argoproj.io/sync-wave: "10"
+spec:
+ template:
+ spec:
+ serviceAccountName: grafana-restart-sa
+ restartPolicy: OnFailure
+ containers:
+ - name: kubectl
+ image: bitnami/kubectl:latest
+ command:
+ - /bin/sh
+ - -c
+ - |
+ kubectl wait --for=condition=available --timeout=300s deployment/prometheus-grafana -n monitoring || true
+ kubectl delete pod -n monitoring -l app.kubernetes.io/name=grafana --ignore-not-found=true
+ backoffLimit: 2
+
+
+This replaces the manual step in the old Justfile that required running kubectl delete pod after every upgrade.
+
+Migration Results
+
+After migrating all 21 applications to ArgoCD:
+
+
+$ argocd app list
+NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY
+alloy https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune
+anki-sync-server https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+audiobookshelf https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+example-apache https://kubernetes.default.svc test default Synced Healthy Auto-Prune
+example-apache-volume-... https://kubernetes.default.svc test default Synced Healthy Auto-Prune
+filebrowser https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+freshrss https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+grafana-ingress https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune
+immich https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+keybr https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+kobo-sync-server https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+loki https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune
+miniflux https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+opodsync https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+prometheus https://kubernetes.default.svc monitoring default Synced Healthy Auto
+pushgateway https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune
+radicale https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+registry https://kubernetes.default.svc infra default Synced Healthy Auto-Prune
+syncthing https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+tempo https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune
+wallabag https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+webdav https://kubernetes.default.svc services default Synced Healthy Auto-Prune
+
+
+All 21 applications: **Synced** and **Healthy**.
+
+ArgoCD Web UI:
+
+
+
+
+
+Benefits Realized
+
+1. Single Source of Truth
+
+The Git repository at https://codeberg.org/snonux/conf now contains the complete cluster configuration. Anyone can clone it and see exactly what's deployed:
+
+
+$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf/f3s
+$ ls argocd-apps/
+alloy.yaml anki-sync-server.yaml audiobookshelf.yaml ...
+
+
+2. Automatic Synchronization
+
+Push to Git, and changes deploy automatically:
+
+
+$ cd conf/f3s/miniflux/helm-chart
+$ vim values.yaml # Change replica count from 1 to 2
+$ git add values.yaml
+$ git commit -m "Scale miniflux to 2 replicas"
+$ git push
+# ArgoCD detects change within 3 minutes and syncs automatically
+
+
+No need to SSH to a workstation, pull the repo, and run just upgrade.
+
+3. Drift Detection and Self-Healing
+
+If someone manually changes a resource in the cluster, ArgoCD detects it:
+
+
+$ kubectl scale deployment miniflux-server -n services --replicas=3
+deployment.apps/miniflux-server scaled
+
+# ArgoCD detects drift within 3 minutes
+$ argocd app get miniflux
+...
+Sync Status: OutOfSync from master (4e3c216)
+
+
+With selfHeal: true, ArgoCD automatically reverts the change back to 2 replicas (the value in Git).
+
+4. Easy Rollbacks
+
+To rollback a change:
+
+
+$ git revert HEAD
+$ git push
+# ArgoCD automatically rolls back to the previous state
+
+
+Or rollback to a specific commit:
+
+
+$ argocd app rollback miniflux <revision-id>
+
+
+5. Disaster Recovery
+
+If the entire cluster is destroyed, recovery is straightforward:
+
+1. Bootstrap a new k3s cluster
+2. Create namespaces
+3. Install ArgoCD
+4. Apply all Application manifests:
+
+$ kubectl apply -f argocd-apps/
+
+5. ArgoCD deploys all 21 applications to their desired state
+
+Total recovery time: ~30 minutes (mostly waiting for pods to pull images and start).
+
+6. Documentation by Default
+
+The Application manifests serve as documentation:
+
+
+
+No more guessing or checking helm list output.
+
+7. Safe Experimentation
+
+Create a feature branch, make changes, and preview them:
+
+
+$ git checkout -b test-prometheus-upgrade
+$ vim argocd-apps/prometheus.yaml # Bump chart version
+$ git commit -am "Test Prometheus 56.0.0"
+$ git push origin test-prometheus-upgrade
+
+# Temporarily point ArgoCD at the feature branch
+$ kubectl patch application prometheus -n cicd \
+ --type merge \
+ -p '{"spec":{"source":{"targetRevision":"test-prometheus-upgrade"}}}'
+
+# Verify changes in ArgoCD Web UI
+# If good: merge to master
+# If bad: revert the patch
+
+
+Challenges and Solutions
+
+Challenge 1: Helm Release Adoption
+
+When creating an Application for an existing Helm release, ArgoCD needs to "adopt" the resources. This failed initially with errors like:
+
+
+The Helm operation failed with an error: release miniflux failed, and has been uninstalled due to atomic being set: timed out waiting for the condition
+
+
+**Solution**: For existing Helm releases, I first ensured the Application manifest matched the current Helm values exactly. ArgoCD then recognized the resources were already in the desired state and adopted them without re-deploying.
+
+Challenge 2: Persistent Volumes Not Tracked by Helm
+
+PersistentVolumes are cluster-scoped resources, not namespace-scoped. Many of my Helm charts created PVs using kubectl apply -f persistent-volumes.yaml outside of Helm.
+
+**Solution**: For simple apps, I moved the PV definitions into the Helm chart templates. For complex apps (like Prometheus), I used the multi-source pattern with PVs in the manifests/ directory with sync wave 0.
+
+Challenge 3: Secrets Management
+
+ArgoCD stores Application manifests in Git, but secrets shouldn't be committed in plaintext.
+
+**Solution (current)**: Secrets are created manually with kubectl create secret and referenced by the Helm charts. The secrets themselves aren't managed by ArgoCD.
+
+**Future enhancement**: Migrate to External Secrets Operator (ESO) to manage secrets declaratively while storing the actual secrets in a separate backend (Kubernetes secrets in a separate namespace, or eventually Vault).
+
+Challenge 4: Grafana Not Reloading Datasources
+
+After updating the Grafana datasources ConfigMap, Grafana wouldn't detect the changes until pods were manually deleted.
+
+**Solution**: Created a PostSync hook that automatically restarts Grafana pods after every ArgoCD sync. This runs as a Kubernetes Job in sync wave 10, ensuring it executes after all other resources are deployed.
+
+Challenge 5: Prometheus With Multiple Sources
+
+Prometheus needed both the upstream Helm chart and custom manifests (recording rules, dashboards, PVs).
+
+**Solution**: Used ArgoCD's multi-source feature to combine:
+
+
+This keeps the upstream chart cleanly separated from custom configuration.
+
+Challenge 6: Sync Ordering for Prometheus
+
+Prometheus resources have dependencies:
+
+
+**Solution**: Added sync wave annotations to all resources in prometheus/manifests/:
+
+
+ArgoCD deploys resources in wave order, ensuring correct sequencing.
+
+Justfile Evolution
+
+The Justfiles evolved from deployment tools to utility scripts:
+
+**Before (Helm deployment)**:
+
+install:
+ helm install miniflux ./helm-chart -n services
+
+upgrade:
+ helm upgrade miniflux ./helm-chart -n services
+
+uninstall:
+ helm uninstall miniflux -n services
+
+
+**After (ArgoCD utilities)**:
+
+status:
+ @kubectl get pods -n services -l app=miniflux
+ @kubectl get application miniflux -n cicd -o jsonpath='Sync: {.status.sync.status}, Health: {.status.health.status}'
+
+sync:
+ @kubectl annotate application miniflux -n cicd argocd.argoproj.io/refresh=normal --overwrite
+
+argocd-status:
+ argocd app get miniflux --core
+
+logs:
+ kubectl logs -n services -l app=miniflux --tail=100 -f
+
+
+The Justfiles now provide:
+
+
+Lessons Learned
+
+1. **Incremental migration is safer than big-bang**: Migrating one app at a time allowed me to validate the pattern and fix issues before they affected all apps.
+
+2. **Start with simple apps**: The first migration (simple services) established the basic pattern. Complex apps (Prometheus) came later after the pattern was proven.
+
+3. **Sync waves are essential for complex apps**: Without sync waves, resources deployed in random order and caused failures. Proper ordering eliminated all deployment issues.
+
+4. **Multi-source is powerful**: Combining upstream Helm charts with custom manifests keeps configuration clean and maintainable.
+
+5. **PostSync hooks replace manual steps**: The Grafana restart hook eliminated a manual step that was easy to forget.
+
+6. **Documentation in Git is better than tribal knowledge**: The Application manifests document exactly what's deployed and how. No more "let me check my shell history to remember how I deployed this."
+
+7. **Self-healing prevents configuration drift**: Multiple times I've manually tweaked something for debugging, forgotten about it, and ArgoCD automatically reverted it back to the desired state.
+
+8. **ArgoCD Web UI is invaluable**: Seeing the resource tree, sync status, and health status at a glance is much better than running multiple kubectl commands.
+
+Future Improvements
+
+1. External Secrets Operator
+
+Currently, secrets are manually created with kubectl create secret. This works but isn't declarative. Plan:
+
+
+
+This makes secrets declarative while keeping them out of Git.
+
+2. ApplicationSet for Similar Apps
+
+Many apps have nearly identical Application manifests (miniflux, freshrss, wallabag, etc.). ArgoCD ApplicationSets can generate multiple Applications from a template:
+
+
+apiVersion: argoproj.io/v1alpha1
+kind: ApplicationSet
+metadata:
+ name: simple-services
+ namespace: cicd
+spec:
+ generators:
+ - list:
+ elements:
+ - app: miniflux
+ - app: freshrss
+ - app: wallabag
+ template:
+ metadata:
+ name: '{{app}}'
+ spec:
+ project: default
+ source:
+ repoURL: https://codeberg.org/snonux/conf.git
+ targetRevision: master
+ path: 'f3s/{{app}}/helm-chart'
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: services
+ syncPolicy:
+ automated:
+ prune: true
+ selfHeal: true
+
+
+One ApplicationSet could replace 10+ individual Application manifests.
+
+3. App-of-Apps Pattern
+
+Currently, all Application manifests are applied manually with kubectl apply -f argocd-apps/ -R. An alternative is the "app-of-apps" pattern:
+
+Create a root Application that deploys all other Applications. With the namespace-organized structure, this could be done per-namespace or for the entire cluster:
+
+
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+ name: root
+ namespace: cicd
+spec:
+ source:
+ repoURL: https://codeberg.org/snonux/conf.git
+ targetRevision: master
+ path: f3s/argocd-apps
+ directory:
+ recurse: true # Recursively find all manifests in subdirectories
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: cicd
+ syncPolicy:
+ automated:
+ prune: true
+ selfHeal: true
+
+
+Or create separate root apps per namespace:
+
+
+# root-monitoring.yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+ name: root-monitoring
+ namespace: cicd
+spec:
+ source:
+ repoURL: https://codeberg.org/snonux/conf.git
+ targetRevision: master
+ path: f3s/argocd-apps/monitoring
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: cicd
+ syncPolicy:
+ automated:
+ prune: true
+ selfHeal: true
+
+
+Then disaster recovery becomes:
+
+$ kubectl apply -f root-app.yaml
+# Root app deploys all 21 applications automatically
+
+# Or apply by namespace
+$ kubectl apply -f root-monitoring.yaml
+$ kubectl apply -f root-services.yaml
+$ kubectl apply -f root-infra.yaml
+
+
+4. ArgoCD Image Updater
+
+For applications with custom Docker images (like the registry, tracing-demo), ArgoCD Image Updater can automatically update the image tag in Git when a new image is pushed:
+
+
+metadata:
+ annotations:
+ argocd-image-updater.argoproj.io/image-list: |
+ app=registry.f3s.buetow.org/miniflux:~^v
+ argocd-image-updater.argoproj.io/write-back-method: git
+
+
+When a new image registry.f3s.buetow.org/miniflux:v2.1.0 is pushed, Image Updater automatically:
+1. Updates the Helm values in Git
+2. Commits the change
+3. ArgoCD syncs the new image
+
+This creates a fully automated CI/CD pipeline.
+
+Summary
+
+Migrating from imperative Helm deployments to declarative GitOps with ArgoCD transformed how I manage the f3s cluster:
+
+**Before**:
+
+
+**After**:
+
+
+The migration took several days spread over a few weeks, migrating one application at a time. The result is a more maintainable, reliable, and recoverable cluster.
+
+All 21 applications are now managed via GitOps, with the configuration living in:
+
+codeberg.org/snonux/conf/f3s
+
+The ArgoCD Application manifests are organized by namespace:
+
+codeberg.org/snonux/conf/f3s/argocd-apps
+
+ArgoCD has become an essential part of the f3s infrastructure, and I can't imagine managing the cluster without it.
+
+Other *BSD-related posts:
+
+2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
+2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
+2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-04-01 KISS high-availability with OpenBSD
+2024-01-13 One reason why I love OpenBSD
+2022-10-30 Installing DTail on OpenBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+
+E-Mail your comments to paul@nospam.buetow.org
+
+Back to the main site
+
+
+
diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml
index 3a5d9294..ed0ef983 100644
--- a/gemfeed/atom.xml
+++ b/gemfeed/atom.xml
@@ -1,6 +1,6 @@
All manifests for the f3s stack live in my configuration repository:
-codeberg.org/snonux/conf/f3s
+codeberg.org/snonux/conf/f3s
+
+Important Note: GitOps Migration
+
+**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for ArgoCD-based continuous deployment.
+
+**To view the exact configuration as it existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:
+
+
+$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3 # Last commit before ArgoCD migration
+$ cd f3s/prometheus/
+
+
+**Current master branch** contains the ArgoCD-managed versions with:
+- Application manifests organized under argocd-apps/{monitoring,services,infra,test}/
+- Resources organized under prometheus/manifests/, loki/, etc.
+- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands
+
+The deployment concepts and architecture remain the same—only the deployment method changed from imperative (helm install/upgrade) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series.
Persistent storage recap
@@ -2386,20 +2410,6 @@ namespace/monitoring created
Installing Prometheus and Grafana
-**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for declarative deployment. To view the exact configuration as it existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:
-
-
-$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3 # Last commit before ArgoCD migration
-$ cd f3s/prometheus/
-
-
-The current master branch contains the ArgoCD-managed versions with Application manifests under argocd-apps/ and resources organized under prometheus/manifests/, loki/, etc. The Justfiles have been updated to trigger ArgoCD syncs instead of direct Helm commands.
-
Prometheus and Grafana are deployed together using the kube-prometheus-stack Helm chart from the Prometheus community. This chart bundles Prometheus, Grafana, Alertmanager, and various exporters (Node Exporter, Kube State Metrics) into a single deployment. Ill explain what each component does in detail later when we look at the running pods.
Prerequisites
@@ -3977,6 +3987,7 @@ p hash.values_at(:a, :c)
https://k3s.io
+Important Note: GitOps Migration
+
+**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests and Helm charts in the repository have been reorganized for ArgoCD-based continuous deployment.
+
+**To view the exact manifests and charts as they existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:
+
+
+$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3 # Last commit before ArgoCD migration
+$ cd f3s/
+
+
+**Current master branch** contains the ArgoCD-managed versions with:
+- Application manifests organized under argocd-apps/{monitoring,services,infra,test}/
+- Additional resources under */manifests/ directories (e.g., prometheus/manifests/)
+- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands
+
+The deployment concepts and architecture remain the same—only the deployment method changed from imperative (helm install/upgrade) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series.
+
Updating
Before proceeding, I bring all systems involved up-to-date. On all three Rocky Linux 9 boxes r0, r1, and r2:
@@ -4822,21 +4856,7 @@ http://www.gnu.org/software/src-highlite -->
codeberg.org/snonux/conf/f3s
-**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests and Helm charts in the repository have been reorganized for declarative deployment. To view the exact manifests and charts as they existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:
-
-
-$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3 # Last commit before ArgoCD migration
-$ cd f3s/
-
-
-The current master branch contains the ArgoCD-managed versions with manifests organized under argocd-apps/ and */manifests/ directories.
-
-Within that repo, the examples/conf/f3s/registry/ directory contains the Helm chart, a Justfile, and a detailed README. Here's the condensed walkthrough I used to roll out the registry with Helm.
+Within that repo, the f3s/registry/ directory contains the Helm chart, a Justfile, and a detailed README. Here's the condensed walkthrough I used to roll out the registry with Helm.
Prepare the NFS-backed storage
diff --git a/index.html b/index.html
index cdd66390..1071568f 100644
--- a/index.html
+++ b/index.html
@@ -13,7 +13,7 @@