Home | Markdown | Gemini

f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD

Table of Contents

• f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD
• ⇢ Introduction
• ⇢ What is GitOps?
• ⇢ What is ArgoCD?
• ⇢ Why ArgoCD for f3s?
• ⇢ Deploying ArgoCD
• ⇢ ⇢ Prerequisites
• ⇢ ⇢ Installing ArgoCD
• ⇢ ⇢ Accessing ArgoCD
• ⇢ ArgoCD Application Structure
• ⇢ Repository Organization
• ⇢ ⇢ Migration Phases
• ⇢ Example Migration: Miniflux
• ⇢ ⇢ Before: Imperative Helm deployment
• ⇢ ⇢ After: Declarative GitOps with ArgoCD
• ⇢ ⇢ Migration procedure
• ArgoCD detects change within 3 minutes and syncs automatically
• ArgoCD detects drift within 3 minutes
• ArgoCD automatically rolls back to the previous state
• Temporarily point ArgoCD at the feature branch
• Verify changes in ArgoCD Web UI
• If good: merge to master
• If bad: revert the patch
• root-monitoring.yaml
• Root app deploys all 21 applications automatically
• Or apply by namespace

Introduction

• No single source of truth: The cluster state depends on which commands were run and when
• Manual synchronization: Every change requires manually running commands
• Drift detection is hard: No easy way to know if cluster state matches the desired configuration
• Rollback complexity: Rolling back changes means re-running old Helm commands
• No audit trail: Hard to track who changed what and when

What is GitOps?

• Declarative: The system's desired state is described declaratively (YAML manifests, Helm values)
• Versioned and immutable: All changes are committed to Git, providing a complete history
• Pulled automatically: An agent in the cluster continuously pulls the desired state from Git
• Continuously reconciled: The agent ensures the actual state matches the desired state, automatically correcting drift

• 1. All manifests, Helm charts, and configuration live in a Git repository
• 2. A tool (ArgoCD in our case) watches the repository
• 3. When changes are pushed to Git, ArgoCD automatically applies them to the cluster
• 4. If someone manually changes resources in the cluster, ArgoCD detects the drift and can automatically revert it

What is ArgoCD?

• Automated deployment: Monitors Git repositories and automatically syncs changes to the cluster
• Application definitions: Defines applications as CRDs (Custom Resource Definitions)
• Health assessment: Understands Kubernetes resources and can determine if an application is healthy
• Web UI and CLI: Provides both a web interface and command-line tool for managing applications
• RBAC: Role-based access control for team collaboration
• SSO integration: Can integrate with existing authentication systems
• Multi-cluster support: Can manage applications across multiple Kubernetes clusters
• Sync waves and hooks: Control the order of resource deployment and run jobs at specific lifecycle points

Why ArgoCD for f3s?

• 1. Bootstrapping a new k3s cluster
• 2. Installing ArgoCD
• 3. Pointing ArgoCD at the Git repository
• 4. All applications automatically deploy to the desired state

Deploying ArgoCD

Prerequisites

$ kubectl create namespace cicd
namespace/cicd created

Installing ArgoCD

$ cd conf/f3s/argocd
$ just install
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd \
    --namespace cicd \
    --version 7.7.12 \
    -f values.yaml
NAME: argocd
LAST DEPLOYED: ...
NAMESPACE: cicd
STATUS: deployed

repoServer:
  volumes:
    - name: repo-cache
      persistentVolumeClaim:
        claimName: argocd-repo-cache-pvc
  volumeMounts:
    - name: repo-cache
      mountPath: /tmp

configs:
  secret:
    createSecret: false

$ ARGOCD_ADMIN_PASSWORD=$(pwgen -s 32 1)
$ BCRYPT_HASH=$(htpasswd -nbBC 10 "" "$ARGOCD_ADMIN_PASSWORD" | tr -d ':\n' | sed 's/$2y/$2a/')
$ kubectl create secret generic argocd-secret \
    --from-literal=admin.password="$BCRYPT_HASH" \
    -n cicd
$ echo "ArgoCD admin password: $ARGOCD_ADMIN_PASSWORD"

server:
  insecure: true

Accessing ArgoCD

$ kubectl get pods -n cicd
NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          45d
argocd-applicationset-controller-66d6b9b8f4-vhm9k   1/1     Running   0          45d
argocd-dex-server-7fb556b7dd-xjr2l                  1/1     Running   0          45d
argocd-notifications-controller-6d8dd4c5f5-b8vwl    1/1     Running   0          45d
argocd-redis-77b8d6c6d4-mz9hg                       1/1     Running   0          45d
argocd-repo-server-5f98f77b97-8xtcq                 1/1     Running   0          45d
argocd-server-6b9c4b4f8d-kxw7p                      1/1     Running   0          45d

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: cicd
  annotations:
    spec.ingressClassName: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
    - host: argocd.f3s.foo.zone
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80

$ argocd login argocd.f3s.foo.zone
$ argocd app list

ArgoCD Application Structure

• Source: Where the manifests live (Git repo, Helm chart repository, or both)
• Destination: Which cluster and namespace to deploy to

ind: Application
metadata:
  name: miniflux
  namespace: cicd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://codeberg.org/snonux/conf.git
    
    targetRevision: master
    path: f3s/miniflux/helm-chart
  destination:
  
    server: https://kubernetes.default.svc
    namespace: services
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m

• source.path: Points to the Helm chart directory in Git
• destination.namespace: Where to deploy the application
• syncPolicy.automated.prune: Delete resources that are removed from Git
• syncPolicy.automated.selfHeal: Automatically revert manual changes in the cluster
• finalizers: Ensures ArgoCD deletes all resources when the Application is deleted

Repository Organization

/home/paul/git/conf/f3s/
├── argocd-apps/               # ArgoCD Application manifests (organized by namespace)
│   ├── README.md              # Documentation of structure
│   ├── monitoring/            # Observability stack (6 apps)
│   │   ├── alloy.yaml
│   │   ├── grafana-ingress.yaml
│   │   ├── loki.yaml
│   │   ├── prometheus.yaml
│   │   ├── pushgateway.yaml
│   │   └── tempo.yaml
│   ├── services/              # User-facing applications (13 apps)
│   │   ├── anki-sync-server.yaml
│   │   ├── audiobookshelf.yaml
│   │   ├── filebrowser.yaml
│   │   ├── immich.yaml
│   │   ├── keybr.yaml
│   │   ├── kobo-sync-server.yaml
│   │   ├── miniflux.yaml
│   │   ├── opodsync.yaml
│   │   ├── radicale.yaml
│   │   ├── syncthing.yaml
│   │   ├── tracing-demo.yaml
│   │   ├── wallabag.yaml
│   │   └── webdav.yaml
│   ├── infra/                 # Infrastructure services (1 app)
│   │   └── registry.yaml
│   └── test/                  # Test/example applications (1 app)
│       └── example-apache-volume-claim.yaml
├── miniflux/                  # Application directories (unchanged)
│   ├── helm-chart/
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   └── templates/
│   └── Justfile               # Updated for ArgoCD
├── prometheus/
│   ├── manifests/             # NEW: Additional manifests
│   │   ├── persistent-volumes.yaml
│   │   ├── grafana-restart-hook.yaml
│   │   ├── freebsd-recording-rules.yaml
│   │   └── ...
│   └── Justfile               # Updated for ArgoCD
└── ...

• monitoring/: 6 observability applications
• services/: 13 user-facing applications
• infra/: 1 infrastructure application (registry)
• test/: 1 test application

Migration Phases

• 1. Create Application manifest in argocd-apps/
• 2. Apply with kubectl apply -f argocd-apps/<app>.yaml
• 3. Verify sync status: argocd app get <app>
• 4. Update Justfile to use ArgoCD commands

• registry (Docker image registry)
• pushgateway (Prometheus metrics ingestion)
• immich (photo management with complex dependencies)

• tempo (distributed tracing)
• loki (log aggregation)
• alloy (log collection)
• prometheus (metrics and monitoring)

• grafana-ingress (separate ingress for Grafana)

Example Migration: Miniflux

Before: Imperative Helm deployment

NAMESPACE := "services"
APP_NAME := "miniflux"

install:
    kubectl apply -f helm-chart/persistent-volumes.yaml
    helm install {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}

upgrade:
    helm upgrade {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}

uninstall:
    helm uninstall {{APP_NAME}} --namespace {{NAMESPACE}}
    kubectl delete -f helm-chart/persistent-volumes.yaml

status:
    @kubectl get all -n {{NAMESPACE}} -l app={{APP_NAME}}

After: Declarative GitOps with ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: miniflux
  namespace: cicd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://codeberg.org/snonux/conf.git
    targetRevision: master
    path: f3s/miniflux/helm-chart
  destination:
    server: https://kubernetes.default.svc
    namespace: services
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m

NAMESPACE := "services"
APP_NAME := "miniflux"

status:
    @echo "=== Pods ==="
    @kubectl get pods -n {{NAMESPACE}} -l app={{APP_NAME}}
    @echo ""
    @echo "=== Services ==="
    @kubectl get svc -n {{NAMESPACE}} -l app={{APP_NAME}}
    @echo ""
    @echo "=== ArgoCD Status ==="
    @kubectl get application {{APP_NAME}} -n cicd -o jsonpath='Sync: {.status.sync.status}, Health: {.status.health.status}' 2>/dev/null && echo ""

sync:
    @echo "Triggering ArgoCD sync..."
    @kubectl annotate application {{APP_NAME}} -n cicd argocd.argoproj.io/refresh=normal --overwrite
    @sleep 2
    @kubectl get application {{APP_NAME}} -n cicd -o jsonpath='Sync: {.status.sync.status}, Health: {.status.health.status}' && echo ""

argocd-status:
    argocd app get {{APP_NAME}} --core

logs:
    kubectl logs -n {{NAMESPACE}} -l app={{APP_NAME}} --tail=100 -f

Migration procedure

$ helm get values miniflux -n services > /tmp/miniflux-backup-values.yaml
$ kubectl get all,ingress -n services -o yaml > /tmp/miniflux-backup.yaml

$ kubectl apply -f argocd-apps/services/miniflux.yaml
application.argoproj.io/miniflux created

$ argocd app get miniflux
Name:               miniflux
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          services
URL:                https://argocd.f3s.foo.zone/applications/miniflux
Repo:               https://codeberg.org/snonux/conf.git
Target:             master
Path:               f3s/miniflux/helm-chart
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to master (4e3c216)
Health Status:      Healthy

$ kubectl get pods -n services -l app=miniflux -w
NAME                                READY   STATUS    RESTARTS   AGE
miniflux-postgres-556444cb8d-xvv2p  1/1     Running   0          54d
``

5. Test the application:

6. Update Justfile and commit changes

Total time: 10 minutes. Zero downtime.

## Complex Migration: Prometheus with Multi-Source


The Prometheus migration was more complex because it combines:
* Upstream Helm chart (kube-prometheus-stack)
* Custom manifests (PersistentVolumes, recording rules, dashboards)
* Sync hooks (PostSync job to restart Grafana)

The `prometheus/manifests/` directory contains:

### Sync Waves and Hooks

ArgoCD allows controlling the order of resource deployment using sync waves (the `argocd.argoproj.io/sync-wave` annotation):

* Wave 0: Infrastructure (PersistentVolumes, RBAC)
* Wave 1: Configuration (Secrets, ConfigMaps)
* Wave 3: Recording rules (PrometheusRule CRDs)
* Wave 4: Dashboards (ConfigMaps with `grafana_dashboard: '1'` label)
* Wave 10: PostSync hooks (Jobs that run after everything else)

The Grafana restart hook ensures Grafana reloads datasources after they're updated:

This *he manual step in the old Justfile that required running `kubectl delete pod` after every upgrade.

## Migration *sults

After * all 21 applications to ArgoCD:

All 21 applications: Synced and Healthy.

ArgoCD Web UI:

=> ./f3s-kubernetes-with-freebsd-part-X/argocd-apps-list.png ArgoCD Applications List

=> ./f3s-kubernetes-with-freebsd-part-X/argocd-app-tree.png ArgoCD Application Resource Tree

## Benefits Realized

### 1. Single Source of Truth

The Git repository at `https://codeberg.org/snonux/conf` now contains the complete cluster configuration. Anyone can clone it and see exactly what's deployed:

### 2. Automatic Synchronization

Push to Git, and changes deploy automatically:

ArgoCD detects change within 3 minutes and syncs automatically

No need to SSH to a workstation, pull the repo, and run `just upgrade`.

### 3. Drift Detection and Self-Healing

If someone manually changes a resource in the cluster, ArgoCD detects it:

ArgoCD detects drift within 3 minutes

With `selfHeal: true`, ArgoCD automatically reverts the change back to 2 replicas (the value in Git).

### 4. Easy Rollbacks

To rollback a change:

ArgoCD automatically rolls back to the previous state

Or rollback to a specific commit:

### 5. Disaster Recovery

If the entire cluster is destroyed, recovery is straightforward:

1. Bootstrap a new k3s cluster
2. Create namespaces
3. Install ArgoCD
4. Apply all Application manifests:

5. ArgoCD deploys all 21 applications to their desired state

Total recovery time: ~30 minutes (mostly waiting for pods to pull images and start).

### 6. Documentation by Default

The Application manifests serve as documentation:

* Which Helm chart version is deployed? → Check `targetRevision`
* What custom values are configured? → Check `valuesObject`
* Which namespace does this deploy to? → Check `destination.namespace`
* Is auto-sync enabled? → Check `syncPolicy.automated`

No more guessing or checking `helm list` output.

### 7. Safe Experimentation

Create a feature branch, make changes, and preview them:

Temporarily point ArgoCD at the feature branch

Verify changes in ArgoCD Web UI

If good: merge to master

If bad: revert the patch

## Challenges and Solutions

### Challenge 1: Helm Release Adoption

When creating an Application for an existing Helm release, ArgoCD needs to "adopt" the resources. This failed initially with errors like:

Solution: For existing Helm releases, I first ensured the Application manifest matched the current Helm values exactly. ArgoCD then recognized the resources were already in the desired state and adopted them without re-deploying.

### Challenge 2: Persistent Volumes Not Tracked by Helm

PersistentVolumes are cluster-scoped resources, not namespace-scoped. Many of my Helm charts created PVs using `kubectl apply -f persistent-volumes.yaml` outside of Helm.

Solution: For simple apps, I moved the PV definitions into the Helm chart templates. For complex apps (like Prometheus), I used the multi-source pattern with PVs in the `manifests/` directory with sync wave 0.

### Challenge 3: Secrets Management

ArgoCD stores Application manifests in Git, but secrets shouldn't be committed in plaintext.

Solution (current): Secrets are created manually with `kubectl create secret` and referenced by the Helm charts. The secrets themselves aren't managed by ArgoCD.

Future enhancement: Migrate to External Secrets Operator (ESO) to manage secrets declaratively while storing the actual secrets in a separate backend (Kubernetes secrets in a separate namespace, or eventually Vault).

### Challenge 4: Grafana Not Reloading Datasources

After updating the Grafana datasources ConfigMap, Grafana wouldn't detect the changes until pods were manually deleted.

Solution: Created a PostSync hook that automatically restarts Grafana pods after every ArgoCD sync. This runs as a Kubernetes Job in sync wave 10, ensuring it executes after all other resources are deployed.

### Challenge 5: Prometheus With Multiple Sources

Prometheus needed both the upstream Helm chart and custom manifests (recording rules, dashboards, PVs).

Solution: Used ArgoCD's multi-source feature to combine:
* Helm chart from `prometheus-community.github.io/helm-charts`
* Additional manifests from `codeberg.org/snonux/conf.git` at path `f3s/prometheus/manifests`

This keeps the upstream chart cleanly separated from custom configuration.

### Challenge 6: Sync Ordering for Prometheus

Prometheus resources have dependencies:
* PVs before PVCs
* Secrets before Prometheus Operator
* PrometheusRule CRDs before Prometheus Operator can process them
* Grafana must be running before the restart hook executes

Solution: Added sync wave annotations to all resources in `prometheus/manifests/`:
* Wave 0: PVs, RBAC
* Wave 1: Secrets, ConfigMaps
* Wave 3: PrometheusRule CRDs (recording rules)
* Wave 4: Dashboard ConfigMaps
* Wave 10: PostSync hook (Grafana restart)

ArgoCD deploys resources in wave order, ensuring correct sequencing.

## Justfile Evolution

The Justfiles evolved from deployment tools to utility scripts:

Before (Helm deployment):

After (ArgoCD utilities):

The Justfiles now provide:
* `status`: Quick health check
* `sync`: Force immediate ArgoCD sync (instead of waiting 3 minutes)
* `argocd-status`: Detailed ArgoCD application status
* `logs`: Tail application logs
* Application-specific utilities (e.g., `port-forward`, `restart`)

## Lessons Learned

1. Incremental migration is safer than big-bang: Migrating one app at a time allowed me to validate the pattern and fix issues before they affected all apps.

2. Start with simple apps: The first migration (simple services) established the basic pattern. Complex apps (Prometheus) came later after the pattern was proven.

3. Sync waves are essential for complex apps: Without sync waves, resources deployed in random order and caused failures. Proper ordering eliminated all deployment issues.

4. Multi-source is powerful: Combining upstream Helm charts with custom manifests keeps configuration clean and maintainable.

5. PostSync hooks replace manual steps: The Grafana restart hook eliminated a manual step that was easy to forget.

6. Documentation in Git is better than tribal knowledge: The Application manifests document exactly what's deployed and how. No more "let me check my shell history to remember how I deployed this."

7. Self-healing prevents configuration drift: Multiple times I've manually tweaked something for debugging, forgotten about it, and ArgoCD automatically reverted it back to the desired state.

8. ArgoCD Web UI is invaluable: Seeing the resource tree, sync status, and health status at a glance is much better than running multiple `kubectl` commands.

## Future Improvements

### 1. External Secrets Operator

Currently, secrets are manually created with `kubectl create secret`. This works but isn't declarative. Plan:

* Deploy External Secrets Operator (ESO)
* Store actual secrets in a Kubernetes Secret in a separate `secrets` namespace
* Create ExternalSecret CRDs that reference the backend secrets
* ArgoCD manages the ExternalSecret CRDs, ESO creates the actual Secrets

This makes secrets declarative while keeping them out of Git.

### 2. ApplicationSet for Similar Apps

Many apps have nearly identical Application manifests (miniflux, freshrss, wallabag, etc.). ArgoCD ApplicationSets can generate multiple Applications from a template:

One ApplicationSet could replace 10+ individual Application manifests.

### 3. App-of-Apps Pattern

Currently, all Application manifests are applied manually with `kubectl apply -f argocd-apps/ -R`. An alternative is the "app-of-apps" pattern:

Create a root Application that deploys all other Applications. With the namespace-organized structure, this could be done per-namespace or for the entire cluster:

Or create separate root apps per namespace:

root-monitoring.yaml

Then disaster recovery becomes:

Root app deploys all 21 applications automatically

Or apply by namespace

### 4. ArgoCD Image Updater

For applications with custom Docker images (like the registry, tracing-demo), ArgoCD Image Updater can automatically update the image tag in Git when a new image is pushed:

Generated with Gemtexter 3.0.1-develop | served by OpenBSD/relayd(8)+httpd(8) | Site Mirrors
Webring: previous | shring | next