Home | Markdown | Gemini

f3s: Kubernetes with FreeBSD - Part 9: Enabling etcd Metrics

Introduction

Important Note: GitOps Migration

$ git clone https://codeberg.org/snonux/conf.git
$ cd conf
$ git checkout 15a86f3  # Last commit before ArgoCD migration
$ cd f3s/prometheus/

• Application manifest: argocd-apps/monitoring/prometheus.yaml
• Multi-source Application combining upstream chart + custom manifests
• Justfile commands updated to trigger ArgoCD syncs instead of direct Helm commands

Enabling etcd metrics in k3s

etcd-expose-metrics: true

systemctl restart k3s

curl http://127.0.0.1:2381/metrics | grep etcd

Configuring Prometheus to scrape etcd

kubeEtcd:
  enabled: true
  endpoints:
    - 192.168.1.120
    - 192.168.1.121
    - 192.168.1.122
  service:
    enabled: true
    port: 2381
    targetPort: 2381

just upgrade

Verifying etcd metrics

kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
  -c prometheus -- wget -qO- 'http://localhost:9090/api/v1/query?query=etcd_server_has_leader' | \
  jq -r '.data.result[] | "\(.metric.instance): \(.value[1])"'

192.168.1.120:2381: 1
192.168.1.121:2381: 1
192.168.1.122:2381: 1

Complete persistence-values.yaml

kubeEtcd:
  enabled: true
  endpoints:
    - 192.168.1.120
    - 192.168.1.121
    - 192.168.1.122
  service:
    enabled: true
    port: 2381
    targetPort: 2381

prometheus:
  prometheusSpec:
    additionalScrapeConfigsSecret:
      enabled: true
      name: additional-scrape-configs
      key: additional-scrape-configs.yaml
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: ""
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
          selector:
            matchLabels:
              type: local
              app: prometheus

grafana:
  persistence:
    enabled: true
    type: pvc
    existingClaim: "grafana-data-pvc"

  initChownData:
    enabled: false

  podSecurityContext:
    fsGroup: 911
    runAsUser: 911
    runAsGroup: 911

ZFS Monitoring for FreeBSD Servers

Node Exporter ZFS Collector

• ARC (Adaptive Replacement Cache) statistics
• Cache hit/miss rates
• Memory usage and allocation
• MRU/MFU cache breakdown
• Data vs metadata distribution

Verifying ZFS Metrics

paul@f0:~ % curl -s http://localhost:9100/metrics | grep node_zfs_arcstats | wc -l
      69

ZFS Recording Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: freebsd-zfs-rules
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
    - name: freebsd-zfs-arc
      interval: 30s
      rules:
        - record: node_zfs_arc_hit_rate_percent
          expr: |
            100 * (
              rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) /
              (rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) +
               rate(node_zfs_arcstats_misses_total{os="freebsd"}[5m]))
            )
          labels:
            os: freebsd
        - record: node_zfs_arc_memory_usage_percent
          expr: |
            100 * (
              node_zfs_arcstats_size_bytes{os="freebsd"} /
              node_zfs_arcstats_c_max_bytes{os="freebsd"}
            )
          labels:
            os: freebsd
        # Additional rules for metadata %, target %, MRU/MFU %, etc.

• ARC hit rate percentage
• ARC memory usage percentage (current vs maximum)
• ARC target percentage (target vs maximum)
• Metadata vs data percentages
• MRU vs MFU cache percentages
• Demand data and metadata hit rates

Grafana Dashboards

• FreeBSD server (f0, f1, or f2)
• ZFS pool (zdata, zroot, or all)

• Pool Capacity gauge (with thresholds: green <70%, yellow <85%, red >85%)
• Pool Health status (ONLINE/DEGRADED/FAULTED with color coding)
• Total Pool Size stat
• Free Space stat
• Pool Space Usage Over Time (stacked: used + free)
• Pool Capacity Trend time series

• Table showing all datasets with columns: Pool, Dataset, Used, Available, Referenced
• Automatically filters by selected pool

• ARC Hit Rate gauge (red <70%, yellow <90%, green >=90%)
• ARC Size time series (current, target, max)
• ARC Memory Usage percentage gauge
• ARC Hits vs Misses rate
• ARC Data vs Metadata stacked time series

• Total Storage Capacity across all servers
• Total Used space
• Total Free space
• Average Pool Capacity gauge
• Pool Health Status (worst case across cluster)
• Total Pool Space Usage Over Time
• Per-Pool Capacity time series (all pools on all hosts)

• Bar gauge showing capacity by host and pool
• Table with all pools: Host, Pool, Size, Used, Free, Capacity %, Health

• Average ARC Hit Rate gauge across all hosts
• ARC Hit Rate by Host time series
• Total ARC Size Across Cluster
• Total ARC Hits vs Misses (cluster-wide sum)
• ARC Size by Host

Deployment

cd /home/paul/git/conf/f3s/prometheus
kubectl apply -f zfs-recording-rules.yaml
kubectl apply -f zfs-dashboards.yaml

install:
    kubectl apply -f persistent-volumes.yaml
    kubectl create secret generic additional-scrape-configs --from-file=additional-scrape-configs.yaml -n monitoring --dry-run=client -o yaml | kubectl apply -f -
    helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring -f persistence-values.yaml
    kubectl apply -f freebsd-recording-rules.yaml
    kubectl apply -f openbsd-recording-rules.yaml
    kubectl apply -f zfs-recording-rules.yaml
    just -f grafana-ingress/Justfile install

Verifying ZFS Metrics in Prometheus

kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=node_zfs_arcstats_size_bytes'

kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=node_zfs_arc_memory_usage_percent'

"result":[
  {"metric":{"instance":"192.168.2.130:9100","os":"freebsd"},"value":[...,"37.58"]},
  {"metric":{"instance":"192.168.2.131:9100","os":"freebsd"},"value":[...,"12.85"]},
  {"metric":{"instance":"192.168.2.132:9100","os":"freebsd"},"value":[...,"13.44"]}
]

Accessing the Dashboards

• "FreeBSD ZFS" - detailed per-host view with pool and dataset breakdowns
• "FreeBSD ZFS Summary" - cluster-wide overview of all ZFS storage

Key Metrics to Monitor

ZFS Pool and Dataset Metrics via Textfile Collector

#!/bin/sh
# ZFS Pool and Dataset Metrics Collector for Prometheus

OUTPUT_FILE="/var/tmp/node_exporter/zfs_pools.prom.$$"
FINAL_FILE="/var/tmp/node_exporter/zfs_pools.prom"

mkdir -p /var/tmp/node_exporter

{
    # Pool metrics
    echo "# HELP zfs_pool_size_bytes Total size of ZFS pool"
    echo "# TYPE zfs_pool_size_bytes gauge"
    echo "# HELP zfs_pool_allocated_bytes Allocated space in ZFS pool"
    echo "# TYPE zfs_pool_allocated_bytes gauge"
    echo "# HELP zfs_pool_free_bytes Free space in ZFS pool"
    echo "# TYPE zfs_pool_free_bytes gauge"
    echo "# HELP zfs_pool_capacity_percent Capacity percentage"
    echo "# TYPE zfs_pool_capacity_percent gauge"
    echo "# HELP zfs_pool_health Pool health (0=ONLINE, 1=DEGRADED, 2=FAULTED)"
    echo "# TYPE zfs_pool_health gauge"

    zpool list -Hp -o name,size,allocated,free,capacity,health | \
    while IFS=$'\t' read name size alloc free cap health; do
        case "$health" in
            ONLINE)   health_val=0 ;;
            DEGRADED) health_val=1 ;;
            FAULTED)  health_val=2 ;;
            *)        health_val=6 ;;
        esac
        cap_num=$(echo "$cap" | sed 's/%//')

        echo "zfs_pool_size_bytes{pool=\"$name\"} $size"
        echo "zfs_pool_allocated_bytes{pool=\"$name\"} $alloc"
        echo "zfs_pool_free_bytes{pool=\"$name\"} $free"
        echo "zfs_pool_capacity_percent{pool=\"$name\"} $cap_num"
        echo "zfs_pool_health{pool=\"$name\"} $health_val"
    done

    # Dataset metrics
    echo "# HELP zfs_dataset_used_bytes Used space in dataset"
    echo "# TYPE zfs_dataset_used_bytes gauge"
    echo "# HELP zfs_dataset_available_bytes Available space"
    echo "# TYPE zfs_dataset_available_bytes gauge"
    echo "# HELP zfs_dataset_referenced_bytes Referenced space"
    echo "# TYPE zfs_dataset_referenced_bytes gauge"

    zfs list -Hp -t filesystem -o name,used,available,referenced | \
    while IFS=$'\t' read name used avail ref; do
        pool=$(echo "$name" | cut -d/ -f1)
        echo "zfs_dataset_used_bytes{pool=\"$pool\",dataset=\"$name\"} $used"
        echo "zfs_dataset_available_bytes{pool=\"$pool\",dataset=\"$name\"} $avail"
        echo "zfs_dataset_referenced_bytes{pool=\"$pool\",dataset=\"$name\"} $ref"
    done
} > "$OUTPUT_FILE"

mv "$OUTPUT_FILE" "$FINAL_FILE"

for host in f0 f1 f2; do
    scp /tmp/zfs_pool_metrics.sh paul@$host:/tmp/
    ssh paul@$host 'doas mv /tmp/zfs_pool_metrics.sh /usr/local/bin/ && \
                    doas chmod +x /usr/local/bin/zfs_pool_metrics.sh'
done

for host in f0 f1 f2; do
    ssh paul@$host 'echo "* * * * * /usr/local/bin/zfs_pool_metrics.sh >/dev/null 2>&1" | \
                    doas crontab -'
done

paul@f0:~ % curl -s http://localhost:9100/metrics | grep "^zfs_pool" | head -5
zfs_pool_allocated_bytes{pool="zdata"} 6.47622733824e+11
zfs_pool_allocated_bytes{pool="zroot"} 5.3338578944e+10
zfs_pool_capacity_percent{pool="zdata"} 64
zfs_pool_capacity_percent{pool="zroot"} 10
zfs_pool_free_bytes{pool="zdata"} 3.48809678848e+11

Summary

• Enabling etcd metrics monitoring for the k3s embedded etcd
• Implementing comprehensive ZFS monitoring for FreeBSD storage servers
• Creating recording rules for calculated metrics (ARC hit rates, memory usage, etc.)
• Deploying Grafana dashboards for visualization
• Configuring automatic dashboard import via ConfigMap labels

Distributed Tracing with Grafana Tempo

Why Distributed Tracing?

• Tracks requests across service boundaries
• Identifies performance bottlenecks
• Visualizes service dependencies
• Correlates with logs and metrics
• Helps debug complex distributed systems

Deploying Grafana Tempo

• Simpler operation than microservices mode
• Suitable for the cluster scale
• Consistent with Loki deployment pattern

• 10Gi storage at /data/nfs/k3svolumes/tempo/data
• 7-day retention (168h)
• Local storage is the only option for monolithic mode

• gRPC: 4317
• HTTP: 4318
• Bind to 0.0.0.0 to avoid Tempo 2.7+ localhost-only binding issue

tempo:
  retention: 168h
  storage:
    trace:
      backend: local
      local:
        path: /var/tempo/traces
      wal:
        path: /var/tempo/wal
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

persistence:
  enabled: true
  size: 10Gi
  storageClassName: ""

resources:
  limits:
    cpu: 1000m
    memory: 2Gi
  requests:
    cpu: 500m
    memory: 1Gi

apiVersion: v1
kind: PersistentVolume
metadata:
  name: tempo-data-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/nfs/k3svolumes/tempo/data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tempo-data-pvc
  namespace: monitoring
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources-all
  namespace: monitoring
data:
  datasources.yaml: |
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        uid: prometheus
        url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/
        access: proxy
        isDefault: true
      - name: Alertmanager
        type: alertmanager
        uid: alertmanager
        url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/
      - name: Loki
        type: loki
        uid: loki
        url: http://loki.monitoring.svc.cluster.local:3100
      - name: Tempo
        type: tempo
        uid: tempo
        url: http://tempo.monitoring.svc.cluster.local:3200
        jsonData:
          tracesToLogsV2:
            datasourceUid: loki
            spanStartTimeShift: -1h
            spanEndTimeShift: 1h
          tracesToMetrics:
            datasourceUid: prometheus
          serviceMap:
            datasourceUid: prometheus
          nodeGraph:
            enabled: true

• Disable sidecar-based datasource provisioning
• Mount grafana-datasources-all ConfigMap directly to /etc/grafana/provisioning/datasources/

cd /home/paul/git/conf/f3s/tempo
just install

kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready

Configuring Grafana Alloy for Trace Collection

// OTLP receiver for traces via gRPC and HTTP
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }
  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

// Batch processor for efficient trace forwarding
otelcol.processor.batch "default" {
  timeout = "5s"
  send_batch_size = 100
  send_batch_max_size = 200
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

// OTLP exporter to send traces to Tempo
otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.monitoring.svc.cluster.local:4317"
    tls {
      insecure = true
    }
    compression = "gzip"
  }
}

cd /home/paul/git/conf/f3s/loki
just upgrade

kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "otlp.*receiver"
kubectl exec -n monitoring <alloy-pod> -- netstat -ln | grep -E ':(4317|4318)'

Demo Tracing Application

User → Frontend (Flask:5000) → Middleware (Flask:5001) → Backend (Flask:5002)
           ↓                          ↓                        ↓
                    Alloy (OTLP:4317) → Tempo → Grafana

• Receives HTTP requests at /api/process
• Forwards to middleware service
• Creates parent span for the entire request

• Transforms data at /api/transform
• Calls backend service
• Creates child span linked to frontend

• Returns data at /api/data
• Simulates database query (100ms sleep)
• Creates leaf span in the trace

flask==3.0.0
requests==2.31.0
opentelemetry-distro==0.49b0
opentelemetry-exporter-otlp==1.28.0
opentelemetry-instrumentation-flask==0.49b0
opentelemetry-instrumentation-requests==0.49b0

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.resources import Resource

# Define service identity
resource = Resource(attributes={
    "service.name": "frontend",
    "service.namespace": "tracing-demo",
    "service.version": "1.0.0"
})

provider = TracerProvider(resource=resource)

# Export to Alloy
otlp_exporter = OTLPSpanExporter(
    endpoint="http://alloy.monitoring.svc.cluster.local:4317",
    insecure=True
)

processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Auto-instrument Flask and requests
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

• Creates spans for HTTP requests
• Propagates trace context via W3C Trace Context headers
• Links parent and child spans across service boundaries

cd /home/paul/git/conf/f3s/tracing-demo
just build
just import
just install

kubectl get pods -n services | grep tracing-demo
kubectl get ingress -n services tracing-demo-ingress

Visualizing Traces in Grafana

• Search by Trace ID
• Search by service name
• Search by tags

{ resource.service.namespace = "tracing-demo" }

{ duration > 200ms }

{ resource.service.name = "frontend" }

{ status = error }

{ resource.service.namespace = "tracing-demo" } && { span.http.status_code >= 500 }

Correlation Between Observability Signals

Generating Traces for Testing

curl http://tracing-demo.f3s.buetow.org/api/process

cd /home/paul/git/conf/f3s/tracing-demo
just load-test

Verifying the Complete Pipeline

kubectl logs -n services -l app=tracing-demo-frontend | grep -i trace

kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i otlp

kubectl logs -n monitoring -l app.kubernetes.io/name=tempo | grep -i trace

Practical Example: Viewing a Distributed Trace

curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process

{
  "middleware_response": {
    "backend_data": {
      "data": {
        "id": 12345,
        "query_time_ms": 100.0,
        "timestamp": "2025-12-28T18:35:01.064538",
        "value": "Sample data from backend service"
      },
      "service": "backend"
    },
    "middleware_processed": true,
    "original_data": {
      "source": "GET request"
    },
    "transformation_time_ms": 50
  },
  "request_data": {
    "source": "GET request"
  },
  "service": "frontend",
  "status": "success"
}

kubectl exec -n monitoring tempo-0 -- wget -qO- \
  'http://localhost:3200/api/search?tags=service.namespace%3Dtracing-demo&limit=5' 2>/dev/null | \
  python3 -m json.tool

{
  "traceID": "4be1151c0bdcd5625ac7e02b98d95bd5",
  "rootServiceName": "frontend",
  "rootTraceName": "GET /api/process",
  "durationMs": 221
}

kubectl exec -n monitoring tempo-0 -- wget -qO- \
  'http://localhost:3200/api/traces/4be1151c0bdcd5625ac7e02b98d95bd5' 2>/dev/null | \
  python3 -m json.tool

Trace ID: 4be1151c0bdcd5625ac7e02b98d95bd5
Services: 3 (frontend, middleware, backend)

Service: frontend
  └─ GET /api/process                 221.10ms  (HTTP server span)
  └─ frontend-process                 216.23ms  (custom business logic span)
  └─ POST                             209.97ms  (HTTP client span to middleware)

Service: middleware
  └─ POST /api/transform              186.02ms  (HTTP server span)
  └─ middleware-transform             180.96ms  (custom business logic span)
  └─ GET                              127.52ms  (HTTP client span to backend)

Service: backend
  └─ GET /api/data                    103.93ms  (HTTP server span)
  └─ backend-get-data                 102.11ms  (custom business logic span with 100ms sleep)

{ resource.service.namespace = "tracing-demo" }

4be1151c0bdcd5625ac7e02b98d95bd5

• **Frontend (221ms)**: Receives GET /api/process, executes business logic, calls middleware
• **Middleware (186ms)**: Receives POST /api/transform, transforms data, calls backend
• **Backend (104ms)**: Receives GET /api/data, simulates database query with 100ms sleep
• **Total request time**: 221ms end-to-end
• **Span propagation**: W3C Trace Context headers automatically link all spans

• Request rates between services
• Average latency for each hop
• Error rates (if any)
• Service dependencies and communication patterns

Storage and Retention

kubectl exec -n monitoring <tempo-pod> -- df -h /var/tempo

• Reduce retention to 72h (3 days)
• Implement sampling in Alloy
• Increase PV size

Complete Observability Stack

• Cluster resource usage
• Application metrics
• Node metrics (FreeBSD ZFS, OpenBSD edge)
• etcd health

• All pod logs
• Structured log collection
• Log aggregation and search

• Distributed request tracing
• Service dependency mapping
• Performance profiling
• Error tracking

• Unified dashboards
• Correlation between metrics, logs, and traces
• Service graphs
• Alerts

Configuration Files

Generated with Gemtexter 3.0.1-develop | served by OpenBSD/relayd(8)+httpd(8) | Site Mirrors
Webring: previous | shring | next