blob: 4968211f34ad9f31a99375708008a5d91605bfe9 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
|
# f3s: Kubernetes with FreeBSD - Part 9: Enabling etcd Metrics
## Introduction
This post covers enabling etcd metrics monitoring for the k3s cluster. The etcd dashboard in Grafana initially showed no data because k3s uses an embedded etcd that doesn't expose metrics by default.
=> ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html Part 8: Observability
## Enabling etcd metrics in k3s
On each control-plane node (r0, r1, r2), create /etc/rancher/k3s/config.yaml:
```
etcd-expose-metrics: true
```
Then restart k3s on each node:
```
systemctl restart k3s
```
After restarting, etcd metrics are available on port 2381:
```
curl http://127.0.0.1:2381/metrics | grep etcd
```
## Configuring Prometheus to scrape etcd
In persistence-values.yaml, enable kubeEtcd with the node IP addresses:
```
kubeEtcd:
enabled: true
endpoints:
- 192.168.1.120
- 192.168.1.121
- 192.168.1.122
service:
enabled: true
port: 2381
targetPort: 2381
```
Apply the changes:
```
just upgrade
```
## Verifying etcd metrics
After the changes, all etcd targets are being scraped:
```
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
-c prometheus -- wget -qO- 'http://localhost:9090/api/v1/query?query=etcd_server_has_leader' | \
jq -r '.data.result[] | "\(.metric.instance): \(.value[1])"'
```
Output:
```
192.168.1.120:2381: 1
192.168.1.121:2381: 1
192.168.1.122:2381: 1
```
The etcd dashboard in Grafana now displays metrics including Raft proposals, leader elections, and peer round trip times.
## Complete persistence-values.yaml
The complete updated persistence-values.yaml:
```
kubeEtcd:
enabled: true
endpoints:
- 192.168.1.120
- 192.168.1.121
- 192.168.1.122
service:
enabled: true
port: 2381
targetPort: 2381
prometheus:
prometheusSpec:
additionalScrapeConfigsSecret:
enabled: true
name: additional-scrape-configs
key: additional-scrape-configs.yaml
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: ""
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
selector:
matchLabels:
type: local
app: prometheus
grafana:
persistence:
enabled: true
type: pvc
existingClaim: "grafana-data-pvc"
initChownData:
enabled: false
podSecurityContext:
fsGroup: 911
runAsUser: 911
runAsGroup: 911
```
## Summary
Enabled etcd metrics monitoring for the k3s embedded etcd by:
* Adding etcd-expose-metrics: true to /etc/rancher/k3s/config.yaml on each control-plane node
* Configuring Prometheus to scrape etcd on port 2381
The etcd dashboard now provides visibility into cluster health, leader elections, and Raft consensus metrics.
=> https://codeberg.org/snonux/conf/src/branch/master/f3s/prometheus prometheus configuration on Codeberg
|
