blob: 294ce20c8d89ab53004ac9089c81f6ba768c1fb0 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
|
# Setup: Prometheus
To use historic mode, backfill mode, auto mode with old data, or watch mode with `-prometheus`, you must enable the Prometheus Remote Write receiver. Without it, Epimetheus can only push realtime data via Pushgateway.
## 1. Enable Remote Write Receiver and Admin API
Example configuration (Prometheus 3.x style). Adjust paths and stack to match your environment (e.g. [conf repository](https://codeberg.org/snonux/conf) at `f3s/prometheus/persistence-values.yaml`):
```yaml
prometheus:
prometheusSpec:
additionalArgs:
- name: web.enable-remote-write-receiver
value: ""
- name: web.enable-admin-api
value: ""
enableFeatures:
- exemplar-storage
- otlp-write-receiver
tsdb:
outOfOrderTimeWindow: 744h # 31 days for backfilling
```
This provides:
- **Remote Write API** at `/api/v1/write` for ingesting metrics with custom timestamps.
- **Admin API** at `/api/v1/admin/tsdb/*` for deleting series and cleaning tombstones.
- **Out-of-order ingestion** so older points can be written for existing series (within the time window).
After changing config, upgrade Prometheus (e.g. `helm upgrade` or your usual apply).
### Verify
```bash
# Remote Write receiver
kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
-o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-remote-write-receiver"
# Out-of-order window
kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
-o jsonpath='{.spec.tsdb.outOfOrderTimeWindow}'
# Admin API
kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
-o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-admin-api"
```
**Note:** In Prometheus 3.x use `additionalArgs` for `web.enable-remote-write-receiver`; the older `enableFeatures: [remote-write-receiver]` is deprecated.
## 2. Scrape Config for Pushgateway
For realtime mode, Prometheus must scrape Pushgateway. Example:
```yaml
# additional-scrape-configs.yaml
- job_name: 'pushgateway'
honor_labels: true
static_configs:
- targets:
- 'pushgateway.monitoring.svc.cluster.local:9091'
```
Apply as a Secret (example):
```bash
kubectl create secret generic additional-scrape-configs \
--from-file=additional-scrape-configs.yaml \
--dry-run=client -o yaml -n monitoring | kubectl apply -f -
```
## 3. Retention
Check retention so you know how far back Epimetheus can write:
```bash
kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
-o jsonpath='{.spec.retention}'
```
For very old data, increase retention or use a dedicated dev/test Prometheus. Enabling out-of-order ingestion and a large `outOfOrderTimeWindow` has memory and I/O trade-offs; see [Prometheus backend](../backends/prometheus.md) and keep production config conservative.
|
