# Setup: Prometheus

To use historic mode, backfill mode, auto mode with old data, or watch mode with `-prometheus`, you must enable the Prometheus Remote Write receiver. Without it, Epimetheus can only push realtime data via Pushgateway.

## 1. Enable Remote Write Receiver and Admin API

Example configuration (Prometheus 3.x style). Adjust paths and stack to match your environment (e.g. [conf repository](https://codeberg.org/snonux/conf) at `f3s/prometheus/persistence-values.yaml`):

```yaml
prometheus:
  prometheusSpec:
    additionalArgs:
      - name: web.enable-remote-write-receiver
        value: ""
      - name: web.enable-admin-api
        value: ""

    enableFeatures:
      - exemplar-storage
      - otlp-write-receiver

    tsdb:
      outOfOrderTimeWindow: 744h  # 31 days for backfilling
```

This provides:

- **Remote Write API** at `/api/v1/write` for ingesting metrics with custom timestamps.
- **Admin API** at `/api/v1/admin/tsdb/*` for deleting series and cleaning tombstones.
- **Out-of-order ingestion** so older points can be written for existing series (within the time window).

After changing config, upgrade Prometheus (e.g. `helm upgrade` or your usual apply).

### Verify

```bash
# Remote Write receiver
kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
  -o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-remote-write-receiver"

# Out-of-order window
kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
  -o jsonpath='{.spec.tsdb.outOfOrderTimeWindow}'

# Admin API
kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
  -o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-admin-api"
```

**Note:** In Prometheus 3.x use `additionalArgs` for `web.enable-remote-write-receiver`; the older `enableFeatures: [remote-write-receiver]` is deprecated.

## 2. Scrape Config for Pushgateway

For realtime mode, Prometheus must scrape Pushgateway. Example:

```yaml
# additional-scrape-configs.yaml
- job_name: 'pushgateway'
  honor_labels: true
  static_configs:
    - targets:
      - 'pushgateway.monitoring.svc.cluster.local:9091'
```

Apply as a Secret (example):

```bash
kubectl create secret generic additional-scrape-configs \
  --from-file=additional-scrape-configs.yaml \
  --dry-run=client -o yaml -n monitoring | kubectl apply -f -
```

## 3. Retention

Check retention so you know how far back Epimetheus can write:

```bash
kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
  -o jsonpath='{.spec.retention}'
```

For very old data, increase retention or use a dedicated dev/test Prometheus. Enabling out-of-order ingestion and a large `outOfOrderTimeWindow` has memory and I/O trade-offs; see [Prometheus backend](../backends/prometheus.md) and keep production config conservative.