diff options
Diffstat (limited to 'docs/backends')
| -rw-r--r-- | docs/backends/clickhouse.md | 92 | ||||
| -rw-r--r-- | docs/backends/prometheus.md | 76 |
2 files changed, 168 insertions, 0 deletions
diff --git a/docs/backends/clickhouse.md b/docs/backends/clickhouse.md new file mode 100644 index 0000000..ad1b5f0 --- /dev/null +++ b/docs/backends/clickhouse.md @@ -0,0 +1,92 @@ +# ClickHouse + +Epimetheus can ingest metrics into ClickHouse in **watch mode** only. ClickHouse is optional: you can use it in addition to Prometheus or as the only backend (by setting `-prometheus=` to disable Prometheus ingestion). + +## Data flow (watch mode only) + +``` +┌─────────────────┐ poll (1s) ┌─────────────────────────────────────┐ +│ CSV file(s) │ ─────────────────▶ │ Epimetheus (watch mode) │ +│ (mtime = │ │ • Parse tabular CSV │ +│ timestamp) │ │ • -metric-name + columns → metrics │ +└─────────────────┘ └─────────────────────────────────────┘ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ + ▼ ▼ │ + ┌───────────────┐ ┌───────────────┐ │ + │ Prometheus │ │ ClickHouse │ │ + │ (optional) │ │ (optional) │ │ + │ -prometheus= │ │ -clickhouse= │ │ + │ Remote Write │ │ HTTP insert │ │ + └───────────────┘ └───────────────┘ │ + │ + At least one of -prometheus or -clickhouse │ +``` + +## When It's Used + +- **Mode:** Watch only. Other modes (realtime, historic, backfill, auto) do not write to ClickHouse. +- **Flags:** + - `-clickhouse` – ClickHouse HTTP URL (e.g. `http://localhost:8123`). If empty, no ClickHouse ingestion. + - `-clickhouse-table` – Table name (default: `epimetheus_metrics`). + +At least one of `-prometheus` or `-clickhouse` must be set for watch mode. + +## Table Schema + +Epimetheus creates the table if it does not exist. Schema: + +```sql +CREATE TABLE IF NOT EXISTS epimetheus_metrics ( + metric String, + labels Map(String, String), + value Float64, + timestamp DateTime64(3) +) ENGINE = MergeTree() +ORDER BY (metric, timestamp) +``` + +- `metric` – metric name (e.g. from `-metric-name` and column headers in tabular CSV). +- `labels` – key-value map of label names and values. +- `value` – sample value. +- `timestamp` – sample time (millisecond precision). + +## Examples + +**Prometheus and ClickHouse:** + +```bash +./epimetheus -mode=watch -file=data.csv -metric-name=myapp \ + -prometheus=http://localhost:9090/api/v1/write \ + -clickhouse=http://localhost:8123 +``` + +**ClickHouse only:** + +```bash +./epimetheus -mode=watch -file=test-data/watch-clickhouse-test.csv \ + -metric-name=watch_test \ + -clickhouse=http://localhost:8123 \ + -prometheus= +``` + +**Custom table:** + +```bash +./epimetheus -mode=watch -file=data.csv -metric-name=myapp \ + -clickhouse=http://localhost:8123 \ + -clickhouse-table=my_metrics +``` + +## Verification + +Use the provided script to check that data landed in ClickHouse: + +```bash +./scripts/verify-clickhouse.sh +# Or with custom URL/table: +./scripts/verify-clickhouse.sh http://localhost:8123 epimetheus_metrics +``` + +The script checks connectivity, row count, distinct metrics, sample rows, and rows per metric. See [Setup: ClickHouse](../operations/setup-clickhouse.md) for getting ClickHouse running. diff --git a/docs/backends/prometheus.md b/docs/backends/prometheus.md new file mode 100644 index 0000000..f8d2a9b --- /dev/null +++ b/docs/backends/prometheus.md @@ -0,0 +1,76 @@ +# Prometheus (and Prometheus-Compatible Backends) + +Epimetheus can ingest metrics into Prometheus via two paths. Any backend that exposes the Prometheus Remote Write API (including **VictoriaMetrics**) is supported by pointing `-prometheus=` at that backend's write URL (e.g. `http://victoriametrics:8428/api/v1/write`). + +## Ingestion paths (overview) + +``` + Epimetheus + │ + ┌───────────────┼───────────────┐ + │ │ │ + ▼ ▼ ▼ + Realtime mode Historic/Backfill Watch mode + (current data) (old data) (CSV file mtime) + │ │ │ + ▼ │ │ + ┌───────────┐ │ │ + │Pushgateway │ │ │ + │ (HTTP POST)│ │ │ + └─────┬─────┘ │ │ + │ Scrape │ │ + │ (15–30s) │ │ + ▼ ▼ ▼ + ┌─────────────────────────────────────────────┐ + │ Prometheus / VictoriaMetrics │ + │ Remote Write API: /api/v1/write │ + │ (realtime: via Pushgateway scrape; │ + │ historic/watch: direct POST) │ + └─────────────────────────────────────────────┘ +``` + +## Ingestion Paths + +### Realtime: Pushgateway + +- **Used by:** realtime mode, and auto mode for samples < 5 minutes old. +- **Flow:** Epimetheus pushes to Pushgateway (HTTP POST); Prometheus scrapes Pushgateway on its schedule. Timestamps become "now" at scrape time. +- **Flags:** `-pushgateway` (default `http://localhost:9091`), `-job`, `-continuous`. + +### Historic: Remote Write API + +- **Used by:** historic mode, backfill mode, auto mode for samples ≥ 5 minutes old, and watch mode (when `-prometheus` is set). +- **Flow:** Epimetheus sends samples to the Remote Write endpoint (e.g. `/api/v1/write`). Timestamps from the data are preserved. +- **Flags:** `-prometheus` (default `http://localhost:9090/api/v1/write`). + +The Remote Write receiver must be enabled on Prometheus for historic/watch/backfill/auto with old data. See [Setup: Prometheus](../operations/setup-prometheus.md). + +## Prometheus-Compatible Backends (e.g. VictoriaMetrics) + +Backends that implement the [Prometheus Remote Write](https://prometheus.io/docs/concepts/remote_write_spec/) API work with Epimetheus without any code changes. Use their write endpoint as the `-prometheus=` URL. + +**Example (VictoriaMetrics):** + +```bash +./epimetheus -mode=watch -file=data.csv -metric-name=myapp \ + -prometheus=http://victoriametrics:8428/api/v1/write +``` + +Replace host/port with your VictoriaMetrics (or other compatible) write URL. Realtime mode still uses Pushgateway (scraped by your Prometheus or VictoriaMetrics); for watch/historic/backfill/auto, only the `-prometheus=` target changes. + +## Time Ranges + +| Time range | Status | Method | +|------------|--------|--------| +| Current (< 5 min) | Supported | Pushgateway | +| 1 hour old | Supported | Remote Write | +| 1 day to 1 month old | Supported | Remote Write | +| 6+ months | May be rejected (retention) | Remote Write | +| Years old | Likely rejected; use `promtool tsdb create-blocks-from` | — | +| Future (> 5 min ahead) | Rejected | — | + +Out-of-order samples (older than existing data for the same series) require out-of-order ingestion to be enabled on the backend, or use different labels. See [Troubleshooting](../operations/troubleshooting.md). + +## Retention and Configuration + +Check your backend's retention (e.g. Prometheus `retention`, VictoriaMetrics settings). For very old data you may need to increase retention or enable out-of-order ingestion. See [Setup: Prometheus](../operations/setup-prometheus.md) for Prometheus-specific options. |