summaryrefslogtreecommitdiff
path: root/docs/guides
diff options
context:
space:
mode:
Diffstat (limited to 'docs/guides')
-rw-r--r--docs/guides/csv-format-flexibility.md52
-rw-r--r--docs/guides/data-formats.md49
-rw-r--r--docs/guides/dns-resolution.md42
-rw-r--r--docs/guides/dtail-metrics-example.md49
-rw-r--r--docs/guides/modes.md130
-rw-r--r--docs/guides/quickstart.md56
6 files changed, 378 insertions, 0 deletions
diff --git a/docs/guides/csv-format-flexibility.md b/docs/guides/csv-format-flexibility.md
new file mode 100644
index 0000000..180dc28
--- /dev/null
+++ b/docs/guides/csv-format-flexibility.md
@@ -0,0 +1,52 @@
+# CSV Format Flexibility
+
+Watch mode works with **any tabular CSV**. You do not need a fixed schema; Epimetheus infers metric names and labels from column headers and value types.
+
+## How It Works
+
+- **First row:** Column headers (automatically sanitized for Prometheus label/metric names).
+- **Numeric columns:** Treated as metric values. Each gets a metric name derived from the base metric name and the column header.
+- **String columns:** Treated as labels. Each row’s value becomes the label value for that series.
+- **Metric name:** Set with `-metric-name` (e.g. `web`, `food`, `network`). It is used as a prefix for all numeric columns.
+
+Column names can contain characters that are invalid in Prometheus (e.g. parentheses, spaces). They are sanitized: for example `min(potatoes)` becomes a valid metric suffix like `min_potatoes`.
+
+## Examples
+
+### Web metrics
+
+```csv
+avg(response_time),p99(latency),endpoint,method
+45.2,120.5,/api/users,GET
+52.1,135.8,/api/orders,POST
+```
+
+With `-metric-name=web` this produces series such as:
+
+- `web_avg_response_time{endpoint="/api/users",method="GET"} 45.2`
+- `web_p99_latency{endpoint="/api/users",method="GET"} 120.5`
+- `web_avg_response_time{endpoint="/api/orders",method="POST"} 52.1`
+- `web_p99_latency{endpoint="/api/orders",method="POST"} 135.8`
+
+### Food / business metrics
+
+```csv
+min(potatoes),last(coke),avg(price),country,store_type
+5.2,10.5,12.99,USA,grocery
+3.8,8.2,9.99,Canada,convenience
+```
+
+With `-metric-name=food` this produces series such as:
+
+- `food_min_potatoes{country="USA",store_type="grocery"} 5.2`
+- `food_last_coke{country="USA",store_type="grocery"} 10.5`
+- `food_avg_price{country="USA",store_type="grocery"} 12.99`
+- and the same metrics with `country="Canada",store_type="convenience"`.
+
+### Summary
+
+- Each **row** becomes one or more samples (one per numeric column).
+- **Numeric columns** → different metrics (same labels for that row).
+- **String columns** → labels shared by all those metrics for that row.
+
+For the standard Epimetheus CSV format (explicit metric name, labels, value, timestamp) see [Data Formats](data-formats.md). For modes and watch options see [Operating Modes](modes.md).
diff --git a/docs/guides/data-formats.md b/docs/guides/data-formats.md
new file mode 100644
index 0000000..24d7755
--- /dev/null
+++ b/docs/guides/data-formats.md
@@ -0,0 +1,49 @@
+# Data Formats
+
+Epimetheus accepts CSV and JSON input for **auto mode** (and for **watch mode**, watch uses tabular CSV; see [CSV Format Flexibility](csv-format-flexibility.md)).
+
+## Epimetheus CSV (auto mode)
+
+Format: one metric per line with explicit metric name, labels, value, and optional timestamp.
+
+```csv
+# Format: metric_name,labels,value,timestamp_ms
+# Labels: key1=value1;key2=value2
+epimetheus_test_requests_total,instance=web1;env=prod,100,1767125148000
+epimetheus_test_temperature_celsius,instance=web2,22.5,1767038748000
+
+# Timestamp optional (uses "now" if omitted)
+epimetheus_test_active_connections,instance=web3,42,
+```
+
+- **metric_name** – Prometheus metric name.
+- **labels** – Semicolon-separated `key=value` pairs.
+- **value** – Numeric value.
+- **timestamp_ms** – Unix milliseconds. Omit or leave empty for "now".
+
+## JSON (auto mode)
+
+Array of objects with `metric`, `labels`, `value`, and optional `timestamp_ms`:
+
+```json
+[
+ {
+ "metric": "epimetheus_test_requests_total",
+ "labels": {"instance": "web1", "env": "prod"},
+ "value": 100,
+ "timestamp_ms": 1767125148000
+ },
+ {
+ "metric": "epimetheus_test_temperature_celsius",
+ "labels": {"instance": "web2"},
+ "value": 22.5,
+ "timestamp_ms": 1767038748000
+ }
+]
+```
+
+Omit `timestamp_ms` for "now".
+
+## Watch mode CSV
+
+Watch mode uses **tabular CSV**: first row = headers, following rows = data. Numeric columns become metrics (with `-metric-name` as prefix), string columns become labels. See [CSV Format Flexibility](csv-format-flexibility.md).
diff --git a/docs/guides/dns-resolution.md b/docs/guides/dns-resolution.md
new file mode 100644
index 0000000..42478a2
--- /dev/null
+++ b/docs/guides/dns-resolution.md
@@ -0,0 +1,42 @@
+# DNS Resolution (Watch Mode)
+
+In watch mode, Epimetheus can resolve IP addresses in label values to hostnames. This improves readability in Grafana and other tools that display label values.
+
+## Default Behaviour
+
+- The label **`ip`** is always resolved by default (when present).
+- Resolution is done via reverse DNS. The result is used as the label value (e.g. `10.50.52.61` → `foo.example.lan`).
+- Failed lookups leave the original IP unchanged.
+- Results are cached in memory to avoid repeated DNS lookups.
+
+## Additional Labels
+
+To resolve other IP-carrying labels, use `-resolve-ip-labels` with a comma-separated list of label names:
+
+```bash
+./epimetheus -mode=watch \
+ -file=network.csv \
+ -metric-name=network \
+ -resolve-ip-labels=source_ip,dest_ip
+```
+
+This resolves:
+
+- `ip` (always, if present)
+- `source_ip`
+- `dest_ip`
+
+Duplicate or empty entries (e.g. listing `ip` again) are ignored.
+
+## Example
+
+- **Input label:** `ip="10.50.52.61"`
+- **After resolution:** `ip="foo.example.lan"` (if reverse DNS returns that name)
+- **If resolution fails:** `ip="10.50.52.61"` (unchanged)
+
+## When to Use
+
+- CSV columns that contain IPs and are used as labels (e.g. `ip`, `host`, `source_ip`, `dest_ip`).
+- When you want dashboards to show hostnames instead of raw IPs.
+
+DNS resolution only applies in **watch mode**. Other modes do not use this feature. See [Operating Modes](modes.md) and [CLI Reference](../reference/cli.md) for full options.
diff --git a/docs/guides/dtail-metrics-example.md b/docs/guides/dtail-metrics-example.md
new file mode 100644
index 0000000..5416726
--- /dev/null
+++ b/docs/guides/dtail-metrics-example.md
@@ -0,0 +1,49 @@
+# Dtail Metrics Example
+
+This page walks through using Epimetheus watch mode with a CSV that could come from a tool like [Dtail](https://dtail.dev/) or any similar log/aggregation export.
+
+## Scenario
+
+You have a CSV file (e.g. `dtail.csv`) with columns that mix numeric stats and identifiers (host, service, etc.). You want to turn those into Prometheus metrics so you can graph them in Grafana.
+
+## Steps
+
+1. **Ensure the CSV has a header row**
+ First line = column names. Epimetheus will sanitize them for use as metric names and labels.
+
+2. **Identify numeric vs string columns**
+ - Numeric columns (e.g. `count`, `avg_latency_ms`, `p99`) become metric values.
+ - String columns (e.g. `host`, `service`, `region`) become labels.
+
+3. **Run watch mode** with a base metric name and your Prometheus (or Prometheus-compatible) write URL:
+
+ ```bash
+ ./epimetheus -mode=watch \
+ -file=dtail.csv \
+ -metric-name=dtail \
+ -prometheus=http://localhost:9090/api/v1/write
+ ```
+
+4. **Optional: resolve IPs to hostnames**
+ If one of your label columns contains IPs (e.g. `host` or `ip`), you can resolve them:
+
+ ```bash
+ ./epimetheus -mode=watch \
+ -file=dtail.csv \
+ -metric-name=dtail \
+ -prometheus=http://localhost:9090/api/v1/write \
+ -resolve-ip-labels=host
+ ```
+
+5. **Query in Prometheus / Grafana**
+ Metrics will appear as `dtail_<column_name>` with your string columns as labels, e.g.:
+
+ ```promql
+ dtail_avg_latency_ms{service="api", region="eu"}
+ ```
+
+## References
+
+- [CSV Format Flexibility](csv-format-flexibility.md) – how column types and names are interpreted.
+- [DNS Resolution](dns-resolution.md) – IP-to-hostname resolution.
+- [Operating Modes](modes.md) – all watch mode options.
diff --git a/docs/guides/modes.md b/docs/guides/modes.md
new file mode 100644
index 0000000..bcbbb6b
--- /dev/null
+++ b/docs/guides/modes.md
@@ -0,0 +1,130 @@
+# Operating Modes
+
+Epimetheus has five modes. Backend support:
+
+| Mode | Prometheus (Pushgateway) | Prometheus (Remote Write) | ClickHouse |
+|-----------|--------------------------|---------------------------|------------|
+| Realtime | Yes | No | No |
+| Historic | No | Yes | No |
+| Backfill | No | Yes | No |
+| Auto | Yes (samples &lt; 5 min) | Yes (samples ≥ 5 min) | No |
+| Watch | Optional | Optional | Optional |
+
+At least one of Prometheus or ClickHouse must be configured for watch mode.
+
+---
+
+## Watch Mode
+
+Monitor CSV files and push metrics using file modification time as the timestamp. Works with any tabular CSV; numeric columns become metrics, string columns become labels.
+
+### Watch mode data flow
+
+```
+┌─────────────────┐ poll (1s) ┌─────────────────────────────────────┐
+│ CSV file(s) │ ─────────────────▶ │ Epimetheus (watch mode) │
+│ │ │ • Parse tabular CSV │
+│ File mtime = │ │ • Numeric columns → metrics │
+│ sample time │ │ • String columns → labels │
+└─────────────────┘ │ • Optional DNS resolution (IPs) │
+ └─────────────────────────────────────┘
+ │
+ ┌────────────────────┼────────────────────┐
+ │ │ │
+ ▼ ▼ │
+ ┌───────────────┐ ┌───────────────┐ │
+ │ Prometheus │ │ ClickHouse │ │
+ │ (optional) │ │ (optional) │ │
+ │ │ │ │ │
+ │ Remote Write │ │ HTTP insert │ │
+ │ /api/v1/write│ │ (batched) │ │
+ └───────────────┘ └───────────────┘ │
+ │ │ │
+ └────────────────────┴────────────────────┘
+ At least one of -prometheus or -clickhouse
+```
+
+```bash
+./epimetheus -mode=watch -file=mydata.csv -metric-name=myapp \
+ -prometheus=http://localhost:9090/api/v1/write
+```
+
+**Options:** `-file`, `-metric-name`, `-prometheus`, `-clickhouse`, `-clickhouse-table`, `-job`, `-resolve-ip-labels`. See [CLI Reference](../reference/cli.md).
+
+**Features:** Format-agnostic CSV, automatic numeric/string detection, label name sanitization, optional DNS resolution for IP labels, timestamp from file mtime, continuous polling (1s), Remote Write (and optionally ClickHouse). See [CSV Format Flexibility](csv-format-flexibility.md) and [DNS Resolution](dns-resolution.md).
+
+---
+
+## Realtime Mode (default)
+
+Push current metrics to Pushgateway with "now" timestamp.
+
+```bash
+./epimetheus -mode=realtime -continuous
+```
+
+**Options:** `-pushgateway` (default `http://localhost:9091`), `-job`, `-continuous`. Pushes every 15 seconds when `-continuous` is set.
+
+---
+
+## Historic Mode
+
+Push a single historic datapoint via Remote Write.
+
+```bash
+./epimetheus -mode=historic -hours-ago=24
+```
+
+**Options:** `-prometheus` (default `http://localhost:9090/api/v1/write`), `-hours-ago` (default 24). Requires Remote Write receiver. See [Backends: Prometheus](../backends/prometheus.md).
+
+---
+
+## Backfill Mode
+
+Import a range of historic data points.
+
+```bash
+./epimetheus -mode=backfill -start-hours=48 -end-hours=0 -interval=1
+./epimetheus -mode=backfill -start-hours=168 -end-hours=0 -interval=6
+```
+
+**Options:** `-start-hours`, `-end-hours` (0 = now), `-interval` (hours between points). Requires Remote Write receiver.
+
+---
+
+## Auto Mode
+
+Route samples by timestamp age: &lt; 5 minutes → Pushgateway; ≥ 5 minutes → Remote Write. Use for mixed or unknown-age data.
+
+### Auto mode data flow
+
+```
+┌─────────────────┐ ┌─────────────────────────────────────┐
+│ CSV/JSON file │ ─────────────────▶ │ Epimetheus (auto mode) │
+│ (per-sample │ │ • Parse file (csv or json) │
+│ timestamps) │ │ • Route by sample age: │
+└─────────────────┘ │ < 5 min → Pushgateway │
+ │ ≥ 5 min → Remote Write │
+ └─────────────────────────────────────┘
+ │
+ ┌────────────────────┴────────────────────┐
+ ▼ ▼ │
+ ┌───────────────┐ ┌───────────────┐ │
+ │ Pushgateway │ │ Prometheus │ │
+ │ (realtime │ │ Remote Write │ │
+ │ samples) │ │ (historic │ │
+ └───────┬───────┘ │ samples) │ │
+ │ └───────────────┘ │
+ │ Scraped by Prometheus │
+ ▼ │
+ ┌───────────────┐ │
+ │ Prometheus │◀──────────────────────────────────┘
+ └───────────────┘
+```
+
+```bash
+./scripts/generate-test-data.sh
+./epimetheus -mode=auto -file=test-all-ages.csv
+```
+
+**Options:** `-file`, `-format` (csv or json), `-pushgateway`, `-prometheus`. See [Data Formats](data-formats.md).
diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md
new file mode 100644
index 0000000..adeea2b
--- /dev/null
+++ b/docs/guides/quickstart.md
@@ -0,0 +1,56 @@
+# Quick Start
+
+Minimal path to push metrics and see them in Prometheus or ClickHouse.
+
+## 1. Build
+
+```bash
+go build -o epimetheus cmd/epimetheus/main.go
+# Or: mage build
+```
+
+## 2. Run (Prometheus path)
+
+**Realtime mode** (Pushgateway + Prometheus):
+
+1. Deploy and expose Pushgateway (see [Kubernetes](../operations/kubernetes.md) or run Pushgateway locally).
+2. Ensure Prometheus scrapes Pushgateway (see [Setup: Prometheus](../operations/setup-prometheus.md)).
+3. Port-forward if needed, then run:
+
+```bash
+kubectl port-forward -n monitoring svc/pushgateway 9091:9091 &
+./epimetheus -mode=realtime -continuous
+```
+
+Metrics are pushed every 15 seconds. Stop with Ctrl+C.
+
+**Watch mode** (Remote Write; preserves timestamps):
+
+1. Enable the Prometheus Remote Write receiver (see [Setup: Prometheus](../operations/setup-prometheus.md)).
+2. Port-forward Prometheus, then run:
+
+```bash
+kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
+./epimetheus -mode=watch -file=mydata.csv -metric-name=myapp \
+ -prometheus=http://localhost:9090/api/v1/write
+```
+
+## 3. View
+
+- **Pushgateway:** http://localhost:9091
+- **Prometheus:** http://localhost:9090 (e.g. query `epimetheus_test_requests_total` or your metric name)
+- **Grafana:** Add Prometheus as a datasource and import the Epimetheus dashboard (see [Grafana Dashboard](../reference/grafana-dashboard.md)).
+
+## ClickHouse path (watch only)
+
+1. Run ClickHouse (e.g. `sudo systemctl start clickhouse-server` or Docker). See [Setup: ClickHouse](../operations/setup-clickhouse.md).
+2. Run watch mode with ClickHouse:
+
+```bash
+./epimetheus -mode=watch -file=test-data/watch-clickhouse-test.csv \
+ -metric-name=watch_test -clickhouse=http://localhost:8123 -prometheus=
+```
+
+3. Verify: `./scripts/verify-clickhouse.sh`
+
+For all modes and options see [Operating Modes](modes.md) and [CLI Reference](../reference/cli.md).
generated by cgit v1.2.3 (git 2.25.1) at 2026-05-17 12:40:51 +0000