path: root/README.md

<div align="center">
  <img src="logo.png" alt="Epimetheus Logo" width="400"/>
</div>

# Epimetheus

A versatile Go tool for pushing metrics to Prometheus with support for both realtime and historic data ingestion.

## Why "Epimetheus"?

In Greek mythology, [Epimetheus](https://en.wikipedia.org/wiki/Epimetheus_(mythology)) is Prometheus's brother, whose name means "afterthought" or "hindsight" (while Prometheus means "forethought"). This name cleverly captures the tool's purpose: bringing data to Prometheus **after** collection, whether it's historic data from hours, days, or weeks ago, or realtime data pushed on-demand.

While Epimetheus is sometimes depicted as foolish in myths (he accepted Pandora's box despite warnings), this tool embraces the "afterthought" aspect productively - it's never too late to bring your metrics home to Prometheus!

## Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│                            Epimetheus                                   │
│                     (Metrics Ingestion Tool)                            │
│                                                                         │
│  Modes:                                                                 │
│  • Realtime  - Current metrics (< 5 min old)                          │
│  • Historic  - Historic metrics (≥ 5 min old)                         │
│  • Backfill  - Range of historic data                                 │
│  • Auto      - Automatic routing based on timestamp age               │
└─────────────────────────────────────────────────────────────────────────┘
         │                                           │
         │ Realtime Data                            │ Historic Data
         │ (via HTTP POST)                          │ (via Remote Write API)
         │ Uses "now" timestamp                     │ Preserves timestamps
         ▼                                           ▼
┌─────────────────────┐                    ┌─────────────────────┐
│   Pushgateway       │                    │    Prometheus       │
│   (Port 9091)       │                    │    (Port 9090)      │
│                     │                    │                     │
│ • Buffers metrics   │                    │ Remote Write API:   │
│ • Scraped by        │──── Scraped ─────▶ │ /api/v1/write       │
│   Prometheus        │    every 15-30s    │                     │
│ • No timestamp      │                    │ Feature Required:   │
│   preservation      │                    │ --enable-feature=   │
│                     │                    │  remote-write-      │
│                     │                    │  receiver           │
└─────────────────────┘                    └─────────────────────┘
                                                      │
                                                      │ Prometheus Query API
                                                      │ /api/v1/query
                                                      ▼
                                           ┌─────────────────────┐
                                           │     Grafana         │
                                           │   (Port 3000)       │
                                           │                     │
                                           │ • Prometheus as     │
                                           │   datasource        │
                                           │ • Dashboards:       │
                                           │   - Epimetheus      │
                                           │     Test Metrics    │
                                           │ • Auto-refresh      │
                                           └─────────────────────┘
```

### Data Flow

1. **Realtime Path** (for current data):
   - Epimetheus → Pushgateway (HTTP POST)
   - Prometheus scrapes Pushgateway periodically
   - Timestamp = "now" when Prometheus scrapes

2. **Historic Path** (for old data):
   - Epimetheus → Prometheus Remote Write API (HTTP POST)
   - Direct write to Prometheus TSDB
   - Timestamp preserved from original data

3. **Visualization**:
   - Grafana queries Prometheus
   - Displays metrics in dashboards
   - Auto-refresh every 10 seconds

## Overview

**epimetheus** is a standalone binary that:
- **Generates** realistic example metrics simulating production applications
- **Pushes** metrics via Pushgateway (realtime) or Remote Write API (historic)
- **Automatically detects** timestamp age and chooses the optimal ingestion method
- **Supports** multiple data formats (CSV, JSON) and all Prometheus metric types
- **Provides** Grafana dashboard for visualizing test metrics

## Quick Start

### 1. Deploy Pushgateway (one-time setup)

The Pushgateway Helm chart is available in the [conf repository](https://codeberg.org/snonux/conf) at `f3s/pushgateway/helm-chart`.

```bash
# Clone the conf repository if you haven't already
git clone https://codeberg.org/snonux/conf.git
cd conf/f3s/pushgateway/helm-chart

# Deploy Pushgateway
helm upgrade --install pushgateway . -n monitoring --create-namespace
```

Alternatively, deploy Pushgateway using the official chart:

```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install pushgateway prometheus-community/prometheus-pushgateway -n monitoring --create-namespace
```

### 2. Run in Realtime Mode

```bash
# Port-forward Pushgateway
kubectl port-forward -n monitoring svc/pushgateway 9091:9091 &

# Push test metrics continuously
cd /home/paul/git/conf/f3s/epimetheus
./epimetheus -mode=realtime -continuous
```

The binary pushes metrics every 15 seconds. Press Ctrl+C to stop.

### 3. View Metrics

```bash
# Pushgateway UI
open http://localhost:9091

# Prometheus UI
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
open http://localhost:9090
```

## Operating Modes

### 👁️ Watch Mode
Monitor CSV files for changes and push metrics to Prometheus with file modification timestamps.

**Works with ANY CSV format** - automatically detects numeric vs string columns and sanitizes names.

**NEW: Automatic DNS Resolution** - IP addresses are automatically resolved to hostnames for better observability in Grafana.

```bash
./epimetheus -mode=watch \
    -file=mydata.csv \
    -metric-name=myapp \
    -prometheus=http://localhost:9090/api/v1/write
```

**Features:**
- 🔍 **Format-agnostic**: Works with any tabular CSV structure
- 📊 **Automatic detection**: Numeric columns → metrics, String columns → labels
- 🏷️ **Name sanitization**: `min(potatoes)`, `avg(time)`, `p99(latency)` → valid metric names
- 🌐 **DNS Resolution**: IP addresses → hostnames (e.g., `10.50.52.61` → `foo.example.lan`)
- 💾 **Smart Caching**: In-memory cache prevents redundant DNS lookups
- ⏱️ **Timestamp preservation**: Uses file modification time
- 🔄 **Continuous monitoring**: Polls file every 1 second
- 💪 **Error resilient**: Continues watching despite failures
- 🎯 **Remote Write**: Pushes to Prometheus (preserves timestamps)

**CSV Format:**
Works with any tabular CSV:
- First row: column headers (automatically sanitized)
- Subsequent rows: data values
- Column names can be anything: `min(x)`, `avg(y)`, `p99(latency)`, etc.

**Example 1** - Web metrics:
```csv
avg(response_time),p99(latency),endpoint,method
45.2,120.5,/api/users,GET
52.1,135.8,/api/orders,POST
```

Generates:
```promql
web_avg_response_time{endpoint="/api/users",method="GET"} 45.2
web_p99_latency{endpoint="/api/users",method="GET"} 120.5
web_avg_response_time{endpoint="/api/orders",method="POST"} 52.1
web_p99_latency{endpoint="/api/orders",method="POST"} 135.8
```

**Example 2** - Food metrics:
```csv
min(potatoes),last(coke),avg(price),country,store_type
5.2,10.5,12.99,USA,grocery
3.8,8.2,9.99,Canada,convenience
```

Generates:
```promql
food_min_potatoes{country="USA",store_type="grocery"} 5.2
food_last_coke{country="USA",store_type="grocery"} 10.5
food_avg_price{country="USA",store_type="grocery"} 12.99
# ... etc
```

Each row generates N samples (N = number of numeric columns).

See [CSV-FORMAT-FLEXIBILITY.md](CSV-FORMAT-FLEXIBILITY.md) for more examples.

**Options:**
- `-file` - CSV file to watch (required)
- `-metric-name` - Base metric name (required, e.g., `food`, `network`, `database`)
- `-prometheus` - Prometheus Remote Write URL (default: http://localhost:9090/api/v1/write)
- `-clickhouse` - ClickHouse HTTP URL (e.g. http://localhost:8123) to also ingest metrics
- `-clickhouse-table` - ClickHouse table name (default: epimetheus_metrics)
- `-job` - Job name for metrics (default: example_metrics_pusher)
- `-resolve-ip-labels` - Additional IP labels to resolve via DNS (default: ip is always resolved)

**ClickHouse Support:**
Watch mode can ingest to ClickHouse in addition to (or instead of) Prometheus:

```bash
# Ingest to both Prometheus and ClickHouse
./epimetheus -mode=watch -file=data.csv -metric-name=myapp \
    -prometheus=http://localhost:9090/api/v1/write \
    -clickhouse=http://localhost:8123

# ClickHouse only (use -prometheus= to disable Prometheus)
./epimetheus -mode=watch -file=test-data/watch-clickhouse-test.csv \
    -metric-name=watch_test -clickhouse=http://localhost:8123 -prometheus=

# Verify data in ClickHouse
./verify-clickhouse.sh
```

**DNS Resolution:**
By default, the `ip` label is automatically resolved to a hostname. To resolve additional IP labels:

```bash
./epimetheus -mode=watch \
    -file=network.csv \
    -metric-name=network \
    -resolve-ip-labels=source_ip,dest_ip
```

This will resolve: `ip` (default) + `source_ip` + `dest_ip`

**Example:**
- Input: `ip="10.50.52.61"`
- Output: `ip="foo.example.lan"`
- Failed lookups: IP remains unchanged

**Documentation:**
- [DNS-RESOLUTION-FEATURE.md](DNS-RESOLUTION-FEATURE.md) - Complete DNS resolution guide
- [CSV-FORMAT-FLEXIBILITY.md](CSV-FORMAT-FLEXIBILITY.md) - Works with ANY CSV format
- [DTAIL-METRICS-EXAMPLE.md](DTAIL-METRICS-EXAMPLE.md) - Detailed dtail.csv example

### 🔄 Realtime Mode (Default)
Push current metrics to Pushgateway with "now" timestamp.

```bash
./epimetheus -mode=realtime -continuous
```

**Options:**
- `-pushgateway` - Pushgateway URL (default: http://localhost:9091)
- `-job` - Job name (default: example_metrics_pusher)
- `-continuous` - Keep pushing every 15 seconds

### ⏰ Historic Mode
Push a single datapoint from the past using Remote Write API.

```bash
# Port-forward Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &

# Push data from 24 hours ago
./epimetheus -mode=historic -hours-ago=24
```

**Options:**
- `-prometheus` - Prometheus URL (default: http://localhost:9090/api/v1/write)
- `-hours-ago` - Hours in the past (default: 24)

### 📦 Backfill Mode
Import a range of historic data points.

```bash
# Backfill last 48 hours with 1-hour intervals
./epimetheus -mode=backfill -start-hours=48 -end-hours=0 -interval=1

# Backfill last week with 6-hour intervals
./epimetheus -mode=backfill -start-hours=168 -end-hours=0 -interval=6
```

**Options:**
- `-start-hours` - Start time in hours ago
- `-end-hours` - End time in hours ago (0 = now)
- `-interval` - Interval between points in hours

### 🤖 Auto Mode (Recommended!)
Automatically detect timestamp age and route to the correct ingestion method.

```bash
# Generate test data
./generate-test-data.sh

# Import mixed current and historic data
./epimetheus -mode=auto -file=test-all-ages.csv
```

**Detection Logic:**
- Data < 5 minutes old → Pushgateway (realtime)
- Data ≥ 5 minutes old → Remote Write (historic)

**Options:**
- `-file` - Input file path
- `-format` - Data format: csv or json (default: csv)
- `-pushgateway` - Pushgateway URL
- `-prometheus` - Prometheus Remote Write URL

## Data Formats

### CSV Format

```csv
# Format: metric_name,labels,value,timestamp_ms
# Labels: key1=value1;key2=value2
epimetheus_test_requests_total,instance=web1;env=prod,100,1767125148000
epimetheus_test_temperature_celsius,instance=web2,22.5,1767038748000

# Timestamp is optional (uses "now" if omitted)
epimetheus_test_active_connections,instance=web3,42,
```

### JSON Format

```json
[
  {
    "metric": "epimetheus_test_requests_total",
    "labels": {"instance": "web1", "env": "prod"},
    "value": 100,
    "timestamp_ms": 1767125148000
  },
  {
    "metric": "epimetheus_test_temperature_celsius",
    "labels": {"instance": "web2"},
    "value": 22.5,
    "timestamp_ms": 1767038748000
  }
]
```

## Test Metrics

All generated metrics use the `epimetheus_test_` prefix to clearly identify them as test data.

### Counter: `epimetheus_test_requests_total`
- **Type:** Counter (monotonically increasing)
- **Description:** Total number of requests processed
- **Use case:** Counting total events, requests, errors

### Gauge: `epimetheus_test_active_connections`
- **Type:** Gauge (can increase or decrease)
- **Description:** Current number of active connections (0-100)
- **Use case:** Current state measurements, capacity

### Gauge: `epimetheus_test_temperature_celsius`
- **Type:** Gauge
- **Description:** Current temperature in Celsius (0-50°C)
- **Use case:** Environmental monitoring

### Histogram: `epimetheus_test_request_duration_seconds`
- **Type:** Histogram (distribution)
- **Description:** Request duration distribution
- **Buckets:** 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds
- **Use case:** Latency measurements, SLO tracking

### Labeled Counter: `epimetheus_test_jobs_processed_total`
- **Type:** Counter with labels
- **Description:** Jobs processed by type and status
- **Labels:**
  - `job_type`: email, report, backup
  - `status`: success, failed
- **Use case:** Categorized counting, multi-dimensional metrics

## Grafana Dashboard

A comprehensive dashboard is available showcasing all test metrics.

### Dashboard Features

- **8 Panels:**
  1. Request Rate (line graph)
  2. Total Requests (stat panel)
  3. Active Connections (gauge with thresholds)
  4. Temperature (gauge with thresholds)
  5. Request Duration Histogram (p50, p90, p99)
  6. Average Request Duration (stat)
  7. Jobs Processed by Type (bar gauge)
  8. Jobs Status Breakdown (table)

- **Auto-refresh:** Every 10 seconds
- **Time range:** Last 15 minutes (customizable)
- **Dark theme optimized**

### Deploy Dashboard

#### Option 1: Helm/Kubernetes ConfigMap (Recommended)

```bash
# Deploy via Kubernetes ConfigMap
kubectl apply -f ../prometheus/epimetheus-dashboard.yaml
```

The dashboard will be automatically discovered by Grafana.

#### Option 2: Manual Import

```bash
# Port-forward Grafana
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

# Open Grafana
open http://localhost:3000

# Go to Dashboards → Import → Upload grafana-dashboard.json
```

#### Option 3: Automated Script

```bash
# Deploy via API
./deploy-dashboard.sh

# Or with custom credentials
GRAFANA_URL="http://localhost:3000" \
GRAFANA_USER="admin" \
GRAFANA_PASSWORD="yourpassword" \
./deploy-dashboard.sh
```

## Example Queries

### Basic Queries

```promql
# View total requests
epimetheus_test_requests_total

# View request rate over last 5 minutes
rate(epimetheus_test_requests_total[5m])

# View current active connections
epimetheus_test_active_connections

# View current temperature
epimetheus_test_temperature_celsius
```

### Histogram Queries

```promql
# 95th percentile request duration
histogram_quantile(0.95, rate(epimetheus_test_request_duration_seconds_bucket[5m]))

# 50th percentile (median)
histogram_quantile(0.50, rate(epimetheus_test_request_duration_seconds_bucket[5m]))

# Average request duration
rate(epimetheus_test_request_duration_seconds_sum[5m]) /
rate(epimetheus_test_request_duration_seconds_count[5m])
```

### Labeled Counter Queries

```promql
# Failed jobs by type
epimetheus_test_jobs_processed_total{status="failed"}

# Job success rate
rate(epimetheus_test_jobs_processed_total{status="success"}[5m]) /
rate(epimetheus_test_jobs_processed_total[5m])

# Total jobs by type
sum by (job_type) (epimetheus_test_jobs_processed_total)
```

### Curl Examples

```bash
# Port-forward Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &

# Query total requests
curl -s "http://localhost:9090/api/v1/query?query=epimetheus_test_requests_total" | jq .

# Query temperature
curl -s "http://localhost:9090/api/v1/query?query=epimetheus_test_temperature_celsius" | jq .

# Query request rate
curl -s "http://localhost:9090/api/v1/query?query=rate(epimetheus_test_requests_total[5m])" | jq .

# Query histogram p95
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.95,rate(epimetheus_test_request_duration_seconds_bucket[5m]))" | jq .
```

## Time Range Limitations

### ✅ Supported Time Ranges

| Time Range | Status | Method |
|------------|--------|--------|
| Current (< 5 min) | ✅ Works | Pushgateway |
| 1 hour old | ✅ Works | Remote Write |
| 1 day old | ✅ Works | Remote Write |
| 1 week old | ✅ Works | Remote Write |
| 1 month old | ✅ Works | Remote Write |

### ⚠️ Potential Issues

- **Future timestamps:** Rejected (> 5 minutes in future)
- **Very old data (6+ months):** May be rejected depending on Prometheus retention
- **Years old:** Likely rejected - use `promtool tsdb create-blocks-from` instead
- **Out-of-order samples:** Can't insert older data into existing time series (use different labels)

### Prometheus Configuration

Check your retention settings:

```bash
# View retention
kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
  -o jsonpath='{.spec.retention}'

# Default is typically 15 days
```

For very old data:
- Increase retention in Prometheus config
- Enable out-of-order ingestion (experimental)
- Use `promtool` for direct TSDB block creation

## Project Structure

```
epimetheus/
├── cmd/
│   └── epimetheus/
│       └── main.go              # Main entry point
├── internal/
│   ├── config/                  # Configuration
│   ├── metrics/                 # Metric generators
│   ├── parser/                  # CSV/JSON parsers (includes tabular CSV)
│   ├── ingester/                # Pushgateway & Remote Write ingesters
│   └── watcher/                 # File watcher for watch mode
├── epimetheus            # Compiled binary
├── grafana-dashboard.json       # Grafana dashboard definition
├── deploy-dashboard.sh          # Dashboard deployment script
├── generate-test-data.sh        # Test data generator
├── run.sh                       # Helper script
└── README.md                    # This file
```

## Setup Requirements

### 1. Enable Prometheus Remote Write Receiver ⚠️ **REQUIRED for Historic Data**

**IMPORTANT**: To use historic mode, backfill mode, or auto mode with old data, you **must** enable the Prometheus Remote Write receiver. Without this feature, Epimetheus can only push realtime data via Pushgateway.

The Remote Write receiver is configured in the [conf repository](https://codeberg.org/snonux/conf) at `f3s/prometheus/persistence-values.yaml`:

```yaml
# In prometheus/persistence-values.yaml (from conf repository)
prometheus:
  prometheusSpec:
    # Enable Remote Write receiver endpoint and Admin API (Prometheus 3.x syntax)
    additionalArgs:
      - name: web.enable-remote-write-receiver
        value: ""
      - name: web.enable-admin-api
        value: ""

    # Enable out-of-order ingestion for backfilling
    # Allows writing data points older than existing data for the same time series
    enableFeatures:
      - exemplar-storage
      - otlp-write-receiver

    # Allow backfilling up to 31 days in the past (provides 1-day buffer for 30-day datasets)
    tsdb:
      outOfOrderTimeWindow: 744h  # 31 days
```

**What This Enables:**
- **Remote Write API**: HTTP endpoint at `/api/v1/write` for ingesting metrics with custom timestamps
- **Admin API**: HTTP endpoints at `/api/v1/admin/tsdb/*` for data deletion and management
- **Out-of-Order Ingestion**: Allows writing data points older than existing data for the same time series
- **31-Day Window**: Can backfill data up to 31 days in the past (provides 1-day buffer for 30-day datasets)

After updating the configuration, upgrade your Prometheus installation:

```bash
cd conf/f3s/prometheus
just upgrade  # Or manually:
# helm upgrade prometheus prometheus-community/kube-prometheus-stack \
#   -n monitoring -f persistence-values.yaml
```

Verify the features are enabled:

```bash
# Check Remote Write receiver flag
kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
  -o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-remote-write-receiver"

# Check out-of-order time window
kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
  -o jsonpath='{.spec.tsdb.outOfOrderTimeWindow}'
# Should output: 744h

# Check admin API flag
kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
  -o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-admin-api"
```

**Performance Considerations:**

This configuration is designed for ad-hoc troubleshooting and development, **NOT production use**. Enabling these features has trade-offs:

- **Increased Memory Usage**: Out-of-order ingestion requires additional memory for buffering and sorting time series
- **Higher TSDB Overhead**: Prometheus TSDB needs to handle non-sequential writes, increasing disk I/O
- **Query Performance**: Queries may be slower due to fragmented data blocks
- **Storage Amplification**: Out-of-order samples can trigger additional compactions, increasing storage usage

**Recommendation for Production:**
- Keep `outOfOrderTimeWindow` as small as possible (or disabled)
- Monitor Prometheus memory and disk usage closely
- Use Remote Write only when necessary
- Consider using dedicated testing/development Prometheus instances

**Note**: The syntax changed in Prometheus 3.x - use `additionalArgs` with `web.enable-remote-write-receiver` instead of the deprecated `enableFeatures: [remote-write-receiver]`.

### 2. Update Prometheus Scrape Config

Ensure Pushgateway is in scrape targets:

```yaml
# additional-scrape-configs.yaml
- job_name: 'pushgateway'
  honor_labels: true
  static_configs:
    - targets:
      - 'pushgateway.monitoring.svc.cluster.local:9091'
```

Apply the configuration:

```bash
kubectl create secret generic additional-scrape-configs \
  --from-file=/home/paul/git/conf/f3s/prometheus/additional-scrape-configs.yaml \
  --dry-run=client -o yaml -n monitoring | kubectl apply -f -
```

## Building from Source

### Using Mage (Recommended)

This project includes a [Magefile](./MAGEFILE.md) for easy building, testing, and running:

```bash
# Install Mage (one-time setup)
go install github.com/magefile/mage@latest

# Build binary
mage build

# Run tests
mage test

# Run with coverage report
mage testCoverage

# Run in realtime mode
mage run

# See all available targets
mage -l
```

See [MAGEFILE.md](./MAGEFILE.md) for complete documentation.

### Using Go directly

```bash
# Build binary
go build -o epimetheus cmd/epimetheus/main.go

# Run tests
go test ./... -v

# Check test coverage
go test ./... -cover
```

## Troubleshooting

### Binary can't connect to Pushgateway

```bash
# Check port-forward is running
ps aux | grep "port-forward.*9091"

# Restart port-forward
kubectl port-forward -n monitoring svc/pushgateway 9091:9091
```

### Metrics not appearing in Prometheus

```bash
# Check Pushgateway has metrics
curl http://localhost:9091/metrics | grep "prometheus_pusher_test"

# Check Prometheus scrape targets
# Open http://localhost:9090/targets - look for "pushgateway" job

# Check Prometheus logs
kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus
```

### "Remote write receiver not enabled" error

```bash
# Verify feature is enabled
kubectl logs -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 | grep "remote-write-receiver"

# Should see: msg="Experimental features enabled" features=[remote-write-receiver]
```

### "Out of order sample" error

This occurs when trying to insert data older than existing data for the same time series.

**Solutions:**
- Use different job labels for historic data (e.g., `job="historic_data"`)
- Enable out-of-order ingestion in Prometheus (experimental)
- Ensure backfill goes from oldest to newest

### Dashboard not appearing in Grafana

```bash
# Check ConfigMap exists
kubectl get configmap -n monitoring | grep epimetheus

# Check labels
kubectl get configmap epimetheus-dashboard -n monitoring -o yaml | grep "grafana_dashboard"

# Restart Grafana to force reload
kubectl rollout restart deployment/prometheus-grafana -n monitoring
```

## Architecture

```
┌─────────────────┐
│  Go Binary      │
│ (prometheus-    │──Push realtime──┐
│  pusher)        │                 │
└─────────────────┘                 ▼
         │                  ┌──────────────────┐
         │                  │  Pushgateway     │◄──Scrape──┐
         │                  │  (Port 9091)     │           │
         │                  └──────────────────┘           │
         │                                                 │
         └──Push historic──────────────────┐              │
                                            ▼              │
                                  ┌─────────────────┐     │
                                  │   Prometheus    │◄────┘
                                  │   (Port 9090)   │
                                  │ Remote Write API│
                                  └─────────────────┘
                                           │
                                           │ Datasource
                                           ▼
                                  ┌─────────────────┐
                                  │    Grafana      │
                                  │   (Port 3000)   │
                                  │   Dashboards    │
                                  └─────────────────┘
```

## Best Practices

### When to Use Pushgateway vs. Remote Write

**Use Pushgateway (realtime mode):**
- Short-lived batch jobs
- Service-level metrics
- Jobs behind firewalls
- Current/recent data (< 5 minutes old)

**Use Remote Write (historic mode):**
- Historic data import
- Backfilling gaps
- Data migration
- Data older than 5 minutes

**Use Auto Mode:**
- Mixed current and historic data
- Importing from files
- Unknown timestamp ages
- General-purpose ingestion

### Metric Design

- **Use appropriate metric types:**
  - Counter for cumulative values (requests, errors)
  - Gauge for point-in-time values (temperature, connections)
  - Histogram for distributions (latency, sizes)

- **Label cardinality:**
  - Include meaningful labels
  - Avoid high-cardinality labels (user IDs, timestamps)
  - Keep label combinations reasonable (< 1000 per metric)

- **Naming conventions:**
  - Use descriptive names
  - Include units in gauge names (\_celsius, \_bytes)
  - Use \_total suffix for counters

## Cleanup

### Cleaning Up Benchmark Data from Prometheus

For cleaning up benchmark metrics from Prometheus, use the provided cleanup script:

```bash
# Port-forward to Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &

# Run the cleanup script
./cleanup-benchmark-data.sh
```

The script will:
1. Delete all `epimetheus_benchmark_*` metrics using the Prometheus Admin API
2. Clean up tombstones to free disk space
3. Provide clear success/error feedback

**Manual cleanup** (if you prefer):

```bash
# Delete specific metric
curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=epimetheus_benchmark_cpu_usage'

# Clean up tombstones
curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
```

### Other Cleanup Tasks

```bash
# Stop port-forwards
pkill -f "port-forward.*9091"
pkill -f "port-forward.*9090"
pkill -f "port-forward.*3000"

# Delete test metrics from Pushgateway
curl -X DELETE http://localhost:9091/metrics/job/example_metrics_pusher

# Uninstall Pushgateway (if needed)
helm uninstall pushgateway -n monitoring
```

## MacOS Setup

### Basic Installation

```bash
brew install prometheus
brew install grafana
go install github.com/prometheus/pushgateway@latest
brew services start grafana
brew services start prometheus
~/go/bin/pushgateway &
```

Once done, login to http://localhost:3000 as admin:admin, you will be prompted to change the password. Afterwards, add http://localhost:9090 as a Prometheus datasource.

### Enable Remote Write Receiver (Required for Watch Mode)

⚠️ **Important**: Watch mode, historic mode, backfill mode, and auto mode require the Prometheus Remote Write receiver to be enabled.

#### Option 1: Permanent Configuration (Recommended)

Edit the Prometheus arguments file:

```bash
# Edit the arguments file
nano /opt/homebrew/etc/prometheus.args
```

Add this line at the end:
```
--web.enable-remote-write-receiver
```

The complete file should look like:
```
--config.file /opt/homebrew/etc/prometheus.yml
--web.listen-address=127.0.0.1:9090
--storage.tsdb.path /opt/homebrew/var/prometheus
--web.enable-remote-write-receiver
--web.enable-admin-api
```

**Note:** `--web.enable-admin-api` is optional but recommended for easier data management (allows deleting old metrics).

Restart Prometheus:
```bash
brew services restart prometheus
```

Verify it's working:
```bash
# Check Prometheus is healthy
curl http://localhost:9090/-/healthy

# Test Remote Write endpoint (should return 400, not 404)
curl -X POST http://localhost:9090/api/v1/write
```

#### Option 2: Temporary (For Testing)

Stop the service and start manually:

```bash
# Stop brew service
brew services stop prometheus

# Start with Remote Write enabled
prometheus --web.enable-remote-write-receiver
```

Keep this terminal open. In another terminal, run your epimetheus commands.

**Note**: This only lasts until you stop the terminal. Use Option 1 for permanent setup.

### Clearing Old Metrics (Optional)

If you need to delete old metrics and start fresh:

```bash
# Delete specific metrics (e.g., blockstore)
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~"blockstore_.*"}'

# Clean up deleted data
curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

# Wait a moment for cleanup
sleep 2
```

**Note:** Admin API must be enabled (add `--web.enable-admin-api` to prometheus.args).

### Verify Setup

Once Remote Write is enabled, test watch mode:

```bash
# Create a test CSV
cat > /tmp/test.csv << EOF
status,count,method
200,100,GET
404,50,POST
EOF

# Watch the file
./epimetheus -mode=watch \
    -file=/tmp/test.csv \
    -metric-name=test \
    -prometheus=http://localhost:9090/api/v1/write
```

You should see:
```
✅ Successfully pushed X samples to Prometheus
```

Query in Prometheus (http://localhost:9090):
```promql
{__name__=~"test_.*"}
```

## Additional Resources

- [Prometheus Documentation](https://prometheus.io/docs/)
- [Pushgateway Documentation](https://github.com/prometheus/pushgateway)
- [Prometheus Remote Write Spec](https://prometheus.io/docs/concepts/remote_write_spec/)
- [Grafana Documentation](https://grafana.com/docs/)

## Version

Current version: 0.0.0

## License

See LICENSE file for details.