From 946374df191a6ccb507bf3e60ae49d8f3c8c5743 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sun, 28 Dec 2025 20:36:56 +0200 Subject: Update tracing example with working distributed trace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaced old trace example with verified working trace that shows complete distributed tracing across all three services. Changes: - Updated curl command and response with actual working output - New trace ID: 4be1151c0bdcd5625ac7e02b98d95bd5 (old: 4e8d5a25ae6f8f8d737b46625920fbb9) - Added kubectl commands to search and fetch traces from Tempo API - Documented complete trace structure with 8 spans across 3 services: * Frontend: 3 spans (GET /api/process, frontend-process, POST) - 221ms * Middleware: 3 spans (POST /api/transform, middleware-transform, GET) - 186ms * Backend: 2 spans (GET /api/data, backend-get-data) - 104ms - Added detailed span annotations explaining each span's role - Included timing information showing distributed request flow - Documented W3C Trace Context header propagation This trace was generated after fixing health check noise by excluding /health endpoints from instrumentation, which allows API traces to be properly exported and visible in Tempo. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 --- ...etes-with-freebsd-part-X-OBSERVABILITY2.gmi.tpl | 79 ++++++++++++++++------ 1 file changed, 60 insertions(+), 19 deletions(-) diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X-OBSERVABILITY2.gmi.tpl b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X-OBSERVABILITY2.gmi.tpl index 4e2bbb7c..db39430a 100644 --- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X-OBSERVABILITY2.gmi.tpl +++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X-OBSERVABILITY2.gmi.tpl @@ -898,7 +898,7 @@ Let's generate a trace and examine it in Grafana. curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process ``` -**Response:** +**Response (HTTP 200):** ```json { @@ -907,7 +907,7 @@ curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process "data": { "id": 12345, "query_time_ms": 100.0, - "timestamp": "2025-12-28T17:15:48.454023", + "timestamp": "2025-12-28T18:35:01.064538", "value": "Sample data from backend service" }, "service": "backend" @@ -926,15 +926,57 @@ curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process } ``` -**2. Find the trace in Tempo:** +**2. Find the trace in Tempo via API:** -The request generated a distributed trace that spans all three services. The trace ID is: +After a few seconds (for batch export), search for recent traces: ``` -4e8d5a25ae6f8f8d737b46625920fbb9 +kubectl exec -n monitoring tempo-0 -- wget -qO- \ + 'http://localhost:3200/api/search?tags=service.namespace%3Dtracing-demo&limit=5' 2>/dev/null | \ + python3 -m json.tool ``` -**3. View the trace in Grafana:** +Returns traces including: + +```json +{ + "traceID": "4be1151c0bdcd5625ac7e02b98d95bd5", + "rootServiceName": "frontend", + "rootTraceName": "GET /api/process", + "durationMs": 221 +} +``` + +**3. Fetch complete trace details:** + +``` +kubectl exec -n monitoring tempo-0 -- wget -qO- \ + 'http://localhost:3200/api/traces/4be1151c0bdcd5625ac7e02b98d95bd5' 2>/dev/null | \ + python3 -m json.tool +``` + +**Trace structure (8 spans across 3 services):** + +``` +Trace ID: 4be1151c0bdcd5625ac7e02b98d95bd5 +Services: 3 (frontend, middleware, backend) + +Service: frontend + └─ GET /api/process 221.10ms (HTTP server span) + └─ frontend-process 216.23ms (custom business logic span) + └─ POST 209.97ms (HTTP client span to middleware) + +Service: middleware + └─ POST /api/transform 186.02ms (HTTP server span) + └─ middleware-transform 180.96ms (custom business logic span) + └─ GET 127.52ms (HTTP client span to backend) + +Service: backend + └─ GET /api/data 103.93ms (HTTP server span) + └─ backend-get-data 102.11ms (custom business logic span with 100ms sleep) +``` + +**4. View the trace in Grafana UI:** Navigate to: Grafana → Explore → Tempo datasource @@ -943,26 +985,25 @@ Search using TraceQL: { resource.service.namespace = "tracing-demo" } ``` -Or directly open the trace by ID in the search box: +Or directly open the trace by pasting the trace ID in the search box: ``` -4e8d5a25ae6f8f8d737b46625920fbb9 +4be1151c0bdcd5625ac7e02b98d95bd5 ``` -**4. Trace visualization:** +**5. Trace visualization:** -The trace shows the complete request flow with timing information: +The trace waterfall view shows the complete request flow with timing: -[SCREENSHOT PLACEHOLDER 1: Trace waterfall view showing Frontend → Middleware → Backend spans with timing (222ms total)] +[SCREENSHOT PLACEHOLDER 1: Trace waterfall view showing Frontend → Middleware → Backend spans with timing (221ms total)] -The trace details reveal: -* Frontend service received the request (span: GET /api/process) -* Frontend called Middleware service via HTTP (span: GET) -* Middleware performed transformation (span: middleware-transform) -* Middleware called Backend service (span: GET) -* Backend simulated database query with 100ms delay (span: GET /api/data) -* Total request time: 222ms +The trace reveals the distributed request flow: +* **Frontend (221ms)**: Receives GET /api/process, executes business logic, calls middleware +* **Middleware (186ms)**: Receives POST /api/transform, transforms data, calls backend +* **Backend (104ms)**: Receives GET /api/data, simulates database query with 100ms sleep +* **Total request time**: 221ms end-to-end +* **Span propagation**: W3C Trace Context headers automatically link all spans -**5. Service graph visualization:** +**6. Service graph visualization:** The service graph automatically generated from traces shows service dependencies: -- cgit v1.2.3