This source did not publish a separate summary. Review SKILL.md before using the skill.
SKILL.md
Monitoring Setup Guide Skill
Produce a complete monitoring setup guide for a service — defining exactly what to measure, how to structure logs, how to configure alerts with actionable thresholds, and how to build dashboards that answer real operational questions. A good monitoring guide eliminates "we don't know what's happening in production" as a root cause category, and gives on-call engineers a single source of truth for what healthy looks like.
Required Inputs
Ask for these if not already provided:
Service name and description — what the service does and its role in the system
Tech stack — language, framework, and infrastructure (e.g. Go/gRPC on Kubernetes, Python/FastAPI on ECS)
Current monitoring tooling — Datadog, Prometheus + Grafana, CloudWatch, New Relic, Honeycomb, or none yet
Key user journeys — the 2–4 most important things a user or consumer does with the service (these drive what to alert on)
Existing alerts — paste any existing alert configurations or describe what's currently monitored
Was it healthy in the past, and is it trending worse? (dashboards + SLO tracking)
Why did something fail? (logs + traces)
This guide defines the answers for [Service Name]. Every alert must be actionable — if an on-call engineer cannot take a specific action in response to the alert, the alert should not exist.
Key user journeys monitored:
Journey 1: [e.g. "User submits a payment — POST /charges, receives confirmation"]
Journey 2: [e.g. "User views transaction history — GET /transactions"]
Apply the four golden signals specifically to [Service Name]:
Latency
Latency measures how long requests take to complete. Track it separately for successful and failed requests — slow failures hide behind fast errors if you only measure aggregate latency.
Metric
Description
Source
Dimensions
[service].request.duration_ms
End-to-end request latency
Application instrumentation
endpoint, method, status_code
[service].db.query_duration_ms
Database query latency
ORM / query instrumentation
query_name, table
[service].external.request_duration_ms
Outbound call latency to dependencies
HTTP client instrumentation
target_service, endpoint
[service].queue.processing_duration_ms
Time to process one message (if applicable)
Consumer instrumentation
queue_name, message_type
Latency SLO targets:
Endpoint / operation
p50 target
p95 target
p99 target
GET /api/v1/[resource]
< [50] ms
< [200] ms
< [500] ms
POST /api/v1/[resource]
< [100] ms
< [400] ms
< [1000] ms
GET /health
< [10] ms
< [20] ms
< [50] ms
[Background job name]
< [5] sec
< [15] sec
< [60] sec
Traffic
Traffic measures demand on the system. Use it to detect unexpected spikes, traffic drops (which can indicate upstream failures), and to capacity-plan.
Metric
Description
Source
[service].request.count
Requests per second
Application / load balancer
[service].request.count_by_endpoint
RPS broken down by endpoint
Application
[service].queue.messages_consumed_per_second
Consumer throughput
Queue consumer
[service].queue.depth
Messages waiting in queue
Queue metrics
Traffic baselines (update after observing production for 2+ weeks):
Time period
Expected RPS
Low-traffic floor
Spike ceiling
Peak (weekday business hours)
[N] RPS
[N × 0.5] RPS
[N × 5] RPS
Off-peak (nights/weekends)
[N × 0.2] RPS
[N × 0.05] RPS
[N] RPS
Errors
Errors measure the fraction of requests that fail. Distinguish between client errors (4xx — caller is doing something wrong) and server errors (5xx — the service is broken).
Metric
Description
Alert on?
[service].request.error_rate
5xx errors / total requests
Yes — see alert rules
[service].request.client_error_rate
4xx errors / total requests
Threshold alert — sudden spike may indicate API misuse
[service].dependency.error_rate
Errors calling downstream dependencies
Yes — upstream health signal
[service].queue.dlq_depth
Messages in dead-letter queue
Yes — indicates processing failures
Saturation
Saturation measures how "full" the service is — how close to maximum capacity are the constrained resources.
Resource
Metric
Alert threshold
Source
CPU
[service].cpu.utilisation_pct
>80% sustained 5 min
Container / VM metrics
Memory
[service].memory.utilisation_pct
>85% sustained 5 min
Container / VM metrics
DB connections
[service].db.connection_pool.utilisation_pct
>75%
Application / DB metrics
Thread pool / goroutines
[service].runtime.goroutine_count / thread_count
>N (establish baseline)
Runtime metrics
Disk (if applicable)
[service].disk.utilisation_pct
>75%
Infrastructure
Queue depth (if applicable)
[service].queue.depth
>[backlog threshold]
Queue metrics
3. Business Metrics
Beyond the golden signals, track metrics that measure whether the service is delivering business value. These matter for SLO reporting and product dashboards.
Metric
Description
Source
Alert?
[service].[primary_action].success_rate
[e.g. "Payment success rate"]
Application
Yes — if drops >5% vs 1h average
[service].[primary_action].count
[e.g. "Payments processed per minute"]
Application
Yes — sudden drop (traffic anomaly)
[service].[resource].created_per_hour
[e.g. "New accounts created"]
Application / DB
No — informational
[service].cache.hit_rate
Fraction of requests served from cache
Cache instrumentation
Yes — if drops below [60]%
[service].job.[name].success_rate
[Background job success rate]
Job framework
Yes — if drops below [99]%
4. Log Strategy
Structured Logging Schema
All logs must be structured JSON. Do not emit unstructured text logs in production. Every log line must include the mandatory fields.
The primary service dashboard must answer "is the service healthy right now?" at a glance. Use this layout:
┌─────────────────────────────────────────────────────────────────────┐
│ [SERVICE NAME] — Service Health Dashboard [Time range ▼] │
├───────────────┬───────────────┬───────────────┬─────────────────────┤
│ Error rate │ p99 Latency │ RPS (current)│ SLO budget remaining│
│ [BIG NUMBER] │ [BIG NUMBER] │ [BIG NUMBER] │ [BIG NUMBER / days] │
│ vs SLO: 0.1% │ vs SLO: 500ms│ vs avg: [N] │ [Error budget gauge]│
├───────────────┴───────────────┴───────────────┴─────────────────────┤
│ Error rate over time (24h) │
│ [Time series: 5xx rate line, SLO threshold line] │
├─────────────────────────────────┬───────────────────────────────────┤
│ Latency percentiles over time │ Request throughput over time │
│ [Lines: p50, p95, p99, p999] │ [Bars: RPS by endpoint] │
│ [SLO threshold horizontal line]│ │
├─────────────────────────────────┴───────────────────────────────────┤
│ Latency heatmap (all requests — shows distribution shape) │
├─────────────────────────────────┬───────────────────────────────────┤
│ CPU utilisation over time │ Memory utilisation over time │
│ [All instances/pods — lines] │ [All instances/pods — lines] │
│ [Alert threshold: 80%] │ [Alert threshold: 85%] │
├─────────────────────────────────┴───────────────────────────────────┤
│ DB: connection pool utilisation│ DB: query latency (p99 per query)│
├─────────────────────────────────┴───────────────────────────────────┤
│ [Business metric 1 over time] │ [Business metric 2 over time] │
│ e.g. Payment success rate │ e.g. Orders created/min │
└─────────────────────────────────┴───────────────────────────────────┘
Second dashboard — Dependency Health:
┌─────────────────────────────────────────────────────────────────────┐
│ [SERVICE NAME] — Dependency Health │
├─────────────────────────────────────────────────────────────────────┤
│ For each dependency: error rate | latency | current status │
│ [Database] [N]% errors | [N]ms p99 | ● Healthy / ⚠ Degraded │
│ [Redis] [N]% errors | [N]ms p99 | ● Healthy │
│ [External API][N]% errors | [N]ms p99 | ● Healthy │
├─────────────────────────────────────────────────────────────────────┤
│ Outbound call latency over time (one line per dependency) │
├─────────────────────────────────────────────────────────────────────┤
│ Circuit breaker / fallback state (if implemented) │
└─────────────────────────────────────────────────────────────────────┘
8. Observability Debt Analysis
Honest assessment of what is missing today and what the priority to add it is:
Gap
Impact
Priority
Effort
Owner
Target date
[e.g. No distributed tracing — can't see cross-service latency]
High — blind to dependency issues
P1
[2 days]
[Name]
[Date]
[e.g. No business metric alerts — only infra alerts]
High — silent business failures
P1
[1 day]
[Name]
[Date]
[e.g. Logs are unstructured text — not searchable]
Medium — slow incident investigation
P2
[3 days]
[Name]
[Date]
[e.g. No dead-letter queue monitoring]
Medium — failed messages go unnoticed
P2
[4 hours]
[Name]
[Date]
[e.g. Alert thresholds not calibrated to production baseline]
Medium — alert fatigue or missed alerts
P2
[1 day]
[Name]
[Date]
[e.g. No latency heatmap — outliers invisible in averages]
Low — harder to spot tail latency issues
P3
[2 hours]
[Name]
[Date]
Total observability debt: [N] items | Estimated effort: [N days]
Quality Checks
Every alert has a named on-call action — no alert says "investigate" without specifying what to investigate first
Alert thresholds are calibrated against production baselines, not set to default values from a template
Structured logging is implemented — no unstructured text log lines in production
PII is explicitly excluded from logs — a named engineer has verified this
Distributed tracing is propagating trace IDs across all service boundaries (verify with a test request)
The primary dashboard answers "is the service healthy?" in under 10 seconds — no hunting for the right panel
Business metrics are tracked alongside infrastructure metrics — not just four golden signals
Observability debt items have owners and dates — not just "would be nice to have"
Anti-Patterns
Do not create alerts without a specific on-call action — an alert that just says "investigate" trains engineers to ignore it
Do not set alert thresholds from a template without calibrating against production baselines — uncalibrated thresholds cause either alert fatigue or missed incidents
Do not log PII, tokens, or secrets — a logging standard is incomplete without an explicit list of what must never be logged
Do not measure only the four golden signals without adding at least one business metric alert — infrastructure health can be green while the business-critical path is silently failing
Do not deploy distributed tracing without verifying that trace IDs propagate across all service boundaries — partial tracing is worse than no tracing because it produces misleading incomplete traces
