Monitoring set up in the wrong order produces a hundred dashboards nobody reads and an alert channel everyone mutes. This order produces something that works, one week at a time.
Before any metrics stack, put an external uptime check on your most important URL at a one to five minute interval. External matters: a monitor running on the same host tells you nothing when the host dies. Several providers offer 50 monitors free, which covers a small fleet. Check the endpoint that proves the system works — a health path that touches the database, not a static file. Add a TLS certificate expiry alert at 21 days in the same pass. By the end of week one you should learn about an outage from a notification rather than from a user.
Latency, traffic, errors and saturation. Concretely: request duration as a histogram so you can read p50, p95 and p99; requests per second by endpoint and status; error rate as a proportion of total rather than a raw count; and saturation as CPU, memory, disk and — the one most often forgotten — database connection pool utilisation. A Prometheus client library plus a scrape endpoint is roughly twenty lines in most frameworks. Use histograms rather than averages: an average latency of 200 ms hides a p99 of four seconds, and the p99 is what people complain about.
An alert that does not require a human to act is not an alert, it is a dashboard. Apply four rules from the start:
for clause. A metric crossing a line for fifteen seconds is noise.Aim for fewer than one page per engineer per week. Above that, people stop reading them and the system has failed regardless of its technical quality.
Build exactly two. A service overview — traffic, error rate, p95 latency, saturation, all on one screen — so anyone can answer "is it healthy" in five seconds. And a debugging dashboard with the breakdowns you reach for during an incident: latency by endpoint, errors by type, dependency call durations. Anything more gets stale. Then start a weekly fifteen-minute review: which alerts fired, which were actionable, which thresholds need moving. That review is what stops the system decaying.
Distributed tracing before you have a service boundary that needs it. Log-based alerting, which is expensive and slow compared with metric-based alerting. Anomaly detection, which needs a baseline you do not have yet. Synthetic transaction monitoring of complex user journeys — valuable later, high maintenance early. And custom business metrics before the technical ones are trustworthy.
Self-hosted Prometheus plus Grafana plus Alertmanager is free in licence terms and costs you storage, upgrades and a scrape configuration to maintain. Hosted options remove that work and bill by metric series or ingested volume, where the trap is cardinality — a label containing a user ID or request ID multiplies series count and cost without bound. Whichever you choose, set a cardinality budget and a spend alert in week one, because both stacks fail the same way: silently, expensively, and only visible at the end of the month.