Automate Your Logging — Step by Step

Logging becomes useful the moment you stop writing sentences for humans and start emitting events for machines. Here is the sequence that gets you from print statements to something you can query, alert on, and afford.

Step 1: emit structured events, one per line

Replace free-text messages with JSON objects: a timestamp, a level, an event name, and fields. {"ts":"2026-04-01T10:00:00Z","level":"warn","event":"payment_retry","order_id":"A-771","attempt":2} is greppable, parseable and aggregatable; "Retrying payment for order A-771 (attempt 2)" is none of those. In Python, structlog or python-json-logger do this with a handler change. In Node, pino is the default choice and is fast enough to leave on in production.

Step 2: use levels as a policy, not a mood

ERROR means a human needs to act. WARN means a degraded path was taken and it happened often enough to notice. INFO is the audit trail of what the system did. DEBUG is for reproducing a specific problem and should be switchable per module at runtime rather than requiring a deploy. If everything is INFO, you have one level and a large bill.

Step 3: add a correlation ID at the edge

Generate a request ID in your first middleware, put it in a context variable (contextvars in Python, AsyncLocalStorage in Node), and attach it to every log line automatically. When someone reports an error, one filter shows you every line from that request across every service. If you run OpenTelemetry, use the existing trace_id and span_id instead so logs and traces line up in the same view.

Step 4: automate rotation before the disk fills

On a VM, drop a file in /etc/logrotate.d/: daily, rotate 14, compress, delaycompress, missingok, notifempty. Prefer copytruncate only if your app cannot reopen its file on SIGHUP — it can lose lines written during the copy. Under systemd, log to stdout and let journald handle it, then read with journalctl -u myapp -f --since "10 min ago". In containers, log to stdout and let the runtime deal with files; writing log files inside a container is a way to lose them.

Step 5: ship and sample

A collector — Vector, Fluent Bit or the OpenTelemetry Collector — reads from files or stdout, parses, enriches with host and service metadata, and forwards. Do the sampling here, not in the application: keep 100% of ERROR and WARN, keep 100% of anything carrying an active trace, and sample high-volume INFO from healthy requests down to a few percent. That one rule typically removes most of the volume without removing anything you would have looked at.

Step 6: redact before it leaves the process

Maintain a deny-list of field names — password, token, authorization, card_number, ssn — and a processor that drops or masks them in the logging pipeline itself. pino has redact paths built in; structlog does it with a processor. Redacting at the aggregator is too late: the secret already crossed a network and sat in a file.

Step 7: alert on patterns, not on lines

A single ERROR is not an incident. Alert on rate and ratio: error events per minute above a threshold, or the proportion of requests logging a failure exceeding a percentage over a five-minute window. Add one alert for the absence of logs — a service that has stopped emitting anything is usually a worse problem than one emitting errors.