Log tooling divides into three jobs: getting logs off the machine, storing them affordably, and querying them when something is on fire. Most stacks are strong at one and weak at the others.
No log platform can recover information you never emitted. Emit JSON with a fixed set of fields on every line — timestamp in UTC with an explicit offset, level, service, version, trace ID, and the message — and put variable data in fields rather than interpolating it into the message string. The difference is concrete: {"level":"error","order_id":"A123","msg":"payment failed"} can be aggregated by order; "payment failed for order A123" can only be grepped. Every mature logging library does this: structlog in Python, zap or slog in Go, pino in Node, tracing in Rust.
Loki indexes only labels, not log content, which is why it stores far more cheaply than a full-text engine. Queries filter by label first and then scan: {app="api", level="error"} |= "timeout" | json | duration > 5s. It is excellent when you know which service and time window you care about, and slow when you want to search all logs everywhere for a string. The failure mode to avoid is high-cardinality labels — putting a user ID or request ID in a label creates millions of streams and destroys performance. Put those in the log body instead.
Full-text indexing means any query is fast, including the ones you did not plan for. The cost is operational: index lifecycle management, shard sizing, and a memory footprint that makes a small cluster a real expense. Storage is typically several times what Loki uses for the same logs. Worth it when investigation is your dominant use case and someone will own the cluster; heavy otherwise.
Whatever the backend, put a collector between the application and the store. Vector is a single Rust binary that tails files or receives over HTTP, transforms with a small language, and fans out to multiple sinks. That indirection is what lets you change backends without touching application code, and it is where you drop noisy log lines, redact fields and sample. Fluent Bit fills the same role and is common on Kubernetes. OpenTelemetry Collector is the option to pick if you want logs, metrics and traces flowing through one pipeline.
If log volume is genuinely large, a columnar database beats both approaches on cost and query speed. Logs are append-only, timestamp-ordered and highly compressible — exactly ClickHouse's shape, with compression ratios that make retention affordable. The trade is that you are building a schema and a query interface rather than installing a product. Several hosted log products are ClickHouse underneath for precisely this reason.
Datadog, Better Stack, Axiom and similar remove all operational work and bill by ingested volume and retention. The trap is that ingestion is what you pay for, and a single misconfigured debug logger can multiply the bill overnight. Whatever you choose, set a volume alert on ingestion itself, not just on spend, so you find out within hours rather than at the end of the month.