A profiler answers one question: where did the time go. The reason most teams never get an answer is that they reach for the wrong kind of profiler — an instrumenting one that changes the program's timing, or a sampling one they cannot run in production. Here is which tool to reach for, and the flags that matter.
An instrumenting profiler wraps every function call and can slow a program by 10× or more, which distorts exactly the ratios you are trying to measure. A sampling profiler interrupts the process at a fixed frequency — commonly 99 or 999 Hz — and records the stack. Overhead lands in the low single-digit percent, which means you can run it against live traffic. Use 99 Hz rather than a round 100 so your sampling does not phase-lock with periodic timers in the application.
perf record -F 99 -g -p $(pgrep -f myservice) -- sleep 30
perf script > out.perf
# then fold and render with Brendan Gregg's FlameGraph scripts
perf report --stdio --sort=dso,symbolCompile with -fno-omit-frame-pointer or the -g stacks will be truncated garbage. On modern kernels --call-graph dwarf works without frame pointers but produces much larger captures. perf stat -d is the underused sibling: it gives you IPC, cache misses and branch misses in one line, which tells you whether you have an algorithmic problem or a memory-layout problem before you read a single stack.
py-spy top --pid 1234 attaches to a running CPython process and shows a live top-style view of hot functions. py-spy record -o profile.svg --pid 1234 --duration 60 writes a flame graph. Neither requires importing anything into the target, which is why it works on a production container you cannot redeploy. Add --subprocesses for Gunicorn workers and --native when the time is inside a C extension such as NumPy or a database driver. For the "it is not CPU, it is waiting" case, --idle includes blocked threads.
Import net/http/pprof, expose it on an internal-only port, and collect with go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30. The heap profile is usually more valuable than the CPU profile in Go services: -http with /debug/pprof/heap and the -sample_index=alloc_space flag shows allocation churn, and reducing allocations is the most reliable way to cut GC time. go tool pprof -diff_base old.pb.gz new.pb.gz compares two captures directly, which is how you prove an optimisation worked.
async-profiler avoids the safepoint bias that makes many Java profilers blame the wrong method. Run it with -e cpu, -e alloc or -e lock and output -o html for an interactive flame graph. Lock profiling is the one people forget; a service that looks idle on CPU while p99 latency climbs is usually contending on a monitor, and -e lock names it directly.
Chrome DevTools Performance panel plus the Performance API covers the front end. Watch Interaction to Next Paint rather than the retired First Input Delay: the Core Web Vitals threshold is 200 ms at the 75th percentile. Long tasks over 50 ms are what push INP over budget, and the panel flags them with a red triangle. performance.measure() calls show up as user timings in the same flame chart, so annotating your own hydration or render phases costs three lines and makes the trace readable.
Local profiles mislead because your laptop has a warm cache, no network latency, one user and different CPU frequency scaling. Continuous profilers — Parca, Pyroscope, Datadog and Google Cloud Profiler all do this — sample every instance at around 100 Hz permanently and let you compare a regression against last week. That comparison is what makes profiling routine instead of an emergency exercise.