Does the data you want exist in the HTML the server sends, or is it assembled by JavaScript after load? Fetch the page with curl and search the raw response for a value you can see on screen. If it is there, you never need a browser, and skipping the browser is worth roughly two orders of magnitude. An HTTP request costs a few milliseconds and a few kilobytes of memory. A headless Chromium tab costs a few hundred milliseconds to seconds and typically 100 to 300 MB of RSS, so a box that could run thousands of concurrent HTTP fetches runs maybe a dozen browsers.
Before writing selectors, check for structured data. Most modern sites hand you clean JSON: a <script type="application/ld+json"> block, a Next.js __NEXT_DATA__ payload, a Nuxt __NUXT__ object, or an internal JSON endpoint visible in the network tab. Parsing those is stable across redesigns. CSS selectors against hashed class names break the week the site ships a new build.
| Tool | Runs a browser | Best at | Weakness |
|---|---|---|---|
| requests + BeautifulSoup | No | Small jobs, static HTML | Obvious TLS fingerprint, no concurrency of its own |
| Scrapy | No | Large crawls, retries, pipelines | Framework conventions, async model to learn |
| curl-impersonate | No | Matching a real browser's TLS handshake | Still no JavaScript execution |
| Playwright | Yes | Auto-waiting, request interception, three engines | Memory and startup cost |
| Puppeteer | Yes | Chrome-only work, PDF rendering | Manual waiting, narrower engine support |
| Selenium | Yes | Legacy grids, exotic browser matrices | Slower, driver version churn |
Blocking has moved below the application layer. Anti-bot services fingerprint the TLS ClientHello, commonly summarised as a JA3 or JA4 hash: cipher order, extension order, supported groups, ALPN. Python's requests on OpenSSL produces a handshake no real Chrome ever produces, so copying the User-Agent header changes nothing. HTTP/2 frame settings and header ordering get checked too.
The fixes in order of effort: use curl-impersonate or a client built on it so the handshake matches a shipped browser build, move to a real browser, or pay a hosted unblocker.
Scrapy defaults to a small number of concurrent requests per domain and ships AUTOTHROTTLE_ENABLED, which measures response latency and backs off when the server slows down. That is the correct behaviour whatever stack you use: treat rising latency and 429s as a signal to reduce concurrency, not as something to retry harder. Set DOWNLOAD_DELAY, honour Retry-After, and cap yourself well below what the site can serve.
Enable the HTTP cache while you develop. Writing a parser normally takes twenty iterations, and re-fetching the same 500 pages twenty times is both slow and rude. Save raw HTML to disk or S3 on first fetch, keyed by URL, and parse from the cache. It also means that when a parse silently produces nulls next month you still have the original bytes to diff.
Datacenter proxies are cheap and widely blocked. Residential and mobile proxies are the ones that work on hard targets, and they are almost universally billed per gigabyte rather than per request, which changes how you write code. A headless browser loading a full page pulls images, fonts, analytics and ad scripts, and can easily move a megabyte or two per page view through the metered proxy. Blocking non-essential resources through request interception, or dropping to plain HTTP for pages that do not need JavaScript, is usually the single biggest cost lever in a scraping budget.
Scraping publicly accessible pages is not automatically unlawful, and US courts have been reluctant to treat access to public data as a Computer Fraud and Abuse Act violation. That is narrow, not a licence. Anything behind a login involves a contract you accepted, so terms of service claims apply. Personal data pulls you into GDPR regardless of whether the page was public. Copyright still applies to what you republish.
Practically: read robots.txt and honour it unless you have a specific reason not to, identify your crawler in the User-Agent with a contact address, rate limit so you are never the reason a page gets slow, take the smallest slice of data that answers your question, and check whether an official API or bulk export exists before you build any of this.