robots.txt is about thirty lines of plain text that can remove a site from search results faster than any other single file. It is also widely misunderstood: most of the damage comes from people using it to hide pages that are already indexed, or from a staging file being deployed to production. This is the syntax, the semantics, and the three failure modes worth memorising.
The file is a series of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. A crawler reads only the single most specific group that names it, falling back to * if there is none, so rules in the wildcard group do not apply to a bot that has its own group.
User-agent: *
Disallow: /admin/
Disallow: /out/
Disallow: /*?sessionid=
Disallow: /*.pdf$
Allow: /admin/public-report.html
Sitemap: https://example.com/sitemap.xmlPath matching is case-sensitive and prefix-based. * matches any run of characters and $ anchors the end of the URL. When an Allow and a Disallow both match, the longer rule wins, and a tie resolves in favour of Allow. That is why the example above can carve one file out of a blocked directory. Sitemap is independent of groups, takes an absolute URL, and can appear multiple times.
This is the distinction that causes the most wasted effort. Disallow tells a crawler not to fetch a URL. It does not tell a search engine not to list it. If other pages link to a blocked URL, it can still appear in results, typically with no snippet and an anchor-text-derived title, because the engine knows the URL exists but was never allowed to read it.
To remove a page from the index you need <meta name="robots" content="noindex"> in the HTML, or an X-Robots-Tag: noindex response header for non-HTML files such as PDFs. Crucially, the page must remain crawlable for that instruction to be seen. Blocking a URL in robots.txt and adding noindex to it at the same time means the noindex is never read, and the page stays listed indefinitely. Deindex first, block later if you still want to.
User-agent: * followed by Disallow: / reaching production is the classic traffic-loss incident. Generate the file from an environment variable, never copy it between environments by hand./_next/, /static/, /assets/ or an image optimiser endpoint stops the renderer from seeing your CSS, JavaScript and images. The page is then evaluated as an unstyled skeleton, which affects both mobile-friendliness and image indexing. Block real routes such as /admin/ and affiliate redirects, not build output.https://www.example.com/robots.txt does not cover the apex domain, a subdomain, or the http version. Each origin needs its own, and it must return 200 with text/plain.Google stops reading after roughly 500 KB, which only matters if you have generated thousands of rules; consolidate with wildcards instead.
The named agents worth knowing are GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot along with its user-triggered and search counterparts from Anthropic; PerplexityBot; Applebot-Extended; CCBot for Common Crawl; and Bytespider. Google-Extended is different again: it is a robots.txt token rather than a crawler, and it controls whether content already fetched by Googlebot may be used for Gemini training and grounding, without affecting Search ranking.
The decision is not one decision. Training crawlers and answer-engine crawlers are separate user-agents, and blocking the second is what costs you citations and referral traffic in AI answers. A common middle position is to allow the search and user-triggered agents and block the bulk training ones. Whatever you choose, remember robots.txt is voluntary. Agents that ignore it are stopped at the WAF or by rate limiting, not by a text file.
Fetch the deployed file and check the status, content type and body in one command: curl -sSI https://example.com/robots.txt followed by the body. Then use the robots.txt report in Search Console to confirm the parsed rules match your intent, and test individual URLs that must stay crawlable, particularly your CSS bundle, one product page and your sitemap URL. If the file is generated by code, write a unit test asserting that a handful of important paths are allowed; it is a ten-line test that prevents a very expensive mistake. Keep the sitemap reference accurate too, since a stale one wastes crawl budget; the sitemap generation guide covers that side.