robots.txt is four lines of syntax with an unusual amount of room for expensive mistakes, because a wrong character can remove a site from search results and nothing will tell you for days. Here is the format as crawlers actually implement it.
User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/public-help.html
Crawl-delay: 5
User-agent: Googlebot
Disallow: /internal/
Sitemap: https://example.com/sitemap.xmlIt must live at the root — https://example.com/robots.txt — served as text/plain. A file at /blog/robots.txt is ignored. Subdomains need their own file; example.com and shop.example.com are separate origins as far as robots is concerned. Rules are case-sensitive on the path, so /Admin/ and /admin/ are different.
This is the rule most people get wrong. A crawler finds the single most specific User-agent group that matches it and obeys only that group. If you have a Googlebot block and a * block, Googlebot reads its own block and completely ignores the wildcard one — including any Disallow lines you assumed were global. If a rule must apply to everyone, it has to be repeated in every group.
* matches any sequence of characters and $ anchors the end of the URL. Both are supported by Google, Bing and most serious crawlers, though they are not in the original 1994 standard.
Disallow: /*? # any URL with a query string
Disallow: /*.pdf$ # PDFs only, not /file.pdf.html
Disallow: /*sessionid= # a specific tracking parameter anywhereWhen both an Allow and a Disallow match, the more specific rule (longer path pattern) wins; on an exact tie, Allow wins. That is what makes the "block a directory, permit one file inside it" pattern work.
A blocked URL can still appear in search results, listed without a description, if other sites link to it. Blocking actually prevents deindexing, because the crawler can no longer fetch the page to see your noindex tag. To remove a page from the index: allow crawling and serve <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header. Google dropped support for a Noindex: directive inside robots.txt in 2019; it does nothing.
Search engines render pages. Blocking /assets/, /static/ or a framework's build directory means the renderer sees an unstyled, non-functional page and judges it accordingly. Block real routes — admin panels, internal search results, cart and checkout, tracking-parameter URLs — not the files needed to draw the page.
The major AI crawlers publish user-agent strings and honour robots.txt: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended (which controls Gemini training without affecting Search), Applebot-Extended, and CCBot for Common Crawl. Decide deliberately: blocking training crawlers protects content, and blocking the search-oriented ones such as OAI-SearchBot also removes you from AI answers that would have cited you. Those are different trades and deserve different rules.
Use the robots.txt report in Search Console to see the file Google last fetched and check specific URLs against it. Locally, curl -sS https://example.com/robots.txt confirms it returns 200 as plain text rather than your framework's HTML 404 page — a surprisingly common failure on SPAs with catch-all routing. Python's standard library gives you a quick offline check:
python -c "
import urllib.robotparser as r
p=r.RobotFileParser(); p.set_url('https://example.com/robots.txt'); p.read()
print(p.can_fetch('Googlebot','https://example.com/admin/'))"Disallow: / in the wildcard group blocks the entire site, and it reaches production most often by a staging robots.txt being deployed with the release. Put an assertion in your smoke tests: fetch robots.txt after deploy and fail the pipeline if it disallows the site root.