What is Robots.txt?
Robots.txt is a plain text file at the root of your domain that tells crawlers which parts of the site they may request. Every well-behaved bot checks it first. It controls access, not indexing, which is the distinction most teams miss, because a blocked URL can still surface in search results if other pages link to it.
Robots Exclusion Protocol File
How robots.txt works
The file has to sit at yoursite.com/robots.txt. Nowhere else. Crawlers request it before anything else, read the rules, and then follow only the group matching their own user agent name.
The syntax stays simple, with User-agent naming the bot, Disallow blocking a path prefix, and Allow carving a narrower exception out of a broader block, plus the wildcards and dollar-sign endings that Google supports. List your XML sitemap location too, since it helps crawlers that arrived without one.
Order matters less than specificity, because Google applies whichever matching rule is most specific, which means an Allow on a deeper path will beat a broader Disallow sitting above it in the file.
The mistakes that cost traffic
The worst one is a single line. Disallow slash, applied to everything, pushed live from staging during a launch, which happens far more often than anyone admits and can go unnoticed until the traffic graph collapses. Check it on day one of every migration.
Second, using robots.txt to hide a page from search, which backfires because blocking the crawl means Google never reads the noindex tag sitting on that page and the URL can linger in results indefinitely.
Third, blocking CSS and JavaScript directories. Google renders pages now, so block the assets and it sees a broken layout. Never put sensitive paths here either. The file is public.
What this looks like in practice
A typical setup for an e-commerce client blocks the cart, checkout, account and internal search paths, blocks the filter parameters generating duplicate listings, and points at a sitemap index. Nothing else. Short files stay correct.
For a real estate portal in Gurugram, add a block on the agent login area and on print view URLs while leaving project pages and location landing pages fully open, then test every change in the robots.txt tester before deploying and confirm in Search Console that key URLs remain fetchable. Two minutes of checking saves a fortnight of lost visibility.
Related terms: RTO (Return to Origin) · Schema Markup · SERP Features.
Where this shows up in the work: SEO & AI Search · Full glossary.
Robots.txt — questions, answered.
Not strictly, since a site with nothing to block works fine without one and crawlers treat a missing file as permission to crawl everything. Having one is still sensible because it lets you declare your sitemap and add rules later.
No, because it stops the crawl rather than the listing, so if external sites link to a blocked URL then Google may still show it with no description. Allow crawling and add a noindex meta tag instead.
Last updated 2026-08-08
Ready to replace guesswork with a growth engine?
Book a 30-minute strategy call. We’ll show you exactly where your funnel is leaking, before you spend a dollar.