Robots.txt Guide: How to Control Search Engine Crawling
Here is the single most misunderstood fact about robots.txt, and it trips up experienced developers: robots.txt controls crawling, not indexing. Blocking a URL with Disallow: tells Googlebot "don't fetch this page." It does not tell Google "don't show this page in search results." If other sites link to a URL you've disallowed, Google can still index it, listing it in results with the dreaded snippet "No information is available for this page." It knows the URL exists; it just isn't allowed to read what's on it.
If your actual goal is to keep a page out of search, robots.txt is the wrong tool. You need a noindex directive, and for noindex to work, the page must be crawlable, which means you must not disallow it. The two mechanisms pull in opposite directions, and confusing them is how pages end up indexed with no description, or hidden when you wanted them found.
What the file actually is
robots.txt is a plain-text file that lives at exactly one place: the root of your host, at https://example.com/robots.txt. It can't live in a subfolder. Each host and subdomain needs its own; blog.example.com does not inherit example.com's rules.
Compliant crawlers (Googlebot, Bingbot, and most reputable bots) fetch it before crawling and obey it. It is a voluntary protocol, not a firewall. A malicious scraper ignores it entirely, which leads to the second rule people get wrong: never use robots.txt to "hide" anything sensitive. The file is public (anyone can read yoursite.com/robots.txt), so a Disallow: /admin-secret-panel/ line is a signpost pointing straight at the thing you wanted hidden. Sensitive paths belong behind authentication, full stop.
The syntax, in the order it matters
A robots.txt is a list of groups. Each group starts with one or more User-agent lines naming the bots it applies to, followed by the rules:
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Sitemap: https://example.com/sitemap.xml
User-agent: *applies to every bot. You can target a specific one withUser-agent: Googlebot.Disallow: /path/blocks crawling of anything starting with that path.Disallow:(empty) blocks nothing.Allow: /pathcarves an exception out of a broaderDisallow. Google resolves conflicts by the most specific (longest) matching rule, not by line order, so theAllow: /search/helpabove wins for that one path even though/searchis disallowed.Sitemap:points crawlers to your XML sitemap. Unlike the other directives it's absolute (full URL) and global, not tied to any user-agent group. Always include it; it's free discovery help. (See our XML sitemap guide for what to put in that file.)
A trailing slash is meaningful. Disallow: /admin blocks /admin, /administrator, and /admin/login. Disallow: /admin/ blocks only things inside the /admin/ directory. Get this wrong and you either over-block or leave a hole.
The mistakes that take a site down
These are the ones that cause real, measurable traffic loss, usually within days.
1. The launch-day full block that never got removed. Staging sites ship with:
User-agent: *
Disallow: /
That Disallow: / blocks the entire site. It's correct for staging. It is catastrophic if it survives the push to production, which it does constantly, because nobody remembers it's there. Pages start dropping out of the index within a week. Make checking this line the first item on every go-live checklist.
2. Blocking CSS and JavaScript. Years ago people blocked /css/ and /js/ to "save crawl budget." Modern Googlebot renders pages like a browser; if it can't fetch your stylesheets and scripts, it sees a broken, unstyled, possibly empty page and judges your content, and its mobile-friendliness, on that wreck. Never disallow your asset directories. If you've inherited an old robots.txt, this is the first thing to delete.
3. Using Disallow when you meant noindex. You want a thank-you page or a faceted filter URL kept out of search, so you disallow it. Now Google can't crawl it, can't see the noindex you also added, and may index the bare URL anyway from internal links. The fix: remove the Disallow, keep the page crawlable, and let the noindex meta tag (or X-Robots-Tag header) do its job.
4. Trying to remove an already-indexed page by disallowing it. This is the inverse trap. If a page is already in Google's index and you disallow it, you've frozen it there; Google can no longer crawl it to discover the noindex that would remove it. To deindex something, leave it crawlable with noindex until it drops out, then disallow it if you like.
Crawling vs. indexing, as a table
| | Disallow (robots.txt) | noindex (meta tag / header) |
| --- | --- | --- |
| What it controls | Crawling, whether the bot fetches the page | Indexing, whether the page appears in search |
| Page can still be indexed? | Yes, from external/internal links | No, it's removed from the index |
| Requires the page to be crawlable? | No (it blocks the crawl) | Yes, must be fetchable to be read |
| Right use | Steer crawl effort, block infinite URL spaces | Keep a real page out of search results |
When you need both, keeping it out of search and stopping wasted crawls, get it deindexed with noindex first, confirm it's gone, then add the Disallow.
Build it without guessing
Hand-writing these rules is where syntax errors creep in: a stray space, a missing slash, an Allow that doesn't actually override. Our Robots.txt Generator builds a valid file from your choices and includes the sitemap line. After you deploy, confirm it's reachable at yourdomain.com/robots.txt and test specific URLs in Google Search Console's robots.txt report, which shows you exactly which rule matches a given path.
Frequently Asked Questions
Does robots.txt remove a page from Google?
No. Disallow stops Googlebot from crawling the page, but the URL can still be indexed if it's linked from elsewhere, often showing up with no description. To remove a page from search, use a noindex meta tag or X-Robots-Tag header and keep the page crawlable so Google can see that directive.
Should I block CSS and JavaScript files?
No. Google renders pages using your CSS and JS to assess layout, content, and mobile-friendliness. Blocking those files makes Googlebot see a broken page and can hurt rankings. Leave your asset directories crawlable.
Is robots.txt a security feature?
No. The file is publicly readable at yoursite.com/robots.txt, so anything you list there is visible to everyone. Protect private content with authentication, not a Disallow rule.
Why does my disallowed page still appear in search results?
Because Disallow only blocks crawling. Google found the URL through links and indexed the address even though it couldn't read the content. Remove the Disallow, add noindex, and wait for it to drop out of the index.
Do I need a robots.txt file at all?
It's optional. With no file, crawlers assume everything is allowed. But adding one lets you point to your sitemap, steer crawl effort away from low-value URL spaces (like internal search results), and document your intent. Most sites benefit from a minimal, correct one.
What's the difference between Allow and Disallow precedence?
Google applies the most specific (longest path) matching rule, not top-to-bottom order. So Allow: /folder/page overrides Disallow: /folder/ for that one page, regardless of which line comes first.