The Ideal robots.txt for WordPress (Copy-Paste Examples)
Most WordPress sites need almost nothing in robots.txt. WordPress generates a virtual one automatically, and for a plain blog or brochure site it is close to adequate. The reason to touch it at all is usually to add your sitemap line and to stop crawlers wasting time on a couple of internal paths. The bigger risk is not an incomplete robots.txt; it is a robots.txt that blocks things it should never block and quietly damages how Google renders your pages.
This is a copy-paste reference: a sensible starter file, the lines that do more harm than good, and variants for WooCommerce and staging. If you want the conceptual background on why robots.txt does not remove pages from search (a separate and commonly confused topic), read robots.txt vs noindex alongside this.
WordPress already has a virtual robots.txt
If you have never created a physical robots.txt file, WordPress serves a generated one anyway. Visit yoursite.com/robots.txt and you will typically see something like:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
That default is reasonable. It keeps crawlers out of the admin area while allowing admin-ajax.php, which some themes and plugins use to load front-end content. The moment you create a real robots.txt file in your site root, or a plugin writes one, that virtual file is replaced by yours. So editing robots.txt in WordPress means either overriding the virtual file with a physical one or using an SEO plugin's editor.
A sensible starter file
For a typical WordPress site, this is a solid baseline. It keeps the admin protections, blocks a couple of genuinely useless-to-crawl paths, and points crawlers at your sitemap.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/sitemap.xml
The /?s= and /search/ lines stop Google crawling your internal search results, which are low-value, near-infinite URL permutations that eat crawl budget and add nothing to the index. Swap example.com for your domain and point the Sitemap line at your real sitemap URL. If you use Yoast or Rank Math, that is usually /sitemap_index.xml rather than /sitemap.xml; check what your plugin generates. Our XML sitemap generator will produce or validate one, and the XML sitemap SEO guide covers what belongs in it.
What you must not block
This is the part that matters more than the starter file, because these mistakes actively hurt sites and they are still passed around in old tutorials.
Do not block /wp-content/. Old guides tell you to Disallow: /wp-content/ to "hide WordPress internals." That directory holds your themes, images, CSS, and JavaScript. Google renders pages the way a browser does, and if you block the CSS and JS it needs, Google sees a broken, unstyled version of your page and may judge it as lower quality or not mobile-friendly. Leave /wp-content/ fully crawlable.
Do not block /wp-includes/ for the same reason; some front-end scripts live there. Do not block your uploads directory if you want your images to appear in Google Images. And do not block CSS or JS files anywhere. Google has said plainly for years that it needs to fetch page resources to render and understand a page. A robots.txt that blocks resources is one of the more common self-inflicted technical SEO wounds we see.
A short table of the usual bad advice and the correction:
| Old advice | Reality |
| --- | --- |
| Disallow: /wp-content/ | Blocks CSS/JS/images Google needs to render; never do this |
| Disallow: /wp-includes/ | Can block front-end scripts; leave it open |
| Disallow: /wp-content/uploads/ | Removes your images from Google Images |
| Block CSS and JS to "save crawl budget" | Breaks rendering; Google explicitly warns against it |
The WooCommerce variant
WooCommerce adds product filtering and cart URLs that generate a lot of low-value crawlable permutations. A reasonable addition on top of the starter file:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?filter_
Sitemap: https://example.com/sitemap_index.xml
The cart, checkout, and my-account pages have no business in search results, and the ?orderby= and ?filter_ patterns catch the faceted-navigation URLs that otherwise multiply endlessly. Adjust the parameter names to match what your store actually generates; look at your own URLs before copying these, because filter parameter names vary by theme and plugin. Blocking crawl on faceted URLs is a crawl-budget decision, not an indexing one; if a specific product-variant URL also needs to stay out of the index, that is a noindex job, per robots.txt vs noindex.
The staging variant
Staging and development copies should be blocked from all crawling, but be careful: blocking in robots.txt does not guarantee a staging URL stays out of the index if it gets linked. For a staging environment, the reliable move is HTTP authentication (a login wall) on the whole site. If you also want the robots.txt belt-and-braces, this is the disallow-everything file:
User-agent: *
Disallow: /
Two lines, and it tells every compliant crawler to stay off the entire site. The failure mode to avoid at all costs is the opposite one: pushing this file to production by accident during a deploy, which deindexes your live site. If your staging and production share a deployment pipeline, make sure robots.txt is environment-specific and not copied verbatim from staging to live.
How to edit robots.txt in WordPress
You have two practical routes. Most SEO plugins include a robots.txt editor: in Yoast it is under Yoast SEO > Tools > File editor, and in Rank Math under Rank Math > General Settings > Edit robots.txt. Editing there overrides the virtual file without you touching the server. This is the easiest option for most people.
The other route is a physical file. Create a plain-text file named robots.txt, put it in your site root (the same directory as wp-config.php), and upload it via SFTP or your host's file manager. A physical file takes precedence over the WordPress virtual one and over most plugin output, so if you go this way, do not also try to manage the file through a plugin or you will confuse yourself about which one is live.
Whichever route you take, build and check the file with our robots.txt generator first, then confirm the live result by loading yoursite.com/robots.txt in a browser and, in Search Console, watching that no important pages get reported as blocked.
Frequently Asked Questions
Does WordPress need a robots.txt file?
Not strictly. WordPress serves a virtual robots.txt automatically that is adequate for many sites. You would create your own mainly to add a Sitemap line and to block low-value paths like internal search. The bigger reason to review it is to make sure nothing important, like CSS or JS, is being blocked.
Should I block /wp-content/ in robots.txt?
No. The /wp-content/ directory holds your themes, images, CSS, and JavaScript, and Google needs those resources to render your pages correctly. Blocking it makes Google see a broken, unstyled page and can lower how it assesses your site. Leave /wp-content/ fully crawlable.
Where do I edit robots.txt in WordPress?
Two options. Use an SEO plugin's editor (Yoast SEO > Tools > File editor, or Rank Math > General Settings > Edit robots.txt), which overrides the virtual file. Or upload a physical robots.txt file to your site root via SFTP, which takes precedence over the virtual one. Do not manage it both ways at once.
What sitemap URL do I add for Yoast or Rank Math?
Both Yoast and Rank Math generate a sitemap index at /sitemap_index.xml by default, so your line is usually Sitemap: https://example.com/sitemap_index.xml. Confirm by loading the URL in a browser, since custom setups can differ, then add exactly the URL that works.
How do I keep a staging site out of Google?
The reliable method is HTTP authentication (a login wall) on the entire staging environment, because a robots.txt block alone does not guarantee a linked staging URL stays out of the index. If you also add Disallow: / in staging robots.txt, make sure that file is environment-specific and never deploys to production.