Robots.txt: how to block all or some crawlers

To block all crawlers, put User-agent: * and Disallow: / in a robots.txt file at the root of your host. To block one bot, name its token in its own group. To block a folder, use Disallow: /folder/. Robots.txt is only a request: polite crawlers obey it, bad bots ignore it, and blocked URLs can still appear in Google.

Copy-ready robots.txt rules

Each block below is a complete file or a group you can paste into one. A group starts with one or more User-agent lines and lists the rules for those crawlers. Paths are case-sensitive, so /Private/ and /private/ are different folders.

Block all crawlers from the whole site:

User-agent: *
Disallow: /

This is what you want on a staging or test copy. On a live site it tells every compliant crawler, Googlebot and Bingbot included, to stop crawling everything. Read the section on the staging accident below before you use it anywhere public.

Block everything except Google and Bing:

User-agent: Googlebot
User-agent: bingbot
Allow: /

User-agent: *
Disallow: /

This works because a crawler follows only one group: the one with the most specific user agent that matches it. Googlebot finds its own name and ignores the * group. Every crawler without a named group falls back to * and is blocked. You can list several User-agent lines on top of one group, as shown.

Block one bot, allow everyone else:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

No * group means no rules for anyone else. GPTBot is OpenAI's crawler for model training data, and CCBot is Common Crawl's crawler. The same pattern works for any token, such as ClaudeBot from Anthropic. Find the exact tokens in our list of bot user agents, and read AI crawlers: block or allow? before you shut out bots that also feed AI search answers.

Block one folder for all crawlers:

User-agent: *
Disallow: /admin/
Disallow: /cart/

Block a folder but allow one subfolder inside it:

User-agent: *
Disallow: /private/
Allow: /private/press-kit/

The order of the lines does not decide this. Google and the standard (RFC 9309) both pick the most specific matching rule, measured by the length of the path. /private/press-kit/ is longer than /private/, so files in the press kit stay crawlable. When an Allow and a Disallow rule are equally specific, Google uses the less restrictive one, which is Allow.

Google also understands two wildcards: * for any sequence of characters and $ for the end of the URL. Disallow: /*.pdf$ blocks every URL that ends in .pdf.

How Google picks which group to follow

Group matching is where most hand-written files go wrong. The rules Google documents:

• One group per crawler. A crawler that finds a group with its own name ignores the * group completely. If you give Googlebot a group with one Disallow line, none of your general rules apply to it anymore.
• Groups for the same crawler are merged. Two separate User-agent: Googlebot groups are combined into one. The * group is never merged with a named one.
• Some Google crawlers ignore *. AdsBot-Google, AdsBot-Google-Mobile and Mediapartners-Google skip the global group. A "block all" file does not stop them. That is good news if you run Google Ads, because the ad landing page checks keep working. To block them, you must name them.
• Only four fields count for Google: user-agent, allow, disallow and sitemap. Google ignores crawl-delay. Some other crawlers, such as ClaudeBot, say they honor it.

Blocking Google-Extended is a special case. It is not a separate crawler, just a token that tells Google not to use your content for training Gemini models and for grounding in some AI products. Google states it does not affect inclusion or ranking in Google Search.

What robots.txt does not do

Robots.txt is a sign on the door, not a lock. The standard says it outright: these rules are not a form of access authorization. Google's own docs say robots.txt cannot enforce crawler behavior.

• Bad bots ignore it. Scrapers, credential stuffers and fake crawlers never read the file, or read it to find the folders you wanted hidden. Never list secret paths in robots.txt.
• User-triggered fetchers may skip it. OpenAI says its ChatGPT-User agent acts on a person's request, so robots.txt rules may not apply to it.
• Disallow does not remove a page from Google. A blocked URL can still be indexed if other pages link to it. It shows up without a description, and Search Console reports it as "Indexed, though blocked by robots.txt".
• It does not cut server load from abusive traffic. A bot that ignores the file keeps hitting your server at the same rate.

To keep a page out of Google, add a noindex meta tag or header and leave the page crawlable. If robots.txt blocks the page, Googlebot never sees the noindex. For private content, use a password. To actually stop a bot, block it at the server or firewall, for example by blocking its IP address, and keep the real Googlebot safe by checking crawler IPs with reverse DNS or against Google's published IP ranges, never by user agent alone.

The classic accident: Disallow: / goes live

A developer builds the new site on a staging host with Disallow: / so it does not get crawled. Launch day comes, the files are copied to production, and the blocking file goes with them. Googlebot obeys and stops crawling. New pages are not picked up, and changes to existing pages go unseen.

How to spot it fast:

1. Open https://yourdomain.com/robots.txt in a browser right after every launch or migration.
2. In Search Console, open the robots.txt report (under Settings). It shows the files Google found for your top hosts, when each was last fetched, and any parse errors. After fixing the file, you can use "Request a recrawl" from the same report.
3. Run a key URL through URL Inspection and look at "Crawl allowed?". "No" means a robots.txt rule blocks it.
4. Watch the Page indexing report for "URL blocked by robots.txt" growing on pages you want in search.

A safer habit: keep staging behind a password instead of relying on robots.txt. Then there is no blocking file to copy by mistake.

Where the file lives and how Google reads it

• Location: /robots.txt in the top-level directory, all lowercase. A file in a subfolder is ignored.
• One file per host, protocol and port. https://example.com/robots.txt does not cover https://www.example.com/, http://example.com/ or https://shop.example.com/. Each subdomain needs its own.
• Size: Google reads the first 500 KiB and ignores the rest.
• Caching: Google generally caches the file for up to 24 hours, longer if it cannot refresh it. Changes are not instant.
• Redirects: Google follows at least five redirect hops, then treats the file as missing.

The status code your server returns matters as much as the content:

• 200: the rules are applied.
• 4xx (except 429): Google treats it as if no robots.txt exists and crawls without restrictions. A missing file is safe. 429 (too many requests) is the exception and does not count as "no file".
• 5xx: for the first 12 hours Google stops crawling the site and keeps retrying the file. Then for up to 30 days it uses the last good cached copy. After that, if the site is otherwise reachable, it crawls as if there were no file.

This is where bot protection can hurt SEO without anyone noticing. A firewall rule, rate limit or challenge that returns 403 or 503 to Googlebot on /robots.txt changes how Google crawls the whole site. If you add bot rules on your server or CDN, check the robots.txt report afterwards. For crawlers that waste server time on junk URLs, see crawl budget and bots.

When robots.txt is the wrong tool

Use robots.txt to steer crawlers that follow rules: keep them out of carts, filters and internal search, or opt out of AI training. If the problem is load, scraping or fake Googlebots, the bots doing the damage are the ones that never read it. That needs rules based on the real source of the traffic, written so that verified search and ad crawlers stay allowed. A free bot traffic report shows which networks, user agents and hours cause the load, so you can tell which bots robots.txt will stop and which need a firewall rule.

FAQ

How do I block all bots with robots.txt?

Use User-agent: * followed by Disallow: /. Compliant crawlers stop crawling, but bots that ignore robots.txt are not affected, and Google's AdsBot crawlers ignore the * group by design.

Will Disallow remove my page from Google?

No. A blocked page can stay indexed without a description if other pages link to it. Allow crawling and add noindex instead, then the page drops out after Google recrawls it.

How long does a robots.txt change take to work in Google?

Google generally caches the file for up to 24 hours. After an urgent fix, use "Request a recrawl" in the Search Console robots.txt report.

Does robots.txt block Google from crawling a folder on my subdomain?

Only if the rule is in the robots.txt of that subdomain. Each host, protocol and port has its own file.

• nginx rate limiting
• Prevent website scraping
• Crawl budget and bots