How to block bots on your website

To block bots without hurting SEO, first find out which bots cause the problem, then stop them where they cannot ignore you: in a firewall rule or on your web server, by user agent, network or request rate. Robots.txt only stops polite crawlers. Whatever you use, let verified search and ad crawlers through first.

Step 1: find out which bots are the problem

"Bots" covers very different visitors: Googlebot and Bingbot, which you need; ad crawlers that check your landing pages; SEO and AI crawlers; and scrapers or floods that slow the site down. Blocking all of them is neither possible nor what you want. Block the ones that cost you something.

Look at the last few days of traffic and sort it four ways:

• User agent. Honest bots name themselves, for example GPTBot or AhrefsBot. Scrapers usually pretend to be a normal browser.
• Network. Floods often come from rented cloud servers in one or two networks, even when the addresses change every minute.
• Pages. Scrapers go for product, price and search pages. Floods often hit search or filter URLs that are slow to build.
• Rate. People read pages. A client asking for dozens of pages a second is a program.

On Cloudflare, the Security analytics show all four. On your own server, use the access log. If the site is behind Cloudflare and the log shows only a few addresses, the real visitor IP is not restored yet: see Cloudflare IP ranges for the fix. Anything that says it is Googlebot needs a check, because scrapers copy Google's user agent. Our Verify Googlebot tool checks an address against Google's DNS and published IP ranges.

Which method stops which bots

In practice you combine two or three: robots.txt for the polite crawlers, a rule by network or user agent for the bot you found, and a rate limit as a safety net.

Block bots with robots.txt

Robots.txt is a request, not a lock. The standard says so directly: its rules are not a form of access authorization. It is still the right tool for crawlers that obey it, and it costs nothing. To keep two crawlers off the whole site and leave everyone else alone:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Never add a User-agent: * group with Disallow: / to a live site unless you mean to stop Google too. Copy-ready rules for every case are in robots.txt: block all or some crawlers, and which AI crawlers are worth keeping is in AI crawlers: block or allow?.

Block bots by user agent on your server

A server rule turns a named bot away with a 403, whatever it thinks of robots.txt. Replace the names with the ones from your log.

Apache (.htaccess):

SetEnvIfNoCase User-Agent "(SpamCrawler|BadScraper)" bad_bot
<RequireAll>
    Require all granted
    Require not env bad_bot
</RequireAll>

nginx:

# http { } context
map $http_user_agent $bad_bot {
    default 0;
    ~*(SpamCrawler|BadScraper) 1;
}
# inside server { }
if ($bad_bot) { return 403; }

Match full bot names, never a short word such as bot or crawl: that also matches Googlebot and bingbot. And remember that a bot can change its user agent in one line, so this works only on bots that are honest about who they are.

Block by IP address or network

When a bot hides behind a browser user agent, its address or network is what gives it away. A single address you can block in cPanel, .htaccess, nginx or a firewall, all covered in how to block an IP address. In our case studies, though, both scrapers switched between thousands of addresses, and what stayed the same was the network they rented servers from. Blocking or challenging that network (its ASN) in Cloudflare works far better: see block an IP, range or ASN in Cloudflare.

For a bot that does not hide at all but simply asks too fast, a rate limit is the cleaner answer. On your own server that is limit_req, explained in nginx rate limiting against bots.

Block bots in Cloudflare without blocking Google

If your site is behind Cloudflare, block bots there: the request never reaches your server, so it costs your server nothing. Every rule needs one part that is easy to forget. The field cf.client.bot is true for Cloudflare's verified bots, which include Googlebot and Bingbot, so adding not cf.client.bot keeps them out of the rule.

Challenge requests from cloud hosting networks (here DigitalOcean, OVH and Hetzner, a common source of scrapers):

(ip.src.asnum in {14061 16276 24940} and not cf.client.bot)

Block a scraping library that names itself:

(http.user_agent contains "python-requests" and not cf.client.bot)

Add them as custom rules in the WAF. For the first one, choose Managed Challenge rather than Block: a real person on a VPN or a work network passes the check, a script does not. What each action does is in Cloudflare challenges explained. Be careful with the one-click Bot Fight Mode on the free plan: it applies to the whole site, and custom rules cannot make exceptions to it. Details in what Bot Fight Mode blocks.

WordPress and other sites without server access

Security plugins for WordPress can block by address, user agent or rate, and they are a reasonable start on shared hosting. Keep in mind that they run inside WordPress: by the time a plugin turns a bot away, your server has already started working on the request. Against a flood that slows the site down, a rule in front of the server, such as Cloudflare on its free plan, helps more. If you use both, make sure the plugin allows Cloudflare's addresses, or it will block your visitors in bulk. Scraping of prices and content has its own set of defenses, covered in how to prevent website scraping.

Check that Google still gets in

Every new rule deserves the same check:

1. In Cloudflare Security events, filter by the Search Engine Crawler category. Nothing there should be challenged or blocked.
2. In Google Search Console, run a live test in URL Inspection on a few important pages. The page must be fetched with a normal status, not a block or challenge page.
3. A week later, look at the crawl stats report. A sudden fall in crawl requests or a rise in errors right after a change points at the change.

FAQ

Can I block all bots except Google?

For polite crawlers, yes: give Googlebot its own group in robots.txt and disallow everyone else. For bots that ignore robots.txt, use firewall rules that skip verified bots with cf.client.bot and challenge the rest by network, user agent or rate.

Does blocking bots hurt SEO?

Only if the rule also catches search crawlers. Blocking scrapers and floods often helps, because the server answers Googlebot faster. Exempt verified bots in every rule and check Search Console after each change.

What is the best way to block bots?

A rule in front of your server, aimed at what the bot cannot easily change: its network, its rate or the browser it pretends to be. Address lists alone rarely hold, because bots switch addresses.

Why do bots ignore my robots.txt?

Because following it is voluntary. Search engines and most AI crawlers obey it; scrapers and attack tools do not. Use a server or firewall rule for those.