How to prevent website scraping
You cannot make public pages 100% scrape-proof: anything a browser can load, a script can copy. What you can do is make scraping slow and expensive. Rate-limit catalog pages, challenge traffic from hosting networks, check fake Googlebots, control AI crawlers, and keep private data behind a login, while verified search crawlers keep full access.
What "preventing scraping" can realistically achieve
Start with an honest goal. A determined scraper with residential proxies and a real browser can copy any page a visitor can see. No setting changes that. Your goal is narrower and very achievable:
- Make it expensive. Every challenge, rate limit and blocked network forces the scraper to spend more on proxies, browsers and time.
- Make it slow. A competitor who needs your prices every hour gets them once a day, or gives up.
- Protect your server. Scrapers hit thousands of uncached product and category URLs in a row. That load is what real customers feel as a slow site.
- Keep analytics clean. Bots that run JavaScript show up as fake sessions, with odd bounce rates and traffic spikes from unexpected countries.
And one hard constraint: none of this may block Googlebot, Bingbot or ad crawlers. A defense that drops your search traffic costs more than the scraping did.
How to recognize scraping in your logs
Scrapers rarely look like people. Before you write any rule, look for these patterns in your server logs or Cloudflare Security Analytics:
- Sequential URLs. One IP or a small group walks through product pages, category pages or pagination in order:
?page=1,?page=2, and so on. - Cloud and hosting networks. Requests come from data center ASNs such as DigitalOcean, OVH, Hetzner or AWS. Real shoppers browse from home and mobile providers.
- Odd or forged user agents. Empty strings, library names like
python-requests, outdated browser versions, or a user agent that says Googlebot from an IP that is not Google's. - No assets. The client requests HTML pages but never loads images, CSS or JavaScript.
- A steady rhythm. One request every second, hour after hour. People click in bursts and pause to read.
- Night-time load. Server load climbs at 3 a.m. local time while your sales and real visits stay flat.
A user agent claiming to be Googlebot proves nothing. Google's own guidance is to confirm the IP with a reverse DNS lookup that resolves to googlebot.com, google.com or googleusercontent.com, then a forward lookup back to the same IP, or to match it against Google's published IP range files. You can do that for any IP in your logs with Verify Googlebot.
Layered defenses that make scraping expensive
No single control stops scraping. Each layer removes one group of scrapers, and together they leave only the expensive ones.
robots.txt: a request, not a lock
A Disallow line tells honest crawlers to stay away. Google states plainly that robots.txt cannot enforce crawler behavior and that other crawlers might not obey it. Use it to steer well-behaved bots away from filters and internal search, but never count on it against a scraper. More in how to block bots with robots.txt.
Rate limiting at the edge or on the server
Rate limiting caps how many requests one IP can make in a time window. On Cloudflare, the Free plan includes 1 rate limiting rule that counts requests per IP over 10 seconds, and its expression can use the path and the verified bot field. Pro allows 2 rules, Business 5, with longer periods. The rule can block or show a challenge. An example is further below.
On your own server, nginx does the same with limit_req. Behind Cloudflare you must first restore the real visitor IP with the realip module and the CF-Connecting-IP header, otherwise nginx limits Cloudflare's own addresses. Setup details are in nginx rate limiting.
SEO risk: Googlebot can crawl fast. A tight server-side limit can return errors to it. Exempt crawlers only by verified IP (reverse DNS or Google's published ranges), never by user agent, which anyone can fake.
Challenges for suspicious traffic
A Managed Challenge lets most real people through without any interaction and stops simple scripts that cannot run a browser. Cloudflare recommends it for most WAF rules. Challenge pages return HTML, so they break API and AJAX calls; keep challenges off endpoints your own front end calls with fetch. See Cloudflare challenges explained.
Challenging hosting networks
Most cheap scraping runs on rented servers. A WAF custom rule on the ip.src.asnum field can challenge traffic from those networks on the pages scrapers want. Free plans include 5 custom rules, enough for this. Keep Google's and Microsoft's networks out of such a list: they carry their own crawlers and services too.
AI crawler controls in Cloudflare
AI crawlers that collect content for model training are a separate case. Cloudflare's AI Crawl Control, available on all plans, shows which AI services visit your site and lets you allow or block each crawler; blocking creates a WAF custom rule. On the Free plan crawlers are identified by user agent only. Under Security settings, Configure AI bot policies sets one action for each group: Search, Agent and Training. The old "Block AI bots" switch was deprecated on September 15, 2026. These controls stop AI companies that identify themselves, not anonymous scrapers. Details in AI crawlers: block or allow?.
Login walls and data design
If data is truly private, such as wholesale prices, a dealer catalog or a member directory, put it behind a login. Google's own guidance for sensitive content is to use a blocking method other than robots.txt. Search engines will not index it, which is the point.
Also check what your site hands out for free. Many stores and WordPress themes load the full catalog, with prices and stock, from a JSON endpoint that is far easier to scrape than the HTML. Return only the fields the page needs, paginate, and apply the same rate limits to those endpoints.
Which defense stops what
| Defense | Stops | Does not stop | Risk to SEO |
|---|---|---|---|
| robots.txt | Honest crawlers that follow the rules | Any scraper that ignores it | High if you disallow pages you want indexed |
| Rate limiting | Fast bulk crawling from one IP | Slow scrapers, rotating proxies | Medium: can hit fast crawlers unless verified bots are excluded |
| Managed Challenge | Scripts that cannot run a browser | Headless browsers with good proxies | Low with not cf.client.bot, high without it |
| Challenging hosting ASNs | Cheap scraping from rented servers | Residential and mobile proxies | Medium: crawlers also live in big cloud networks |
| Verifying fake Googlebots | Scrapers hiding behind a crawler user agent | Scrapers with ordinary browser user agents | None if you verify by IP and DNS |
| AI crawler controls | Declared AI crawlers you choose to block | Undeclared scrapers | Low, but check which crawlers also feed search |
| Login wall | All anonymous access | Scrapers with paid or stolen accounts | Content behind it is not indexed |
Two Cloudflare rules to start with
Rule 1: challenge hosting networks on product pages. In WAF custom rules, create a rule with this expression and the action Managed Challenge. The ASNs here are DigitalOcean (14061), OVH (16276) and Hetzner (24940); replace them with the networks your own logs show, and adjust the path to your URL structure.
(ip.src.asnum in {14061 16276 24940} and http.request.uri.path contains "/product/" and not cf.client.bot)Rule 2: challenge high-rate crawling of category pages. In rate limiting rules, use this expression, count requests by IP, and choose Managed Challenge as the action. Start with a generous threshold that a fast human shopper never reaches, then tighten it after watching Security Events for a few days.
(http.request.uri.path contains "/category/" and not cf.client.bot)The not cf.client.bot part matters in both. It is true for Cloudflare's verified bots, so Googlebot, Bingbot and other verified crawlers skip the rule even when they come from a cloud network or crawl quickly. In the traffic reports we have prepared, the usual SEO damage came from hand-written rules without it: a country, cloud network or user agent pattern that caught Google's or Bing's crawlers by accident.
After any new rule, filter Security Events by the Search Engine Crawler category and confirm nothing there is challenged or blocked.
Where to start on your own site
Guessing at scrapers usually blocks the wrong traffic. Look first at which paths get hammered, from which networks and at what hours, then pick the two or three layers that fit. If your site is on Cloudflare, a free bot traffic report breaks your traffic down by source and shows where scrapers load the server, before you write a single rule.
FAQ
Can you completely prevent web scraping?
No. Anything a browser can display can be copied by a script driving a real browser. Defenses raise the cost and slow scrapers down, and they protect your server and analytics. Only a login wall keeps anonymous visitors out entirely.
How do I stop AI companies from scraping my site?
Add rules for their crawlers in robots.txt and, on Cloudflare, block them in AI Crawl Control or with AI bot policies. This works for crawlers that identify themselves. Check which crawlers also power search features before you block them.
Will anti-scraping rules hurt my SEO?
Only if they catch search crawlers. Exclude verified bots in every Cloudflare rule with not cf.client.bot, never allow crawlers by user agent alone, and check Security Events and Search Console crawl stats after each change.
How do I prevent scraping on WordPress?
The same layers apply: rate-limit product, category and search URLs, challenge hosting networks, and check what the REST API and theme endpoints expose. A security plugin helps on the server, but edge rules stop the load before it reaches WordPress.
Rather not deal with this yourself?
If bots slow down your site, copy it or fill your analytics with fake visits, we can sort it out. We find out which bots visit your site, stop the harmful ones and check that Google, Bing and ad crawlers still get in. It starts with a free report.