Case studies: slow sites, crashes and scrapers

Real stories from two online stores, each starting with what the owner saw and searched for: the site is slow, it keeps going down, "508 Resource Limit Is Reached". Then what was really going on, what we did and what happened next. Names and addresses are removed; the numbers and charts are real.

Site slow and going down several times a week

What the owner saw. An online store selling in several European countries. Several times a week pages stopped loading and the site went down. To stop it, the owner switched on Cloudflare's "I'm Under Attack" mode for two days.

What was really going on. One bot was sending half of all the site's traffic.

• Up to 928,000requests a day from rented servers in one network: more than a thousand addresses, all pretending to be the same Chrome browser on Linux. A week earlier this "browser" sent under 3,000 a day.
• Google locked out"I'm Under Attack" mode also turned away Google: its phone crawler about 2,200 times, the Google Ads checker 1,021 times, and the Google Shopping crawler.
• Bing locked outA ready-made "block AI bots" rule was blocking the real Bing crawler: about 35,000 refusals in 11 days, on every page of the site.

What we did. The report named the bot by the two things it could not change: the network it rented servers from and the browser it pretended to be. The strict mode was already off by then; the owner added both to a block rule they already had. For Bing, the report pointed to the exact rule to fix.

What happened next. From the same evening the bot no longer reached the store, and Google got normal pages again.

Facebook crawler slowing the site down

What the owner saw. Two weeks later the same store got slow again, this time in waves during the day.

What was really going on. Meta's ad crawler: the bot that opens the pages your Facebook and Instagram ads point to, a relative of the better-known facebookexternalhit and meta-externalagent. On its first wave it sent 272,000 requests in a day instead of the usual 2,000 to 3,000 and took 79% of the server's time. On 7 October the store went from 25,000 to 78,000 requests an hour, the slowest pages went from 2 seconds to 54, and shoppers saw 194 error pages at the peak.

Page requests per hour, thousands

Slowest page loads, seconds

Show the numbers

Time (UTC)	Page requests per hour, thousands	Slowest page loads, seconds
10-07 00:00	21.4	2.2
10-07 01:00	14.9	2.9
10-07 02:00	18.3	2.4
10-07 03:00	15.8	2.9
10-07 04:00	20.4	2.3
10-07 05:00	19.2	1.8
10-07 06:00	23.4	2.1
10-07 07:00	25.7	2.0
10-07 08:00	36.0	1.8
10-07 09:00	53.2	1.7
10-07 10:00	78.1	10.0
10-07 11:00	67.4	28.1
10-07 12:00	62.0	38.9
10-07 13:00	56.6	54.1
10-07 14:00	58.3	50.5
10-07 15:00	60.3	46.3
10-07 16:00	51.5	47.6
10-07 17:00	32.0	2.0
10-07 18:00	32.4	1.9
10-07 19:00	37.4	2.0
10-07 20:00	41.0	2.3
10-07 21:00	33.7	2.3
10-07 22:00	31.5	2.4
10-07 23:00	29.9	2.1

Why not just block it. The store's ads would stop working: Meta has to be able to open the pages the ads lead to.

What we suggested. Keep the ad pages open and limit only what the crawler has no reason to do: send forms and open catalog pages with filters. We also flagged rented Google Cloud servers that were crawling the store. They are not Google's search crawler, so they get a quick automatic check that real crawlers skip.

What happened next. Each wave ended on its own; nothing was blocked, so the ads were not affected.

Site slow, but it was not bots

What the owner saw. Ten days later pages slowed down again: the slowest went from about 2 seconds to 15, and category pages took up to two minutes. The owner expected another bot wave.

What was really going on. The opposite. Fewer requests were reaching the server than usual: 14,000 to 15,000 an hour instead of 17,000 to 24,000. Yet everything got slow at once: the catalog, other pages and the admin panel. Visitors who gave up waiting went from 150 to 300 an hour to as many as 1,429.

Page requests to the server per hour, thousands

Slowest page loads, seconds

Show the numbers

Time (UTC)	Page requests to the server per hour, thousands	Slowest page loads, seconds
10-17 12:00	18.9	1.6
10-17 13:00	21.1	1.4
10-17 14:00	22.7	1.4
10-17 15:00	21.5	1.5
10-17 16:00	22.1	1.5
10-17 17:00	19.7	1.8
10-17 18:00	17.1	1.8
10-17 19:00	16.9	1.7
10-17 20:00	19.6	1.7
10-17 21:00	21.6	1.6
10-17 22:00	19.4	1.6
10-17 23:00	17.7	1.8
10-18 00:00	19.1	1.9
10-18 01:00	18.4	1.7
10-18 02:00	18.6	1.6
10-18 03:00	14.4	1.5
10-18 04:00	16.3	1.6
10-18 05:00	16.9	1.5
10-18 06:00	20.4	1.5
10-18 07:00	20.0	1.7
10-18 08:00	20.6	2.0
10-18 09:00	22.4	1.6
10-18 10:00	24.3	1.7
10-18 11:00	21.1	1.8
10-18 12:00	21.5	1.9
10-18 13:00	18.4	1.7
10-18 14:00	18.8	6.5
10-18 15:00	13.7	15.5
10-18 16:00	14.6	15.3
10-18 17:00	15.2	17.2
10-18 18:00	14.4	16.0

Less traffic and a slower site means the problem is the server or the database, not visitors. The slowdown started a few hours after someone began unusually heavy work in the admin panel, so the first things to check were a product import, a scheduled task or a cleared cache.

What we suggested. Send the developer and the hosting company the exact hours and the slow pages, and check the database's log of slow queries. Blocking more bots would not have fixed this. The report still named one catalog scraper on cloud servers that caused about a fifth of the load on category pages, with a ready rule for it.

"508 Resource Limit Is Reached" on shared hosting

What the owner saw. A store on shared hosting kept hitting its plan's limit. On six days shoppers, Google and Bing got "508 Resource Limit Is Reached" instead of the page.

Error pages per day

Show the numbers

Day	Error pages per day
09-30	6
10-01	2
10-02	0
10-03	0
10-04	5
10-05	1
10-06	0
10-07	1
10-08	4
10-09	1
10-10	3
10-11	2
10-12	9
10-13	22
10-14	135
10-15	405
10-16	3
10-17	2
10-18	2
10-19	276
10-20	188
10-21	159
10-22	19
10-23	101
10-24	2
10-25	4
10-26	4
10-27	4

What was really going on. A scraper that used 130,000 different addresses sent 196,000 requests to the server in 4 weeks and took 27% of all its work. 91% of what it asked for did not exist: it glued menu links together into endless made-up page addresses, and the server had to answer each one.

• One scraper bot: 25.0 h (27%)
• Shoppers, Google and everything else: 68.5 h (73%)

Show the numbers

Part	Hours	Share
One scraper bot	25.0	27%
Shoppers, Google and everything else	68.5	73%

What we did. One rule that gives this bot a quick automatic check that people pass and bots fail, while real Google, Bing and ad crawlers skip it. Before suggesting it we made sure no shopper would be caught: this "browser" never loaded a single style or script file, which every real browser does. The report also found that 15% of all requests came from the store's own server, which one server setting fixes.

The full report for this store is our sample report: every finding, the exact rules and how we checked them.

What these cases have in common

• Quick fixes often lock out Google. "I'm Under Attack" mode and ready-made "block AI bots" rules stopped Google in one case and Bing in another.
• Blocking addresses rarely helps. Both scrapers switched between thousands of addresses. What stayed the same was the browser they pretended to be and the network they came from.
• Some bots you need. Facebook's ad crawler has to see your ad pages. Limiting what it does is safer than blocking it.
• Not every slowdown is bots. The numbers tell you whether you need a rule or a developer.