Crawl budget: when it matters and how bots waste it

Crawl budget is how many URLs Googlebot can and wants to crawl on your site. Google defines it as two things: the crawl capacity limit and crawl demand. Most small sites never need to think about it. What matters for everyone: a slow or failing server makes Googlebot crawl less, and bot floods are a common cause.

What is crawl budget?

Google describes a site's crawl budget as the set of URLs Googlebot can and wants to crawl. It is decided by two elements:

• Crawl capacity limit. How much Googlebot can crawl without overloading your server. Google puts it as a limit on the total time your server spends holding connections open for Google. It goes up when the site responds quickly and consistently, and down when it slows down or returns errors.
• Crawl demand. How much Google wants to crawl. It depends on the site's size, how often it updates, page quality and relevance. Popular URLs and URLs that have gone stale get crawled more.

Put together: even if your server could take more, Googlebot crawls less when demand is low. And even if demand is high, it backs off when your server can't keep up.

There is no fixed number to calculate. Search Console does not show a "budget". It shows what Googlebot actually did, which is the useful part.

Who needs to care about crawl budget

Google is direct about this. Its crawl budget guide is written for two kinds of sites:

• Large sites (1 million+ unique pages) with content that changes moderately often (once a week)
• Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)

Google adds that these are not exact thresholds. It also adds a third group: sites with a large share of URLs that Search Console reports as "Discovered - currently not indexed".

If your site has a few hundred or a few thousand pages and new pages get crawled the same day you publish them, Google says you don't need the guide. Keeping your sitemap up to date and checking the Page Indexing report is enough. Most blogs, small stores and company sites are in that group.

There is one exception that has nothing to do with size. The capacity half of crawl budget applies to every site. If your server is slow or erroring, Googlebot crawls less, whether you have 50 pages or 5 million.

How bot load and server errors cut Google's crawling

This is the part most crawl budget guides skip. Googlebot watches your server's health while it crawls. Google's guide says the capacity limit drops when the site slows down (latency and time to first byte get longer) or when it responds with 5xx server errors or rate limiting signals such as HTTP 429. The Crawl Stats help page says the same thing: if your site responds slowly or your server error rate rises, Googlebot throttles back its requests to avoid overloading your server.

Now think about where that slowness comes from on a small or mid-size site. Very often it is not real visitors. It is scrapers, SEO tools, AI crawlers and scripts that hit search pages, filters and login forms at hundreds of requests per minute. Your server spends its CPU on them, response times climb, a few requests time out with 500 or 503, and Googlebot reads all of that as "this server is struggling". It slows down. Your new and updated pages wait longer to be crawled.

The same mechanism works on purpose. Google's documentation on reducing crawl rate says that returning 500, 503 or 429 to crawl requests makes Googlebot slow down, and that 503 and 429 responses can carry a Retry-After header. Google limits this to emergencies: a couple of hours, or 1-2 days. If Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from Google's index.

That last point is where bot protection goes wrong. A rate limit or firewall rule that is too broad can answer Googlebot with 429 or 403 every day, not just during an attack. In the traffic reports we have prepared, the usual cause is a hand-written rule that blocks a country, a cloud network or a user agent pattern and catches Google's or Bing's crawlers along the way. If you add an nginx rate limit or an edge rule, exclude verified crawlers. Check them by reverse DNS or against Google's published IP ranges, never by user agent alone, because user agents are easy to fake. Our Googlebot verification tool does that check for an IP from your logs.

What helps and what does not

A lot of crawl budget advice tries to "save" budget with tricks Google says don't work. Here is what Google's guide actually says.

• noindex does not save crawling. Google still requests the page to see the noindex tag, so it wastes crawling time. Use it to keep pages out of results, not to save budget.
• robots.txt is for URLs you never want crawled. Faceted filters, endless sort orders and infinite scroll pages that duplicate other pages are the typical examples.
• robots.txt does not reallocate budget on the fly. Google says not to use it to temporarily move budget to other pages. Freed capacity goes to other pages only if Google is already hitting your capacity limit.
• Removed pages should return 404 or 410. A 404 is a strong signal not to crawl that URL again. Soft 404s, pages that say "not found" but return 200, keep getting crawled and waste budget.
• You can't ask for more. Google's request form only reports a crawl rate that is too high. You cannot request an increase.

Faceted navigation deserves a closer look because it is the most common real budget leak on stores. Filters can generate an almost infinite number of URLs. Google's faceted navigation guide recommends blocking them with robots.txt, for example:

User-agent: Googlebot
Disallow: /*?*color=
Disallow: /*?*size=

Keep your main category pages crawlable and block only the filter parameters. A mistake here can hide real pages, so read how to block bots with robots.txt before you change the file. Google also recommends returning 404 for filter combinations that have no results.

How to check crawl budget in Google Search Console

The Crawl Stats report shows how Googlebot crawls your site. Open Search Console, click Settings, then Crawl stats under property settings. It works only for root-level properties: a Domain property or a URL-prefix property like https://example.com, not a subfolder.

What to look at:

1. Total crawl requests. All requests for your URLs, successful or not. A sudden long drop is the signal to dig.
2. Average response time. The time to fetch your resources. If it climbs, expect crawl requests to fall after it.
3. Host status. robots.txt fetching, DNS resolution and server connectivity. A red mark means failures crossed Google's threshold on some days.
4. By response. Look for growing shares of 5xx, 429 and 403. On a healthy site nearly everything is 200, 301 or 304.
5. By purpose. Discovery is new URLs, Refresh is known ones. A store whose new products are not discovered has a problem here.

Then line the dates up with your own traffic. If average response time spiked on the same days your server logs or Cloudflare show a surge of non-human requests, the bots are eating your crawl capacity. For individual URLs, the URL Inspection tool can show a "Hostload exceeded" error, which means Google could not crawl because your server lacked capacity.

Crawl budget optimization checklist

Fix things in this order. The first two help every site, the rest matter more the larger the site gets.

Google's two direct answers to "how to increase crawl budget" are the first row and quality: add server capacity if your server is the limit, and improve your content so Google wants to crawl more of it. Everything else in the table stops crawling from being wasted.

If you suspect bots are behind your slow days, measure before you block. A free bot traffic report shows which networks, user agents and hours cause the server load, and whether any of your current rules already hit Googlebot or Bingbot. Our case studies show what that looks like on real sites.

FAQ

What is crawl budget in SEO?

It is the number of URLs Googlebot can and wants to crawl on your site, set by your server's crawl capacity and Google's crawl demand. It matters for SEO because a page has to be crawled before it can be indexed and shown in search.

Does a small site need crawl budget optimization?

Usually not. Google says sites without many rapidly changing pages, or whose new pages get crawled the same day, don't need its crawl budget guide. A fast, error-free server and an up-to-date sitemap cover it.

Can I see my crawl budget in Google Search Console?

Not as a single number. The Crawl Stats report under Settings shows total crawl requests, download size, average response time, host status and a breakdown by response code, which tells you how Googlebot actually crawls your site.

Does blocking bad bots increase Google's crawling?

It can, if the bots were slowing your server or causing errors, because Google crawls more when the server responds quickly and consistently. It only helps if your rules never block or rate limit Googlebot itself.

• Block traffic by country
• Cloudflare challenges
• Block or allow an IP in Cloudflare