Nginx rate limiting against bots

Nginx rate limiting takes two directives: limit_req_zone sets the key (usually the client IP), a shared memory zone and a rate such as 10r/s, and limit_req applies it with optional burst and nodelay. Return 429 instead of the default 503, restore real IPs behind Cloudflare, and exempt Google's published crawler ranges.

How limit_req works

Nginx counts requests per key in a shared memory zone and lets them through at a fixed average rate, using the "leaky bucket" method. You define the zone once in the http block and apply it where you need it:

http {
    limit_req_zone $binary_remote_addr zone=site:10m rate=10r/s;

    server {
        location / {
            limit_req zone=site burst=20 nodelay;
        }
    }
}

• $binary_remote_addr is the key: the client IP in compact form. The key can be any variable or combination of variables, and requests with an empty key are not counted at all. That last rule is what makes clean exemptions possible.
• zone=site:10m is the name and size of the memory zone. One megabyte holds about 16,000 states on 32-bit systems or about 8,000 on 64-bit ones, so 10m tracks roughly 80,000 IPs. When the zone fills up, nginx drops the least recently used state.
• rate=10r/s is the average rate per key. Nginx accepts only requests per second (r/s) and requests per minute (r/m), so half a request per second is written as 30r/m.
• burst=20 is how many requests over the rate nginx will queue. By default burst is zero, so every request above the rate is rejected.
• nodelay serves the burst immediately instead of spacing it out. Without it, extra requests wait in line, which makes normal pages feel slow.
• delay= (nginx 1.15.7 and later) is the middle ground: the first requests over the rate pass at once, the rest of the burst is delayed.

Rejected requests get 503 by default. Set limit_req_status 429; instead. 429 Too Many Requests is the status code defined for rate limiting, and it tells clients and monitoring tools what happened rather than suggesting your server is down.

Can nginx limit per hour or per day?

Not directly. There is no r/h or r/d. The closest you get is a per-minute rate with a burst, for example rate=1r/m burst=60, which allows a quick run of 60 requests and then one per minute. That smooths traffic, but it is not a quota. A real "100 requests per hour per user" counter belongs in your application or in a service at the edge.

Behind Cloudflare or a proxy: fix the client IP first

If your site sits behind Cloudflare, a load balancer or any reverse proxy, nginx sees the proxy's address, not the visitor's. Every visitor then shares a handful of Cloudflare IPs, one busy scraper uses up the limit for all of them, and real customers and Googlebot start getting 429s together.

The fix is the realip module. Cloudflare passes the visitor's address in the CF-Connecting-IP header. Tell nginx to trust that header only when the request comes from Cloudflare's own ranges:

# http block: one line per range from https://www.cloudflare.com/ips-v4 and /ips-v6
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
# ... the rest of Cloudflare's list
real_ip_header CF-Connecting-IP;

After this, $remote_addr and $binary_remote_addr hold the visitor's IP, so the rate limit, your logs and the Googlebot exemption below all work on real addresses. Cloudflare's list changes from time to time, so refresh it regularly. The module is not compiled in by default; check that nginx -V shows --with-http_realip_module.

For other proxies, use real_ip_header X-Forwarded-For; with real_ip_recursive on; when there are several proxies in a chain. Never use $http_x_forwarded_for as the rate limit key directly. Anyone can send that header with any value, and a scraper that rotates it gets a fresh limit on every request.

Keep Googlebot and ad crawlers out of the limit

A rate limit that hits Googlebot does more harm than the bots it stops. Google's documentation says that 500, 503 and 429 responses make Googlebot reduce its crawl rate for the whole hostname, and if the same URL keeps returning them for several days, it may be dropped from the index. Crawling recovers once the errors stop, but new and updated pages wait in the meantime. More on how that adds up in how bots eat your crawl budget.

Exempt crawlers by IP, never by user agent. Anyone can send "Googlebot" in the user agent header, and a user agent exemption is the first thing a scraper will exploit. Google publishes its crawler ranges as JSON files (see Googlebot IP ranges). Turn them into an nginx include file:

curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
        https://developers.google.com/static/crawling/ipranges/special-crawlers.json \
  | jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " 1;"' \
  > /etc/nginx/google-crawlers.geo

common-crawlers.json covers Googlebot; special-crawlers.json covers special-case crawlers such as AdsBot. Run the command on a schedule and reload nginx, because the lists change. Then map those IPs to an empty key:

geo $rate_exempt {
    default 0;
    include /etc/nginx/google-crawlers.geo;
}

map $rate_exempt $limit_key {
    0 $binary_remote_addr;
    1 "";
}

Use $limit_key instead of $binary_remote_addr in every zone. Requests from Google's ranges now have an empty key, and nginx does not count them. Add other crawlers you rely on the same way when their operator publishes ranges. To check whether an IP in your logs is really Google, use Verify Googlebot; Google's own method is a reverse DNS lookup followed by a forward lookup.

Ready configs: whole site, login, search, connections

Good limits are loose for the site and tight for the few paths bots abuse. This setup goes in the http block, after the geo and map above:

limit_req_zone  $limit_key zone=site:10m   rate=10r/s;
limit_req_zone  $limit_key zone=login:10m  rate=5r/m;
limit_req_zone  $limit_key zone=search:10m rate=30r/m;
limit_conn_zone $limit_key zone=conn:10m;

limit_req_status  429;
limit_conn_status 429;

server {
    limit_req  zone=site burst=50 nodelay;
    limit_conn conn 30;

    location = /wp-login.php {
        limit_req zone=site  burst=50 nodelay;
        limit_req zone=login burst=5  nodelay;
        # your usual fastcgi_pass or proxy_pass lines
    }

    location /search {
        limit_req zone=site   burst=50 nodelay;
        limit_req zone=search burst=10 nodelay;
        # your usual fastcgi_pass or proxy_pass lines
    }
}

• Whole site: 10 requests per second with a burst of 50 does not bother a person clicking around, but stops a script pulling hundreds of pages a minute. If your pages load many images and scripts from the same server, move this limit into the location that passes requests to PHP or your app so static files are not counted.
• Login: 5 per minute with a burst of 5 is plenty for a human who mistypes a password and useless for password guessing. An exact location = match wins over your PHP location, so copy the PHP handler lines into it.
• Search: search pages are expensive to build and a favorite target of scrapers. WordPress search uses ?s= on the home page, so limit it by argument instead of path: map $arg_s $search_key { "" ""; default $limit_key; } and use $search_key in the search zone. Requests without a search term get an empty key and are not counted.
• Connections: limit_conn caps how many requests one key can have in progress at once, which catches slow, parallel scrapers that stay under the rate. Over HTTP/2 and HTTP/3, each concurrent request counts as a separate connection, so do not set it too low.

Note the repeated site line inside each location. A location with its own limit_req does not inherit the one from the server level, and several limit_req directives can work together.

Rate limiting by header, user or API key

The key can be any variable, so you can limit per API key, per logged-in user cookie or per any header. A map picks the header value when it is present and falls back to the IP when it is not:

map $http_x_api_key $api_key {
    ""      $binary_remote_addr;
    default $http_x_api_key;
}

limit_req_zone $api_key zone=api:10m rate=60r/m;

Only do this with values your application validates. A header key is chosen by the client, so a bot that invents a new value on every request also gets a new limit every time. For unauthenticated traffic, stay with the client IP.

Test before you enforce, and why limits "don't work"

Turn on limit_req_dry_run on; (nginx 1.17.1 and later) for a few days. Nginx counts excess requests but lets them through. Add $limit_req_status (1.17.6 and later) to your log_format and you see PASSED, DELAYED, REJECTED or REJECTED_DRY_RUN on every line, so you know who would be blocked before anyone is.

When a limit seems to do nothing, check these first:

1. The location that handles the request has its own limit_req, so the server-level one is not inherited.
2. Behind a proxy without realip, every request has the same key and the limit hits everyone at once.
3. The key is empty for the traffic you want to limit, for example a header that bots simply do not send.
4. Nginx was not reloaded, or nginx -t reported an error.

Nginx or the edge?

Nginx limits work after the request has reached your server. If you are on Cloudflare, rate limiting rules can do the same counting at the edge before traffic arrives; the Free plan includes one rule with a 10-second counting period. For a flood from a few fixed addresses, blocking the IP addresses is simpler than any rate.

Either way, limits work best when they target the paths and patterns that cause the load, not the whole site with one tight number. A free bot traffic report shows which URLs, networks and hours actually generate your traffic before you choose the numbers.

FAQ

What is the default rate limit in nginx?

There is none. Nginx limits nothing until you define a limit_req_zone and apply it with limit_req. Once a limit applies, the default burst is zero and the default rejection code is 503.

What status code should a rate limit return?

429 Too Many Requests, set with limit_req_status 429;. Google treats 429 like 503 and slows crawling, so keep crawler IPs exempt and limits targeted.

How do I rate limit only POST requests?

Use a map on $request_method that returns your key for POST and an empty string for everything else, then use that variable as the zone key. Empty keys are not counted.

How do I rate limit by X-Forwarded-For?

Do not use the header as the key. Set real_ip_header X-Forwarded-For; with set_real_ip_from for your trusted proxies, then limit on $binary_remote_addr, which now holds the real client IP.

• Prevent website scraping
• Crawl budget and bots
• Block traffic by country