Reduce Google’s crawling of our website

Google ignores Crawl-delay. How to slow down its crawling: block fake Googlebots by IP and return 503 or 429 with Retry-After on Apache, LiteSpeed and nginx.

Author: Carlos Sánchez

Topics: Crawling

Published: 2026-11-02

Reviewed: 2026-11-02

Index show

It's true that we will rarely ask ourselves how to make Google crawl our website less often. But when analyzing the logs and the uptime of our website, we may find a correlation: it goes down at the peaks when Google is crawling it.

Or, to get trickier, we may be in the middle of changes in production where we don't want Google to crawl us too much, so that it doesn't keep low-quality content in its cache, but the change isn't drastic enough to put the website in 5XX (or to serve 5XX only to Google).

The problem with the Crawl-delay directive

The Crawl-delay directive in the robots.txt file doesn't work with Google.

Other search engines, such as Bing, do respect it to space out their requests.

It's also true that Bing makes more of an effort: in Bing Webmaster Tools you even have a crawl control where you can tell Bing the best hours to crawl you.

However, not all bots are that considerate when it comes to letting us organize the crawling of our website. In fact, sometimes we need to know how to block access to user agents, even with measures much more powerful than robots.txt. Googlebot, which is also a bot we generally want on our website, ignores the Crawl-delay directive completely, and Search Console doesn't give us an option to change the crawl rate either.

So we could give up and decide it's better not to touch Google and let it crawl whatever it wants. But in some projects there can be a peak in Google's crawling at times when the server can't handle that much load.

I've seen Crawl-delay applied directly to Googlebot, and it can sometimes lead to rules being applied in unexpected ways, because Googlebot ignores this directive and that has its risks, as we can see in this example:

Problem with ignored robots.txt directives

My recommendation would always be to have the server ready for peak times, but we may want to limit Google because it crawls us excessively.

Blocking by IP range

The most likely case when a website goes down because of excessive crawling by Google's bots is that they aren't really Google's bots. And although it's relatively complex to implement, if we don't use a CDN that does this for us, we can block on our website every Google user agent that doesn't come from this IP range.

Google's crawler IP range lists

Although it isn't what I'd recommend, this list is very important for websites that are, for example, banned in the US and need to block it, but still need to appear on Google.

For what concerns us here, blocking crawling for anything that doesn't fall within the IP range often works quite well.

Airbnb, for example, does this: if we pretend to be Google, it blocks us.

But when we use a Google service, with Google's user agent and from Google's IP range, we can see that it lets us through without much trouble.

For anyone wondering, neither this implementation nor the hyper-personalization it does for the user hurts its rankings; if anything, the opposite.

Now, what happens if we make these changes and still find that our legitimate Google is really the one crawling us excessively? Well, we have an ace up our sleeve.

503 or 429 status codes and the Retry-After header

The technically correct way to make Googlebot stop is to return a 5XX or 429 HTTP status code.

This can come with the Retry-After header. That way you protect your infrastructure immediately and give Google a standardized instruction on when it's safe to come back. Otherwise, you leave it up to Google when to try again: it will reduce crawling considerably and take a while to make changes in the index, but with Retry-After you're in control.

On October 6, Google updated its guide to reducing the crawl rate and explicitly added the use of this header (as defined in RFC 9110).

The guide shows how the delay can be given in seconds:

HTTP/1.1 503 Service Unavailable Retry-After: 120

Or as an absolute date and time in UTC:

HTTP/1.1 503 Service Unavailable Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

How to send `Retry-After` on Apache, LiteSpeed and nginx

Two decisions: which responses it goes on (503 for maintenance, 429 for rate limiting) and with what value (seconds, or an HTTP date such as Retry-After: Wed, 07 Oct 2026 18:00:00 GMT). And one rule: the header goes with the status, not with the path. If you add it to every response in a directory, the 200s will get it too, and there it means nothing.

Apache. Two modules you already have do the job: mod_rewrite sets the status and mod_headers the header, tied to the status with expr= (Apache 2.4.10 or later). It works the same in .htaccess and in the VirtualHost.

Whole site · .htaccess in the root, or the VirtualHost

# Maintenance for the whole site: 503 and a one-hour Retry-After.
# Your IP still gets in, and the maintenance page (and its folder) isn't blocked.
<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{REMOTE_ADDR} !^203\.0\.113\.10$
  RewriteCond %{REQUEST_URI} !^/maintenance
  RewriteRule ^ - [R=503,L]
</IfModule>
ErrorDocument 503 /maintenance.html

# The header only when the response is a 503 (Apache 2.4.10 or later).
# "always": without it, Apache only adds headers to 2xx responses.
<IfModule mod_headers.c>
  Header always set Retry-After "3600" "expr=%{REQUEST_STATUS} == 503"
</IfModule>

Just one directory · .htaccess inside /shop/

# Only /shop/ returns 503; the rest of the site stays up.
# In the VirtualHost it would be the same inside <Location /shop/> … </Location>
<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{REMOTE_ADDR} !^203\.0\.113\.10$
  RewriteRule ^ - [R=503,L]
</IfModule>
ErrorDocument 503 /maintenance.html

<IfModule mod_headers.c>
  Header always set Retry-After "7200" "expr=%{REQUEST_STATUS} == 503"
</IfModule>

429 for a specific bot · Apache doesn't limit requests on its own

# Real rate limiting is a job for a WAF or mod_security. But if you want to stop
# a bot with a 429 (which says "come back later") instead of a 403 (which says "go away"):
<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|SemrushBot) [NC]
  RewriteRule ^ - [R=429,L]
</IfModule>

<IfModule mod_headers.c>
  Header always set Retry-After "600" "expr=%{REQUEST_STATUS} == 429"
</IfModule>

LiteSpeed. It reads Apache's .htaccess, but not all of it: LiteSpeed Enterprise understands Header with always and with env=, but not Apache 2.4's expr= condition (and a directive it doesn't recognize doesn't throw an error: it simply does nothing); OpenLiteSpeed only reads the mod_rewrite rules. So here PHP sets the status and the header: the rule rewrites to a script, and the script responds.

Whole site or one directory · the same .htaccess, in the root or inside /shop/

# Maintenance: everything that doesn't come from your IP is served by maintenance.php,
# which is what returns the 503 and the header. No R=503, no mod_headers and no
# ErrorDocument: it works the same on LiteSpeed Enterprise and on OpenLiteSpeed
# (on OpenLiteSpeed, with "Auto Load from .htaccess" enabled under Rewrite).
RewriteEngine On
RewriteCond %{REMOTE_ADDR} !^203\.0\.113\.10$
RewriteCond %{REQUEST_URI} !^/maintenance
RewriteRule ^ /maintenance.php [L]

maintenance.php · in the root: the status, the header and the page

<?php
// 503 and Retry-After from PHP: any server understands this.
http_response_code(503);
header('Retry-After: 3600');        // seconds, or an HTTP date in GMT
header('Cache-Control: no-store');  // so that no cache keeps the 503
header('Content-Type: text/html; charset=utf-8');
?>
<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>Maintenance</title></head>
<body>
  <h1>We'll be back in an hour</h1>
  <p>We're making some changes. Please try again a little later.</p>
</body>
</html>

429 for a specific bot · .htaccess rewrites, limit.php responds

# LiteSpeed's per-IP limit (Per Client Throttling, in the WebAdmin) slows clients
# down, but it doesn't return a 429 or send Retry-After: it leaves the extra requests
# waiting. To tell a bot "come back in ten minutes", let PHP say it:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|SemrushBot) [NC]
RewriteCond %{REQUEST_URI} !^/limit\.php$
RewriteRule ^ /limit.php [L]

limit.php · in the root: a few lines

<?php
http_response_code(429);
header('Retry-After: 600');
header('Content-Type: text/plain; charset=utf-8');
echo 'Too many requests. Please try again in ten minutes.';

nginx. The header is declared once with a map on $status: nginx doesn't send a header whose value is empty, so Retry-After only goes with 503s and 429s. The status is set by return or limit_req.

The header, just once · in http { } and in the server { }

# nginx doesn't send a header whose value is empty: with a map, Retry-After
# only goes with the statuses that need it. This applies to everything below.
map $status $retry_after {
    503     3600;   # maintenance: one hour
    429     10;     # rate limiting: ten seconds
    default "";
}

server {
    # …
    add_header Retry-After $retry_after always;   # "always": on errors too (1.7.5 or later)
    # Careful: if a location has its own add_header, it doesn't inherit this one. Repeat it there.
}

Whole site · inside the server { }

# Turned on by creating the file maintenance.flag in the root and turned off
# by deleting it: no config changes and no nginx reload.
set $maintenance 0;
if (-f $document_root/maintenance.flag) { set $maintenance 1; }
if ($remote_addr = 203.0.113.10)        { set $maintenance 0; }
if ($maintenance)                       { return 503; }

error_page 503 @maintenance;
location @maintenance {
    rewrite ^ /maintenance.html break;
}
# The maintenance page, with its CSS inlined: any separate resource
# (a stylesheet, a logo) would get the 503 too.

Just one directory · the location for /shop/

location /shop/ {
    set $maintenance 0;
    if (-f $document_root/maintenance.flag) { set $maintenance 1; }
    if ($remote_addr = 203.0.113.10)        { set $maintenance 0; }
    if ($maintenance)                       { return 503; }

    # … whatever you already had here (try_files, fastcgi_pass…)
}
# The error_page 503 and the header map from the first block already cover this.

429 from rate limiting · nginx does this on its own

# In http { }: one zone per IP, ten requests per second.
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;

# In the server { } or in the location you want to protect:
limit_req zone=perip burst=20 nodelay;
limit_req_status 429;   # by default nginx responds 503, which is a lie (1.3.15 or later)
# With the map from the first block, every 429 carries Retry-After: 10

Check that the header comes out only with the status it belongs to, and from an IP that isn't on your allowlist (your phone without Wi-Fi will do): curl -sI https://your-domain.com/ | grep -iE "^(HTTP|retry-after)"

Per-IP limit: 429 and `Retry-After` only for whoever crawls too much

A 503 for maintenance stops everyone equally. But excessive crawling almost always has a name: a scraper, an SEO tool in a hurry, an AI bot or someone pretending to be Googlebot. For that you don't need to touch anyone else: I count the requests from each IP and return 429 Too Many Requests only to the one that goes over its quota, with a Retry-After that tells it when it can come back.

The quota depends on who it is, and the user agent isn't enough to know that: anyone can write "Googlebot". For Googlebot and Bingbot I verify the IP, as in the IP range block above, and I give them more room, because Google doesn't slow down just that IP: when it sees enough 429s, it reduces crawling across the whole hostname, and it recovers on its own once they stop. Whoever claims to be Googlebot from an IP that isn't Google's counts as just another bot.

Person5 requests per second, burst of 60Retry-After: 12Any browser. Hard to reach, even when sharing an IP in an office or on a mobile network.
Verified search engine2 per second, burst of 120Retry-After: 60Googlebot or Bingbot from an IP that really is theirs. If it hits the cap, Google slows down across the whole host.
Bot1 every 2 seconds, burst of 60Retry-After: 120SEO tools, scrapers, AI bots… and the fake Googlebot.

The Retry-After comes from the same calculation in all three: how long the client takes to get its whole burst back, burst ÷ rate. After that time it can come back at full speed; before then, only at its own rate. Google accepts it with a 429 just as it does with a 503. And robots.txt stays out of the limit: if Google gets a 429 when requesting it, it stops crawling the whole site for the first 12 hours while it keeps retrying.

Apache and LiteSpeed. Neither of them has a per-IP limit that returns a 429 (LiteSpeed's slows clients down without telling them), so PHP does it, before WordPress gets to work. It needs nothing installed, because it keeps the counters in the temp folder, and it also works behind an nginx you can't touch. Whatever the page cache or the CDN serves doesn't go through PHP: it neither counts nor costs anything.

limit-per-ip.php · in wp-content/mu-plugins/ (create it if it doesn't exist): WordPress loads it on its own, without activating it

<?php
/**
 * Plugin Name: Per-IP limit (429 + Retry-After)
 * Description: 429 Too Many Requests for the IP that crawls too much, with a quota and a Retry-After based on who it is.
 */// In WordPress, in wp-content/mu-plugins/. On any other PHP site, with auto_prepend_file.

if ( PHP_SAPI === 'cli' || defined( 'DOING_CRON' ) ) {
    return;
}

// Per client: requests per second and burst. The same bucket as nginx's limit_req.
const RL_SOURCES = [
    'person'    => [ 5, 60 ],     // a browser never gets there: static files don't go through PHP
    'searchbot' => [ 2, 120 ],    // Googlebot or Bingbot with a verified IP
    'bot'       => [ 0.5, 60 ],   // everything else, including whoever claims to be Googlebot and isn't
];
// Retry-After = the time it takes to refill the burst: 12 s, 60 s and 120 s.

function rl_dir(): string {
    $dir = sys_get_temp_dir() . '/per-ip-limit-' . md5( __DIR__ );    // one folder per site
    if ( ! is_dir( $dir ) ) {
        @mkdir( $dir, 0700, true );
    }
    if ( mt_rand( 1, 1000 ) === 1 ) {    // now and then, remove whatever hasn't been touched for a day
        foreach ( glob( $dir . '/*' ) ?: [] as $old ) {
            if ( @filemtime( $old ) < time() - 86400 ) {
                @unlink( $old );
            }
        }
    }
    return $dir;
}

/* The client is the IP; for IPv6, its /64 network, which is what each home or server gets. */function rl_client( string $ip ): string {
    $bin = @inet_pton( $ip );
    return ( false !== $bin && strlen( $bin ) === 16 ) ? bin2hex( substr( $bin, 0, 8 ) ) . '/64' : $ip;
}

/* Is it really Googlebot or Bingbot? Reverse DNS to its domain and forward back to the same IP,
   as Google explains. The answer is cached for a day per IP. */function rl_verified_searchbot( string $ip, string $ua ): bool {
    if ( ! preg_match( '/googlebot|googleother|google-inspectiontool|bingbot/i', $ua ) ) {
        return false;
    }
    $cached = rl_dir() . '/dns-' . md5( $ip );
    if ( is_file( $cached ) && filemtime( $cached ) > time() - 86400 ) {
        return file_get_contents( $cached ) === '1';
    }
    $host = gethostbyaddr( $ip );
    $ok   = is_string( $host ) && preg_match( '/\.(googlebot\.com|search\.msn\.com)$/i', $host );
    if ( $ok ) {
        $ips = strpos( $ip, ':' ) !== false
            ? array_column( @dns_get_record( $host, DNS_AAAA ) ?: [], 'ipv6' )
            : ( gethostbynamel( $host ) ?: [] );
        $ok = in_array( inet_pton( $ip ), array_map( 'inet_pton', $ips ), true );
    }
    @file_put_contents( $cached, $ok ? '1' : '0' );
    return $ok;
}

function rl_source( string $ip, string $ua ): string {
    if ( rl_verified_searchbot( $ip, $ua ) ) {
        return 'searchbot';
    }
    $bots = '/bot|crawl|spider|slurp|scrapy|python|curl|wget|httpclient|java\/|go-http|headless/i';
    return ( '' === $ua || preg_match( $bots, $ua ) ) ? 'bot' : 'person';
}

function rl_apply(): void {
    // Logged-in WordPress users don't count (without setting the current user yet).
    if ( function_exists( 'wp_validate_auth_cookie' ) && wp_validate_auth_cookie( '', 'logged_in' ) ) {
        return;
    }
    // Never robots.txt: with a 429 there, Google stops crawling the whole site for up to 12 hours.
    if ( strpos( $_SERVER['REQUEST_URI'] ?? '', '/robots.txt' ) === 0 ) {
        return;
    }
    $ip = $_SERVER['REMOTE_ADDR'] ?? '';
    if ( '' === $ip ) {
        return;
    }
    $source           = rl_source( $ip, $_SERVER['HTTP_USER_AGENT'] ?? '' );
    [ $rate, $burst ] = RL_SOURCES[ $source ];

    // The client's bucket: each request spends a token; $rate tokens come back per second, up to $burst.
    $f = @fopen( rl_dir() . '/' . md5( $source . rl_client( $ip ) ), 'c+' );
    if ( ! $f ) {
        return;                        // without a temp folder, nobody is limited
    }
    flock( $f, LOCK_EX );
    $now     = microtime( true );
    $bucket  = explode( ' ', (string) stream_get_contents( $f ) );
    $before  = isset( $bucket[1] ) ? (float) $bucket[1] : $now;
    $tokens  = isset( $bucket[1] ) ? (float) $bucket[0] : $burst;
    $tokens  = min( $burst, $tokens + ( $now - $before ) * $rate );
    $allowed = $tokens >= 1;
    if ( $allowed ) {
        $tokens--;
    }
    ftruncate( $f, 0 );
    rewind( $f );
    fwrite( $f, $tokens . ' ' . $now );
    flock( $f, LOCK_UN );
    fclose( $f );
    if ( $allowed ) {
        return;
    }

    // A Retry-After that makes sense: the seconds it takes to get the whole burst back.
    $wait = (int) ceil( ( $burst - $tokens ) / $rate );
    http_response_code( 429 );
    header( 'Retry-After: ' . $wait );
    header( 'Cache-Control: no-store' );
    header( 'Content-Type: text/plain; charset=utf-8' );
    echo 'Too many requests. Try again in ' . $wait . ' seconds.';
    exit;
}

// In WordPress, once the session can be checked; in any other PHP, right away.
if ( function_exists( 'add_action' ) ) {
    add_action( 'plugins_loaded', 'rl_apply', 0 );
} else {
    rl_apply();
}

Outside WordPress · .htaccess in the root, with LiteSpeed Enterprise or with Apache and mod_php

# PHP runs the limiter before any other file on the site (absolute path).
<IfModule LiteSpeed>
  php_value auto_prepend_file /home/user/site/limit-per-ip.php
</IfModule>
<IfModule mod_php.c>
  php_value auto_prepend_file /home/user/site/limit-per-ip.php
</IfModule>

With PHP-FPM · .user.ini in the root, because there PHP doesn't read .htaccess

; PHP rereads this file every five minutes.
auto_prepend_file = /home/user/site/limit-per-ip.php

On OpenLiteSpeed neither works: the same php_value line goes in the WebAdmin, under Virtual Hosts → General → php.ini Override.

nginx. No PHP needed here: limit_req keeps each IP's bucket, and with one zone per source each one gets its own rate. The source comes from crossing the user agent with the IP lists that Google and Bing publish. If your hosting doesn't let you touch nginx, the PHP limiter in the other tab works just the same.

The zones and the header · in http { }

# The IPs of Google's and Bing's crawlers (the file is generated below).
geo $searchbot_ip {
    default 0;
    include /etc/nginx/searchbot-ips.conf;
}

# The source: searchbot if the user agent and the IP match; bot if the user agent
# gives it away, including a Googlebot that doesn't come from a Google IP; person, the rest.
map "$searchbot_ip:$http_user_agent" $source {
    default                                                          person;
    "~*^1:.*(googlebot|googleother|google-inspectiontool|bingbot)"   searchbot;
    "~*(bot|crawl|spider|slurp|scrapy|python|curl|wget|httpclient|java/|go-http|headless)"  bot;
}

# robots.txt is never limited: with a 429 there, Google stops crawling the whole
# site while it retries (for up to 12 hours).
map $request_uri $exempt { "~^/robots\.txt" 1; default 0; }

# One key per zone, empty if the request comes from another source or is exempt: it doesn't count.
map "$exempt:$source" $key_person    { "0:person"    $binary_remote_addr; default ""; }
map "$exempt:$source" $key_searchbot { "0:searchbot" $binary_remote_addr; default ""; }
map "$exempt:$source" $key_bot       { "0:bot"       $binary_remote_addr; default ""; }

limit_req_zone $key_person    zone=people:10m     rate=5r/s;
limit_req_zone $key_searchbot zone=searchbots:10m rate=2r/s;
limit_req_zone $key_bot       zone=bots:10m       rate=30r/m;
limit_req_status 429;

# Retry-After by status and source: the time each bucket takes to empty
# (burst ÷ rate). It replaces the map $status from the previous block.
map "$status:$source" $retry_after {
    "~^503:"         3600;
    "429:person"     12;
    "429:searchbot"  60;
    "429:bot"        120;
    default          "";
}

Where it applies · in the PHP location

# In the PHP location, not the whole server: every page pulls in dozens of
# static files, and those shouldn't count.
location ~ \.php$ {
    limit_req zone=people     burst=60  nodelay;
    limit_req zone=searchbots burst=120 nodelay;
    limit_req zone=bots       burst=60  nodelay;

    # … whatever you already had here (include fastcgi_params, fastcgi_pass…)
}
# The header is still set by the add_header Retry-After $retry_after always
# in the server { }. If this location has its own add_header lines, repeat it here.

The search engines' IP list · /etc/cron.daily/searchbot-ips

#!/bin/sh
# /etc/cron.daily/searchbot-ips (with chmod +x): downloads the IP lists that Google
# and Bing publish, turns them into geo format and reloads nginx. Needs curl and jq.
# Run it once by hand before adding the geo block: without the file, nginx won't start.
set -e
T=/tmp/searchbot-ips
mkdir -p "$T"
curl -fsS -o "$T/google.json" https://developers.google.com/static/crawling/ipranges/common-crawlers.json
curl -fsS -o "$T/bing.json"   https://www.bing.com/toolbox/bingbot.json
jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " 1;"' "$T/google.json" "$T/bing.json" > "$T/conf"
test -s "$T/conf"
cp "$T/conf" /etc/nginx/searchbot-ips.conf
nginx -t -q && nginx -s reload

To check it · a burst with a bot user agent; each request carries a different parameter to get past the cache

for i in $(seq 100); do
  echo "$i $(curl -s -o /dev/null -A 'test-bot' -w '%{http_code} %header{retry-after}' "https://your-domain.com/?limit=$i")"
done | grep -m1 ' 429 '
# 62 429 120   ← the first 429 response, with its Retry-After (curl 7.84 or later)
# It doesn't affect you while browsing: the bot's bucket and your browser's are separate.

Two warnings before you switch it on. If the site sits behind Cloudflare, Bunny or any other proxy, the IP the server sees is the proxy's, and with it everyone would share the same quota: first make the server see the visitor's IP (set_real_ip_from and real_ip_header in nginx, mod_remoteip in Apache, "Use Client IP in Header" in LiteSpeed). And if Googlebot gets the 429 every day, it's no longer a peak: raise its quota or fix the load, because Google warns that errors lasting more than one or two days can end up removing URLs from the index.

If you like this article, you would help me a lot by sharing my content: