Google updated its crawl budget documentation with new information about how Google’s different crawlers all share a website’s crawl capacity and added a recommendation to use a 304 server response to reduce the amount of server resources used by crawlers.
Both of these additions are to Google’s Optimize Your Crawl Budget Documentation that is for enterprise websites with over a million pages and medium sized websites of over 10,000 web pages that rapidly change. Although the documentation is not aimed at every website, the information on that page is useful for understanding how crawling works.
Google’s Crawlers Share A Website’s Crawl Capacity
Google added the following passage:
“While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others.”
This means that if a site gets crawled by Googlebot-Image and Googlebot, both of the crawlers are sharing the crawl budget for the website. The impact is that a high rate of crawling by one bot can reduce the capacity available to other Google crawlers.
Google Recommends 304 Server Response
The other interesting recommendation that Google added was the suggestion to use 304 Not Modified server response code. According to the Mozilla Developer Network web documentation:
“The HTTP 304 Not Modified redirection response status code indicates that there is no need to retransmit the requested resources.”
The thing about the 304 Not Modified response code is that there is no redirection happening, even though technically all 3xx server response codes are classed as redirection codes. Despite that technical detail, nothing is being redirected.
Google’s updated documentation says:
“Use HTTP caching: Support 304 (Not Modified) HTTP status codes. If a page hasn’t changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources.”
A 304 Not Modified response tells Google’s crawlers that a page has not changed since the last time it was crawled. The result is that the server doesn’t serve the page to Googlebot and Googlebot can go focus on indexing other pages.
Read Google’s updated documentation here. The two changes described here are the only new additions. There are other changes that here and there that make Google’s documentation more precise, which is documented here.