Google’s newly updated crawl budget optimization guide has been updated to be more precise, with emphasis given to the different parts of crawling in order to provide greater detail that is more precise than the previous version of the documentation. Aside from improvements in precision and understandability, Google adds an entirely new section.
The previous version of the documentation had the feel of a first draft that was corrected for spelling and punctuation but was not corrected for how it would be understood by the readers, something that the updated documentation addresses.
What I would suggest is that, aside from the value in reading Google’s explainer, there is value in understanding Google’s editorial decisions because they show how a document can be improved with three kinds of precision.
- From general to specific: The revision replaces general descriptions with more specific ones.
- Replaces superfluous information: Google replaces superfluous descriptions of “why” their crawler does something with descriptions of the effect, which is more useful.
- Sentence construction: Sentences are rewritten with a more logical progression of ideas.
That last one, sentence construction, is important because it affects how well humans (and AI) understand content.
From General To Specific
The revised documentation replaces general descriptions with specific descriptions.
Before: “responds quickly for a while”
After: “responds consistently and its response times (including latency and Time-to-First Byte) remain stable or improve”
Before: “server errors”
After: “5xx HTTP status codes or HTTP 429”
Before: “every available URL”
After: “every publicly accessible URL”
Before: “increase your budget”
After: “increase your crawl budget”
Before: “serving limit”
After: “crawl capacity limit”
Those are just some examples out of many in which the new documentation is more precise. This contributes to it becoming easier to read, precise, more meaningful and helpful.
Replaces Superfluous Information
The updated version replaces descriptions of why Google does something with descriptions of the effect and what it does. This is interesting because the “why” way of writing did not provide useful information; it just got in the way. This might not always be the case at all times but for this specific context it made a difference.
In the following example, the before and after come after this sentence:
“Use the appropriate tools to tell Google which pages to crawl and which not to crawl. If Google spends too much time crawling URLs that it shouldn’t,…”
Before: “Google’s crawlers might decide that it’s not worth the time to look at the rest of your site.”
After: “Google’s crawlers might not explore the rest of your site.”
The second version is shorter and shows the effect. The first part of the section says, “If this happens” and the second part completes it by saying, “then this will happen.” There’s a harmony between stating a cause and then the effect. The cool part about the revised version is it says more with fewer words.
Here’s another instance of where Google removes a “why” and replaces it with a concrete what:
Before: “This is calculated to provide coverage of all your important content…”
After: “This ensures Google can cover all your important content…”
The before part is wishy-washy and indirect, needlessly explaining the why (calculated to provide) and replacing it with the more direct, “ensures Google can cover.”
Google replaced general statements and concepts with the exact effects that are directly relevant to the reader.
Sentence Construction
Circling back to the “before” version (“Google’s crawlers might decide…”), that part about the crawlers making a decision needlessly anthropomorphizes Google’s crawlers. Anthropomorphize means to ascribe human-like motivations or actions to something that does not have that kind of agency.
Internally at Google it probably makes sense to say that crawlers make a decision because that’s probably how they talk about Googlebot. It may have origins in how some algorithms use a decision tree in which a branching series of yes/no questions guides it to taking steps.
But that wording is confusing and may take a fraction of a second for it to process. Words and phrases that trigger a reading pause are what I call a comprehension road bumps.
When editing or improving your articles, look for those road bumps and remove them to make the page easier to read and understand.
These changes affect how sentences are constructed so that each sentence doesn’t introduce multiple and clashing ideas. This affects readability and comprehension.
Have you ever read a sentence, reached the end of it, then realized you have no idea of what you just read? That’s what happens with poorly written sentences that introduce multiple ideas that have little to no relation to each other. Those multiple ideas are the road bumps that interrupt the flow of information to the reader’s mind.
AI natural language algorithms were written to approximate the human comprehension of written words. So it’s natural that if you make content that’s easily understood by humans, it will also be understood by AI. This is the secret to my approach to writing.
There’s an SEO idea that recommends writing short sentences and paragraphs. That’s a superficial approach to writing that misses the point. It’s not about writing short sentences and paragraphs; it’s about improving readability and comprehension. I will show two examples of how Google fixed this.
Example 1: Clashing Ideas In A Sentence
The following example shows how the original version introduced two clashing ideas:
“As a result, there are limits to how much time Google’s crawlers can spend crawling any single site, where a site is defined by the hostname.”
The first part talks about limits to Google’s crawling. The second part abruptly interrupts that idea with a definition of what a site is. The last part forces the reader to shift their mental attention from limits to Google’s crawling to a site being defined as a hostname.
An additional problem with that sentence is the word hostname, which will be mumbo-jumbo to the average SEO and site owner. It’s not a word that Google’s average reader uses and understands. The consequence is that Google loses the reader’s attention and comprehension.
The corrected version of that sentence looks like this:
“As a result, there are limits to how much time and resources Google can devote to crawling any single site.”
It’s immediately apparent why that sentence is better.
Example Two: More Clashing Ideas In A Sentence
In this example, Google starts the paragraph with an easy-to-understand sentence that introduces what the paragraph is about:
“Google wants to crawl your site without overwhelming your servers.”
The original version of the documentation follows that opening sentence with a virtual car crash of concepts:
“To prevent this, Google’s crawlers calculate a crawl capacity limit, which is the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches. “
That sentence is grammatically correct. But it’s incorrect when viewed from the perspective of clarity.
- The phrase, “simultaneous parallel” is redundant.
- The word “parallel” also introduces a roadblock to comprehending the sentence because it’s not something the average site owner would immediately understand. Parallel means two things next to each other, right? But the word “simultaneous” means two or more things happening at the same time. That’s why using the words “simultaneous parallel” together does nothing but work against the goals communicating an idea. Comprehension suffers because it makes a reader pause for a second or ignore that phrase altogether. It’s a reading road bump.
- The sentence is defining “Crawl capacity limit” as both a number (max connections) and a time delay. This introduces another reading road bump.
This is the before again:
“To prevent this, Google’s crawlers calculate a crawl capacity limit, which is the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches. “
This is what that sentence looks like after revision:
“To prevent this, Google’s crawlers calculate a crawl capacity limit (also known as hostload).”
The rewrite of that section removes the clashing ideas within the one sentence and is easier to read and understand.
An argument could be made for removing “(also known as hostload)” but there’s a reason for adding that into that section. The word “hostload” is used later on without an explanation or context. The purpose for defining the word “hostload” here is so that the reader understands what the word means when it is used later on.
If it were my decision, I would remove the word “hostload” altogether because all a reader needs to know is “crawl capacity limit.” There is no reason to force the reader to learn Google’s internal jargon. Nevertheless, Google’s motivation for introducing a definition of that word is sound.
Takeaways
- Google improved their crawl budget guide by replacing general descriptions with more precise explanations and guidance.
- The revised documentation removes unnecessary explanations and focuses on details that matter to the reader.
- The update improves sentence construction by removing or incongruent ideas.
- Several revisions remove what I call “reading road bumps” such as unnecessary anthropomorphism, and redundancy.
- The guidelines now reads less like a first draft of a guideline and more like a purposeful and helpful document.
Featured Image by Shutterstock/One Artist