Blog

Crawl budget is probably not your problem.

It is the most satisfying explanation available when pages will not index, because it is technical, external and implies a clever fix. Google has been unusually direct about this: most sites do not need to think about crawl budget at all, and for a site of a few thousand URLs it is essentially never the reason a page is missing. Reaching for it first usually means the actual cause goes uninvestigated.

Most sites are unaffectedCheck the logs, not the theorySprawl is the real version

What it actually is

Crawl budget is shorthand for two things Google balances. How much your server can take without struggling, which it works out by watching response times and errors and backing off when the site slows. And how much Google wants to crawl you, which follows from whether the URLs it fetches turn out to be worth having.

Note what the second half means. Crawl demand is largely an output of quality and structure, not a fixed allowance you are issued. A site producing thousands of near-identical URLs teaches Google that fetching more of them is not worth much, and the crawl rate reflects that judgement rather than causing it.

The rough threshold

Google's own guidance puts it at roughly this: if your site has under a few thousand URLs, it will generally be crawled efficiently and crawl budget is not something to manage. Above that, and particularly above around a hundred thousand URLs or with rapidly changing inventory, it becomes a genuine consideration. Most sites worrying about this are firmly in the first group.

How to tell instead of guessing

Server logs settle it, and nothing else does. Filter to verified Googlebot requests over a few weeks and ask two questions. Is Google fetching your important pages at a sensible frequency? And where are the requests actually going?

The Crawl stats report in Search Console gives a coarser version of the same picture without log access. What you are looking for is a specific pattern: a large share of requests landing on parameter URLs, filter combinations, internal search results, session variants or paginated tails, while your money pages get fetched rarely. That is a crawl problem with evidence behind it. A page not being indexed on a 200-page brochure site is not.

When it genuinely is the problem

Large ecommerce with faceted navigation is the classic case, where colour times size times brand times sort order generates a combinatorial explosion of crawlable URLs from a modest catalogue. That is the same arithmetic that makes programmatic pages dangerous when nobody prunes them.

Then large publishers and directories with deep archives and pagination, sites with genuinely fast-changing inventory where recrawl speed is commercially material, and enterprise sites where a template decision multiplies across hundreds of thousands of pages. Also, quietly, any site slow enough that Google throttles itself: performance work can increase crawl rate simply by removing the reason to back off.

What to do if it is

The work is reduction, not persuasion. Stop generating URLs that should not exist, which usually means controlling faceted navigation at the source rather than papering over it with canonicals that still get crawled. Return honest status codes so dead URLs leave the crawl set. Keep internal links pointed at canonical URLs so you are not routing crawlers through redirects. Keep sitemaps to real, indexable URLs with honest lastmod values.

Notice that all of that is ordinary architecture work. There is no crawl budget setting to adjust, which is part of why the diagnosis is so attractive and so rarely actionable in the way people hope.

What it usually is instead

If a page is not indexed, the status in Search Console tells you which conversation you are actually having. Discovered, currently not indexed means Google knows about the URL and has not judged it worth fetching, which on a small site is about internal linking and perceived value rather than an allowance running out. Crawled, currently not indexed means it read the page and declined it, which is a content judgement and has nothing to do with crawling at all.

Both of those are harder to hear than "we have run out of crawl budget", which is exactly why the myth is durable. It relocates a quality problem into an infrastructure one.

Book me

Told your crawl budget is the issue?

Send the crawl stats report or the server logs. You get an evidence-based answer on where Googlebot is actually spending its requests, and whether that is genuinely what is holding your pages back.

Reply within one working day No obligation Your details stay with us

Takes about 60 seconds. No newsletter and no CRM sequence. Your details are used to reply to this enquiry and nothing else. See the privacy notice.