Crawl budget for small sites: what actually matters

Most crawl budget writing is aimed at enterprise sites with millions of URLs. If you run a few hundred pages, almost none of it applies — but the underlying idea still shapes how quickly your changes get noticed.

What crawl budget actually is

Two things, often conflated:

• Crawl rate limit — how hard a crawler is willing to hit your server without degrading it. Slow responses and 5xx errors push this down.
• Crawl demand — how much the engine wants to crawl you, based on how often your content changes, how popular your URLs are, and how much it trusts the site.

For a small site the rate limit is rarely the constraint. Demand is. And demand is precisely what a change notification protocol addresses.

Why small sites get crawled infrequently

An engine learns your update cadence and allocates accordingly. If it visits five times and finds nothing new each time, it comes back less often. Publish once a month and you may be crawled roughly that often — which means a new page could sit undiscovered for weeks.

This is the trap: infrequent updating causes infrequent crawling, which makes your next update slower to be found, regardless of how important it is.

Things that genuinely help

1. Tell them instead of waiting

This is the direct fix, and the only one that changes the discovery timeline rather than nibbling at efficiency. For Bing, Yandex, Seznam, Naver, Yep and Amazon, IndexNow does it. For Google, a well-maintained sitemap with accurate lastmod values is the supported route.

2. Accurate lastmod

Worth stating plainly: lastmod is only useful if it is true. A CMS that rewrites every lastmod to today on each rebuild is actively harmful — the engine learns your dates are meaningless and stops using them. Many CMSes do this by default. Check yours.

3. Stop wasting crawls on pages you do not want indexed

The classic offenders on small sites:

• Faceted navigation generating combinatorial URLs
• Internal search result pages
• Tag and date archives with one item each
• Session IDs or tracking parameters in URLs
• Paginated sequences that go on forever

Block them in robots.txt, canonicalise them, or stop generating them. Every fetch spent on a tag archive is one not spent on your new product page.

4. Fix redirect chains and soft 404s

Each hop in a redirect chain is a separate fetch. Soft 404s — pages returning 200 with "not found" content — are worse: they get crawled repeatedly and pollute your index with nothing pages.

5. Make the server fast

Response time affects the rate limit directly. This is one of the few areas where the enterprise advice transfers unchanged: a consistently fast server gets crawled more.

Things that do not help

• Resubmitting unchanged URLs. It signals nothing and can get you rate limited.
• Pinging services. Most of the old blog-ping infrastructure is defunct.
• Buying crawl. Not a thing, whatever anyone tells you.
• Publishing filler to seem "active". Thin pages consume crawl and lower site-level quality signals. This makes things worse in both directions.

The realistic goal

You are not optimising a crawl budget in the enterprise sense. You are shortening the gap between "I published this" and "a search engine knows it exists" — from weeks to minutes, for the engines that support being told. That is the achievable win, and it is a real one.