Automating IndexNow: what you actually have to build

Sending a URL to IndexNow is a single HTTP request. That is why so many integrations start as a ten-line function and quietly stop working three months later. Here is the full list of what a durable integration has to deal with — useful whether you build it or buy it.

1. Deciding what changed

The core problem, and the one most implementations get wrong.

• Sitemap lastmod comparison is the cheapest signal, but only counts if the date moved forward. Many CMSes rewrite every date on every build; naively trusting that resubmits your whole site daily.
• Content fingerprinting is the fallback when lastmod is absent or untrustworthy. You must normalise before hashing — strip CSRF tokens, asset hashes, timestamps, view counters and rotating recommendation blocks — or every page looks changed on every fetch.
• Application events are the most precise, but only capture changes that flow through your application. Bulk imports, supplier feeds and direct database edits bypass them entirely.

In practice you want a scheduled sitemap comparison as the safety net and events as the fast path.

2. Host discipline

Every URL in a batch must be on the host named in the payload. One absolute URL pointing at a CDN, a subdomain or the opposite www variant rejects the whole batch with a 422. Sitemaps mixing hosts are common, so filtering is not optional.

3. Batching

10,000 URLs maximum per request; more returns 422. Chunk, and handle a partial failure part-way through a large set without either losing the remainder or resubmitting the ones that already succeeded.

4. Rate limiting and back-off

• Enforce a minimum interval between resubmissions of the same URL — the guidance is at least five minutes.
• Stop on a 429 rather than continuing through remaining chunks.
• Honour Retry-After when present; use a sensible default when it is not.
• Retry transient network failures with exponential backoff, and distinguish them from permanent rejections. A timeout is not a 403.

5. Key file monitoring

The key file goes missing more often than you would expect — a redesign, a deploy that rebuilds the document root, a migration to a new host. When it goes, every submission fails with 403 and, if nothing is watching, silently. Re-verify periodically and alert when it breaks.

6. Logging that answers questions

Minimum viable log: per batch, the endpoint, HTTP status, raw response body, timestamp, duration and the list of URLs it carried; per URL, when it was last submitted, how many times, and the last error.

Without this you cannot answer "did you actually submit my new product page, and what did Bing say?" — which is the only question anyone ever asks.

7. Scheduling and concurrency

Per-domain frequency, jobs that do not pile up when a crawl runs long, and a cap on how much work one domain can consume. A sitemap index pointing at fifty sitemaps of 50,000 URLs each will exhaust a naive worker.

8. Crawl guard rails

Depth limits on nested sitemap indexes, a cap on sitemaps read per run, a cap on total URLs, timeouts on every fetch, and gzip handling for .xml.gz sitemaps. Also protection against a sitemap that points at itself, which happens.

Rough effort

A competent developer gets a working version in a day. Getting it to the point where it is still correct a year later, across sites with different sitemap quirks, is more like a week or two — plus ongoing maintenance when something changes.

Whether that is worth building depends entirely on how many domains you run and what your time costs. For one site you control and understand, building it is reasonable. For a dozen client sites, it is a maintenance liability.

Index-now is this list, run as a service, for $12 per domain per year. Either way, now you know what "IndexNow integration" actually means.