AI crawler readiness · How-to

IndexNow setup — push, not poll.

IndexNow is the search-side equivalent of webhooks. Instead of waiting for engines to re-crawl your site on their schedule, you push them a notification the instant a URL changes. Bing, Copilot and Yandex consume it directly; through Bing's distribution layer, the same ping reaches Yahoo, DuckDuckGo, and parts of ChatGPT's web tool. Setup is a one-time 30-minute job.

Companion to AI crawler readiness. Updated 2026-05-07.

Why IndexNow matters even though Google does not consume it

Google announced in 2021 that it would test IndexNow and has not adopted it. Search-side, that's a real loss — Google's index is bigger than Bing's. AI-side, the picture is different. Bing's index feeds Microsoft Copilot directly, and a substantial fraction of ChatGPT's web tool calls hit Bing as the upstream search provider. Pages that Bing has indexed quickly, with current content, are pages those AI surfaces can cite. Pages that Bing hasn't indexed yet are pages those surfaces cannot cite.

For sites that publish frequently — pricing changes, new product launches, breaking research, news — the gap between content edit and Bing index can be days without IndexNow and minutes with it. The compound effect across hundreds of edits per year is a meaningful difference in surface area for AI citation.

What IndexNow is not. It is not a substitute for sitemap.xml — engines still use the sitemap as the canonical URL list. It is not a ranking lever — pinging a URL does not boost it; it just gets it indexed faster. And it is not a way to get a page indexed that engines have decided not to index for quality reasons. Treat it as a freshness signal, not a discovery or a ranking signal.

Setup: five steps, ~30 minutes

Step 1: generate a key

IndexNow keys are 8–128 character strings made of [a-zA-Z0-9-]. Generate one with any secure random source. Bing Webmaster Tools will generate one for you, or you can roll your own:

# macOS / Linux
openssl rand -hex 16
# 8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b

# Or via Python
python3 -c "import secrets; print(secrets.token_hex(16))"

Save the key somewhere durable. You will need it on every IndexNow ping. It identifies your site's authority to push URLs.

Step 2: deploy the key file

Place a file at https://www.example.com/<key>.txt whose body is exactly the key as plain text. For example, if your key is 8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b, deploy:

# File path on the host root
8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b.txt

# File contents (just the key, no surrounding whitespace)
8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b

The file must return HTTP 200 with content-type text/plain. The IndexNow service fetches this file before accepting any push to verify ownership of the host.

On S3:

echo -n "8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b" > key.txt
aws s3 cp key.txt s3://www.example.com/8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b.txt \
  --content-type "text/plain; charset=utf-8" \
  --cache-control "no-cache"

Verify: curl -s https://www.example.com/8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b.txt should return the key string and nothing else.

Step 3: register your site in Bing Webmaster Tools

Open bing.com/webmasters/, sign in with a Microsoft account, and add the site. Verify ownership via XML file, DNS TXT, or meta tag — whichever is easiest in your stack. Once verified, navigate to Settings → API access → IndexNow. Bing will associate the key file you deployed in Step 2 with your verified property.

Step 4: ping the API on every URL change

The IndexNow API accepts both single-URL GETs and multi-URL POSTs.

Single URL (GET):

curl "https://api.indexnow.org/IndexNow?url=https://www.example.com/foo/&key=8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b"

Multiple URLs (POST):

curl -X POST "https://api.indexnow.org/IndexNow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "www.example.com",
    "key": "8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b",
    "keyLocation": "https://www.example.com/8e3f9a2b4c1d5e7f0a3b6c2d4e8f1a9b.txt",
    "urlList": [
      "https://www.example.com/foo/",
      "https://www.example.com/bar/",
      "https://www.example.com/baz/"
    ]
  }'

A 200 or 202 response means accepted. 400 / 403 means the key file isn't reachable or doesn't match. 422 means a URL in the list isn't on the verified host.

Step 5: integrate with your publishing flow

The integration point depends on your stack:

The cost is one HTTP request per content change. Rate limits are generous — Microsoft cites "up to 10,000 URLs in a single submission" and reasonable submission frequency without hard caps for legitimate use.

Which engines actually consume an IndexNow ping

EngineConsumes IndexNow?Notes
BingYesDirect consumer; fastest update path to Bing's index.
Microsoft CopilotYes (via Bing)Copilot's web grounding uses Bing's index.
ChatGPT web toolPartially (via Bing)OpenAI's web tool uses Bing as one upstream provider; faster Bing indexing → faster ChatGPT availability.
YandexYesDirect consumer; matters mainly for RU-language audiences.
DuckDuckGoYes (via Bing)DDG sources from Bing's index.
Yahoo SearchYes (via Bing)Same.
NaverYesDirect consumer; matters for KR-language audiences.
Seznam (CZ)YesDirect consumer.
GoogleNoGoogle has not adopted IndexNow. Use sitemap submission + GSC URL inspection for Google.
BaiduNo (separate protocol)Baidu has its own push API with similar semantics. See China AI visibility for the Mainland-CN engines.
DeepSeek / Qwen / DoubaoNo direct supportMainland Chinese AI engines do not currently consume IndexNow. Their indexing pipeline uses Baidu's protocol or direct crawl.

Gotchas worth knowing before you ship

Key file content-type

The key file must be text/plain. Several teams deploy it as application/octet-stream from S3 (the default if you don't set --content-type) and IndexNow rejects the verification. Set the content type explicitly on upload.

One key per host

A key is scoped to a host. www.example.com and blog.example.com can share a key only if you deploy the key file on both hosts. Subdomain coverage is per-deployment, not inherited.

HTTPS required

IndexNow rejects pings for HTTP URLs. The key file must also be served over HTTPS. If your site is mid-migration to HTTPS, complete the migration first.

Don't ping unchanged URLs

The protocol explicitly asks publishers to push only on real content changes. Pinging the same URLs daily without changes wastes the engine's budget and can be deprioritised. Diff against the last submitted state and push only the deltas.

Sitemap is still required

IndexNow is a freshness signal, not a discovery signal. Engines still need sitemap.xml to discover URLs initially and to maintain canonical inventory. Don't drop the sitemap after enabling IndexNow.

Use this with the rest of the readiness layer

IndexNow is the change-notification layer of AI crawler readiness. It pairs with sitemap.xml for discovery, robots.txt for gating, llms.txt for AI-specific indexing, and Markdown alternates for clean extraction. Read the parent reference for the full configuration. If you're not seeing freshness pickup after enabling IndexNow, run the blocking-mistakes diagnostic.

Run an AI visibility audit →