Multi-Domain IndexNow Workflow

Multi-Domain IndexNow Workflow

Each domain or subdomain that sends IndexNow submissions needs its own verification key file placed at the root of that specific host. IndexNow key files are domain-scoped¹. A key file at example.com does not authorize submissions from blog.example.com or shop.example.com; each subdomain requires a separate key file, and each can use either its own unique key or the same key value shared across all properties under the same operator.

Managing multiple domains efficiently requires either a centralized submission pipeline or a per-domain CMS plugin setup. Agencies and digital teams managing 5-20 sites often benefit from a shared script or service that handles key rotation, submission batching, and error logging across all domains from a single interface. Conversely, for teams managing 2-3 sites, per-domain plugin setup is simpler and introduces less operational complexity than a shared pipeline.

Key file requirements per domain

The IndexNow protocol requires a verification text file named with the key value (e.g. abc123.txt) to be placed at the exact root of the submitting host, and this file must be publicly accessible via HTTPS for the validation check to succeed¹. For example.com submissions, the file must be at https://example.com/abc123.txt. For blog.example.com submissions, the file must be at https://blog.example.com/abc123.txt. The file content must contain only the key value as plain text².

Per-domain verification files

Multiple domains may share the same key value; the shared key is called an API key and identifies the submitter operator. Using the same key across domains simplifies management because the key does not need to be rotated independently per domain, and a single key value can be deployed to dozens of domain roots without any additional configuration overhead. Each domain still requires its own key file even if the key value is identical. The key file must be placed at the exact root of each submitting host, which means a key file at example.com does not cover fr.example.com or blog.example.com, and forgetting to deploy the file to every subdomain before sending submissions is one of the most common multi-domain configuration errors that agencies encounter when onboarding a new client site.

Centralized vs per-domain submission setup

A centralized submission pipeline uses a single script or service that receives publish events from all managed CMSes (via webhooks, API calls, or a message queue) and routes each URL to the IndexNow endpoint with the appropriate domain key. This approach gives a team a single submission log, unified retry logic, and centralized rate-limit management. Building on this, a centralized service can batch URLs from multiple domains into time-window buckets, reducing the number of API calls during high-publish periods.

Per-domain plugin setup installs an IndexNow CMS plugin (WordPress, Drupal, Joomla) on each site individually. Each plugin manages its own key, submission log, and event hooks. This approach is simpler to set up but harder to audit across many domains, because there is no single dashboard showing submission status for all client sites, and a team managing 15 domains would need to log into 15 separate plugin settings pages to verify that every site is actively submitting and that no plugin has been deactivated by a CMS update or a well-meaning editor.

Rate-limit management across domains

IndexNow rate limits apply per host, not per operator. Submitting 600 URLs per minute (approximate) from example.com and 600 from blog.example.com simultaneously does not combine into a single rate limit; each domain gets its own rate-limit ceiling³. Consequently, centralized pipelines that fan out to multiple domains can submit higher total volumes per minute than a single-domain setup.

Yet each participating engine applies its own crawl budget per domain regardless of the IndexNow rate limit. Submitting 10,000 URLs per day across 10 domains at 1,000 each keeps each domain inside Bing's daily crawl budget⁴. Concentrating all 10,000 submissions on a single domain may exhaust that domain's budget while leaving other domains underserved, which is why a well-designed centralized pipeline tracks per-domain submission volumes and distributes the daily allocation evenly rather than allowing a single high-traffic site to consume the entire pool.

Shared key values across domains and operator identity in IndexNow

Using the same key value across all your domains simplifies multi-domain key management. The IndexNow protocol treats the key as an operator identifier⁵; a single key value can appear in key files at multiple domain roots simultaneously, authorizing submissions from all of them under the same operator identity. Key rotation then becomes a single update operation: use one IndexNow key across domains by generating a new key, deploying its file everywhere, and updating the value in your submission tooling.

Each domain's key file must contain only the key value as plain text, with no surrounding whitespace, HTML, or formatting². Mismatches between the key value in the submission request and the key file content at the submitted URL's domain root cause silent rejection. Validating each key file with curl https://[domain]/[key].txt after deployment confirms the content is correct before any submissions are sent.

Key file content requirements and common format mistakes

The most frequent multi-domain key file mistake is adding a newline or BOM (byte-order mark) to the file content, which causes validation failures even when the key value appears correct in a text editor. Generate the file using a simple echo command or a script that writes the key as raw bytes, and verify the byte count matches the key length exactly. A 32-character key should produce a 32-byte file; any larger file indicates extra characters that will cause rejection.

Cloudflare Workers as a centralized multi-domain dispatch layer

Cloudflare Workers provide a free-tier serverless environment that works well as a centralized IndexNow submission service for multi-domain teams. A single Worker can receive publish events from multiple CMSes via HTTP POST, extract the submitted URL and domain, look up the correct key from Workers KV (Cloudflare's key-value store), and dispatch the IndexNow request with the correct key for that domain.

Workers KV stores per-domain key mappings as simple key-value pairs: the domain name is the key, the IndexNow API key is the value. The Worker reads the submitted URL's hostname, queries KV for the matching key, and assembles the IndexNow request body. This architecture allows you to add a new domain by adding one KV entry and deploying its key file, without modifying the Worker code.

Rate-limit enforcement in a multi-domain Worker

The Worker should enforce per-domain rate limits independently. Building a queue using Cloudflare Queues (available on paid Workers plans) or a time-window counter in KV prevents a burst of submissions from one domain from affecting another domain's submission rate. For teams on the free Workers plan, a simpler approach batches all events within a 10-second window per domain and flushes each batch as a single IndexNow POST, staying comfortably under the 600-URL-per-minute ceiling for each host.

Add a per-domain failure alert to the Worker so one client's key mismatch or 4xx response does not get lost in an aggregated log. Route the alert to include the domain name and the failing URL, because a silent rejection on a single domain can persist for weeks if no one is watching the consolidated output. The Worker can also cache the last successful submission timestamp per domain and surface a warning when a domain goes quiet longer than its normal publish cadence, turning a passive submission service into an active monitoring layer.

When to use this

Use a centralized IndexNow pipeline when managing 5 or more domains where operational consistency and logging matter. Use per-domain CMS plugins for 2-4 sites where simplicity is preferred. Ensure each domain has its own key file at the domain root before any submissions fire.

Examples

Each CMS plugin submits independently; no centralized log; rate limits managed per-site

Publish events from all sites route to a central Worker or server that batches by domain, enforces rate limits, logs responses, and alerts on `4xx` failures.