IndexNow URL Submitter Reference

What Is IndexNow?

When a URL changes, IndexNow gives search engines a direct signal before their next scheduled crawl. With one HTTP POST to api.indexnow.org, you can notify participating engines about new, updated, and deleted URLs at the same time: Bing, Yandex, Naver, Seznam.cz, and Yep¹.

Before IndexNow, the only way to accelerate indexing for non-Google engines was to submit a sitemap and wait for each engine's crawl schedule. IndexNow replaces that wait with a push model: the moment you publish or update a page, you send an immediate signal, and the engine prioritizes a crawl within minutes rather than hours or days¹. Consequently, IndexNow is especially valuable for content types where freshness determines ranking position, such as news articles, product pages, and time-sensitive event content.

What is IndexNow?

How IndexNow works

To submit via IndexNow, you need two components: a verification key file at your domain root and an HTTP POST request to the shared endpoint. The key file is a plain text file named with your key value (e.g. abc123.txt) placed at the root of the submitting domain. The file contains only the key value and confirms that you control the domain.

Your submission POST contains the host domain, the key value, the key file location, and a list of URLs. The endpoint at api.indexnow.org accepts the POST and distributes the notification to all participating engines³. Each engine independently crawls the submitted URLs on its own schedule, but your IndexNow submissions enter a priority queue ahead of standard crawl traffic.

Which search engines participate

As of mid-2026, confirmed IndexNow participants are Bing, Yandex, Naver, Seznam.cz, and Yep⁴. The shared endpoint at api.indexnow.org is operated by Microsoft and distributes submissions to all participants from a single call³. Furthermore, each participating engine processes submissions independently; one engine accepting the notification does not guarantee that all others will index the URL at the same speed.

What the shared endpoint confirms

The shared endpoint confirms that your notification entered the IndexNow distribution route. It does not prove that every participant has crawled or indexed the URL yet, because each engine still applies its own crawl scheduler and quality checks, and the time between submission and actual crawl varies by engine, domain authority, and current crawl queue depth.

Google does not participate in IndexNow. For Google, continue using Google Search Console's URL Inspection tool and XML sitemap submission as the indexing path. This means that a complete indexing strategy in 2026 requires maintaining both IndexNow for Bing and other participating engines and Google Search Console for Google, because no single protocol or tool covers all major search engines simultaneously.

IndexNow and Google

Google has not joined the IndexNow protocol as of mid-2026. Industry analysts note that IndexNow creates an incentive structure that favors large publishers who can afford frequent real-time submissions, while Google's crawl algorithms already prioritize freshness for quality content organically. Regardless, the practical consequence for publishers is that IndexNow and Google Search Console must be maintained in parallel: IndexNow covers Bing and other engines, Google Search Console covers Google. Some SEO professionals have speculated that Google could eventually join the protocol, but as of mid-2026 no official announcement has been made, and Google's existing crawl infrastructure already handles freshness signals through its own mechanisms, reducing the incentive to adopt a protocol designed around a different crawl model.

IndexNow adoption across CMS platforms and CDN integrations

IndexNow has been integrated into a wide range of CMS platforms since Microsoft and Yandex jointly launched the protocol in 2021. If you use WordPress, install the Microsoft Bing Webmaster Tools plugin, which automatically submits URLs to IndexNow on post publish and update. Cloudflare's Crawler Hints feature supports IndexNow, sending notifications to participating search engines when cache signals indicate content has changed⁵. Wix, Squarespace, and Shopify have added native IndexNow support to their publishing pipelines, submitting product and content URLs without requiring user configuration.

For custom or headless CMS setups, you can implement the IndexNow API directly: send a single POST request to https://api.indexnow.org/indexnow with a JSON body containing the host, key, keyLocation, and urlList fields. Your HTTP client library should send this request in under 20 lines of code in most languages. CDN providers including Netlify, Vercel, and Fastly allow webhook configurations on build completion that can trigger an IndexNow submission immediately after a new deployment goes live, providing push-based notification without CMS-level integration.

Choosing the right integration point for your stack

Choose the integration point based on how and when your site's pages become publicly accessible. For server-rendered sites, submit at the point where the page is written to the public file system or served by the web server. For SPA or SSG sites deployed to a CDN, submit at build completion after the deployment pipeline reports success. For database-backed CMS platforms, submit inside the content-saved event handler but only after confirming the content is set to published status. Getting the timing right prevents premature submissions that point to not-yet-live URLs.

Push-based indexing compared to scheduled crawl for fresh content

Before IndexNow, search engines discovered new and updated content primarily through two mechanisms: following links from already-known pages, and reading sitemaps on a scheduled basis. Both mechanisms are reactive and operate on the engine's schedule rather than your publishing schedule. A search engine's crawler might revisit a popular page every few hours but visit a less-prominent page only every few days, meaning a price correction or a breaking-news update could take days to appear in search results.

IndexNow converts this to a push model. When you publish or update a page, you push a notification to participating engines immediately, and they queue a crawl of the affected URL. For time-sensitive content like event listings, news articles, product prices, and job postings, this difference is significant: content can move from published to indexed in hours rather than days. The protocol does not guarantee indexing speed, since final indexing depends on each engine's crawl scheduler and quality assessment, but it removes the discovery delay entirely.

What IndexNow does not replace

IndexNow handles URL discovery and signals content freshness, but it does not replace sitemaps or the practice of earning links from external pages. Sitemaps remain the authoritative inventory of a site's public URL space; engines use them to audit crawl coverage and to discover URLs not yet submitted via IndexNow. External links remain the primary ranking signal for most engines, and IndexNow provides no ranking benefit beyond ensuring the content is crawled promptly. If you rely solely on IndexNow without a complete and accurate sitemap, you risk incomplete crawl coverage for URLs that were never submitted individually.

Try in the tool

What Is a Sitemap?

An XML sitemap is a structured file that lists the URLs on your website, telling search engines which pages exist, when they were last modified, and how often they typically change. Submitting the sitemap to Google Search Console and Bing Webmaster Tools ensures that every page on your site eventually enters the search engine's crawl queue, including pages that are not linked from anywhere else on the site¹.

Sitemaps are a crawl-discovery tool, not a real-time indexing tool. They operate on each engine's schedule: the engine fetches your sitemap periodically and queues any new or updated URLs it finds. This scheduled approach covers your full URL inventory but introduces a lag for freshly published pages. For real-time indexing coverage on Bing and other engines, IndexNow complements the sitemap by pushing individual URLs at the moment of publication.

What is XML Sitemap?

What a sitemap file contains

A standard XML sitemap follows the sitemaps.org protocol. Each URL entry in the file uses a <url> element with a required <loc> tag containing the canonical URL and optional tags for <lastmod> (last modified date in ISO 8601 format), <changefreq> (how often the page changes, from always to never), and <priority> (a relative importance score from 0.0 to 1.0, used as a hint rather than a hard directive by most engines)².

Sitemap index files (sitemap_index.xml) list multiple child sitemap files, each covering a segment of the URL inventory. This structure supports very large sites by staying within the per-sitemap limit of 50,000 URLs and 50 MB uncompressed. For an e-commerce site with 200,000 product pages, a typical setup uses a sitemap index file that references four child sitemaps of 50,000 URLs each, keeping every file within the protocol limits while ensuring the entire product catalogue is discoverable by search engines that parse the index file and queue each child sitemap for independent fetching.

How sitemaps and IndexNow work together

Sitemaps and IndexNow solve complementary problems³. A sitemap provides a complete URL inventory that engines discover on their own schedule; IndexNow sends real-time change signals for individual URLs, bypassing the schedule. Using both ensures that every URL on your site eventually appears in the crawl queue (sitemap) while freshly changed URLs receive priority attention without waiting for the next scheduled sitemap fetch (IndexNow).

Building on this, for new domains where crawl frequency has not yet been established, IndexNow submissions alongside sitemap submission can accelerate the domain's crawl history, because participating engines process IndexNow submissions from new domains in a priority queue that builds trust signals faster. A new domain that submits its first 50 pages via IndexNow during the first week of launch gives each participating engine immediate evidence that the site produces fresh, publicly accessible content, which encourages the engine to increase its crawl frequency and daily budget allocation for that domain more quickly than sitemap-only discovery would achieve.

News sitemaps, video sitemaps, and image sitemaps

Specialty sitemaps extend the standard sitemaps.org protocol with additional metadata fields specific to content types. Google News sitemaps add <news:news> elements containing publication name, language, publication date, and keywords; they notify Google News specifically, separate from the main web crawl. Video sitemaps add <video:video> elements with title, description, thumbnail URL, and content location or embed URL; Bing and Google both parse these for video-specific rich results.

Google's news sitemap documentation requires that news articles submitted via news sitemap were published within the past two days for initial inclusion in Google News Top Stories⁴. Submitting articles older than 48 hours in a news sitemap produces no additional news indexing benefit over the standard web sitemap. IndexNow complements the news sitemap for Bing: submitting the article URL via IndexNow at publication time reaches Bingbot immediately, while the news sitemap handles Google's separate news crawl pipeline.

Image sitemaps and product photography indexing

Image sitemaps add <image:image> elements to standard URL entries, letting you declare image URLs, captions, geographic location, license URLs, and titles for each image on a page. Google Image Search uses this data to enhance image results and attribute images correctly. For e-commerce sites with product photography, adding image sitemap markup to product page entries helps Google associate product images with the correct product URL. Most sitemap generation tools support image sitemap extensions as an optional flag; enable it for any site where image search discovery matters.

Common sitemap errors that block crawl coverage

Malformed XML is the most common sitemap error across all site types. The sitemaps.org protocol requires UTF-8 encoding without a byte-order mark, and special characters in URLs (ampersands, spaces, brackets) must be percent-encoded or XML-entity-escaped. An unescaped & in a URL breaks the entire sitemap XML document, causing parsers to reject the file and leaving the engine with no URL list to process. Validate your sitemap by fetching it through Google Search Console's Sitemaps report or by running it through an XML validator before submitting.

Robots.txt blocks that appear inside sitemap-listed URLs create another significant coverage problem. When a URL is listed in your sitemap but disallowed by robots.txt, engines skip it rather than flagging an error, which makes the omission invisible without cross-referencing your sitemap against your robots.txt rules. Audit this by exporting the URL list from your sitemap and running each URL through a robots.txt testing tool.

File size limits and sitemap index requirements

Sitemaps must not exceed 50,000 URLs or 50 MB uncompressed per file, as defined by the sitemaps.org specification². Sites exceeding these limits need a sitemap index file (sitemap_index.xml) that references multiple child sitemaps, each within the per-file limits. gzip compression reduces the uncompressed size significantly; both Google and Bing accept gzip-compressed sitemaps with the .xml.gz extension. Forgetting to update the sitemap index file after splitting an oversized sitemap is a frequent oversight that leaves new child sitemaps undiscoverable until the index is updated and resubmitted.

Automated sitemap generation and the robots.txt Sitemap directive

CMS platforms generate sitemaps automatically through built-in features or plugins. WordPress sites with Yoast SEO generate dynamic sitemaps at /sitemap_index.xml by default, with separate child sitemaps for posts, pages, categories, and products. Drupal's simple_sitemap module (version 4.x) generates sitemaps at /sitemap.xml and supports custom entity types and multilingual URL variants. Shopify generates a sitemap at /sitemap.xml automatically with no configuration required, covering all published products, collections, and pages.

Automated sitemaps remove the need to manually update the URL list when content changes. However, most automated sitemaps include all published URLs without filtering out low-value pages such as filtered search results, empty category pages, or session-scoped URLs. Adding a noindex meta tag to low-value pages and confirming the sitemap generator excludes noindexed URLs keeps the sitemap focused on content worth crawling.

Adding the Sitemap directive to robots.txt

Adding a Sitemap: https://yourdomain.com/sitemap.xml directive to your robots.txt file lets Bingbot and Googlebot discover your sitemap automatically during their robots.txt fetch⁵, without requiring manual submission to Bing Webmaster Tools or Google Search Console. Both engines read this directive during the robots.txt fetch that precedes every crawl session. Including multiple Sitemap lines in robots.txt lists all sitemap files, which is equivalent to a sitemap index file for the purpose of engine discovery. After adding the directive, verify robots.txt is reachable and parses without errors, because a broken robots.txt file blocks crawling of the entire domain. The robots.txt Sitemap directive is particularly valuable for large sites that split their URL inventory across multiple child sitemaps, because a single Sitemap index line in robots.txt registers every child sitemap with all discovering crawlers without requiring separate submissions for each individual file.

Place the Sitemap directive on its own line at the bottom of robots.txt rather than inside a grouped block, because some older crawlers stop reading the file once they hit an unexpected structured element or a malformed rule. A clean directive separate from any Disallow or Allow rules is parsed reliably by every major engine. Re-check the directive after any robots.txt edit, since a rule added above it can change how the file is tokenized and occasionally push the Sitemap line past what the parser considers valid.

Try in the tool

What Is Crawl Budget?

Crawl budget is the number of pages a search engine will crawl and index on your site within a given time window.¹ Every domain receives a crawl budget from each engine; the budget is not fixed but scales with the domain's authority, server response speed, and crawl efficiency history. Pages that return errors, load slowly, or are blocked by robots.txt consume crawl budget without contributing to the index, leaving fewer budget units for pages that matter.

For most small to medium websites, crawl budget is not a limiting factor.¹ A site with hundreds of pages and fast server response rates typically receives more crawl budget than it needs. Crawl budget becomes relevant for large e-commerce catalogs, news publishers with thousands of articles, or sites with significant crawl waste from parameter URLs, duplicate content, or redirect chains.

What is Crawl Budget?

Factors that affect crawl budget

Domain authority is the primary factor. Established domains with strong backlink profiles and consistent crawl histories receive higher crawl budgets. Server response speed is a secondary factor: a site that responds in 100ms allows the crawler to fetch more pages per unit time than a site that responds in 2 seconds, effectively expanding the practical crawl budget without changing its nominal allocation.²

URL quality is the third factor. Domains that consistently return high-quality, unique content on crawled pages train the engine to allocate more budget over time. Conversely, domains with many crawled URLs that return errors, duplicates, or thin content see budget reductions as the engine models the expected return from crawling the site.

How IndexNow interacts with crawl budget

IndexNow submissions prioritize specific URLs within a domain's crawl budget allocation.³ Submitting a URL via IndexNow does not add to the total budget; it moves submitted URLs to the front of the crawl queue. For sites operating near their crawl budget ceiling, this means IndexNow submissions accelerate high-priority pages at the expense of lower-priority queue position for other pages.

Building on this, submitting only changed and new URLs via IndexNow (rather than all URLs on a broad schedule) maximizes crawl budget efficiency by ensuring the priority queue contains only content that has actually changed since the last crawl. Submitting unchanged URLs wastes crawl budget and reduces the quality signal associated with the domain's submissions over time.

Eliminating crawl waste from parameter URLs and faceted navigation

Faceted navigation generates the most crawl waste of any URL pattern in e-commerce and content sites.⁴ A product category with hundreds of items, multiple color and size filters, and several sort orders can generate tens of thousands of URL combinations through parameter combinations. Each fetched URL that returns near-duplicate content consumes crawl budget without contributing a unique indexed page, leaving fewer crawl budget units for new products and recently updated canonical URLs.

Adding noindex meta tags to faceted navigation URLs stops engines from including them in the index but does not stop crawling.⁴ Disallowing the parameter patterns in robots.txt stops crawling entirely, which is more efficient but requires a careful audit to confirm none of the disallowed parameter combinations are canonical URLs you want indexed. In Bing Webmaster Tools, the URL Parameters feature under Index lets you specify how Bing should handle known parameter patterns, reducing crawl load without requiring robots.txt changes.

Preventing session IDs and tracking parameters from reaching crawlers

Session ID URLs (?sessionid=abc123, ?sid=xyz) create a new URL for every user session, generating unlimited crawl waste when linked from page source or sitemap.² Adding a canonical tag pointing from each session URL to the corresponding canonical URL teaches engines to consolidate signals, but the crawl budget consumed by session URL fetches still counts against the domain. Disallowing known session ID parameter patterns in robots.txt eliminates crawl waste entirely. Tracking parameters (utm_source, fbclid, gclid) receive similar treatment from Bing's URL parameter handling rules, which collapse tracked variants to their canonical base URL before queueing.

Monitoring crawl activity in Bing Webmaster Tools

Bing Webmaster Tools provides reports relevant to crawl budget monitoring.⁵ The Crawl Information report under Reports and Data shows total pages crawled per day, crawl errors by type, and the crawl rate over time. A sudden drop in daily crawl count after an IndexNow submission campaign indicates that bulk submissions exceeded Bing's crawl budget allocation, causing a throttle that reduces daily fetch volume until crawl quality recovers.

The IndexNow Insights section shows submitted URLs alongside their index status and error details. Filtering for failed submissions reveals URLs that returned 4xx errors, were flagged as duplicates, or failed key verification. Cross-referencing failed URLs with the Crawl Information report's error list identifies whether the failure came from a submission validation issue or a crawl-time fetch error.

Setting up Crawl Control in Bing Webmaster Tools

Bing Webmaster Tools includes a Crawl Control feature that lets you specify crawl rate preferences by hour and day of the week.⁵ Sites with peak server loads at predictable times can reduce Bing's crawl rate during those hours without reducing overall crawl volume for the rest of the week. After a bulk IndexNow submission, temporarily lowering the crawl rate ceiling gives the priority queue time to drain before re-enabling full crawl speed. This prevents server load spikes from a sudden burst of IndexNow-triggered fetches arriving simultaneously at the origin.

Crawl budget for new domains during the trust-building period

New domains start with minimal crawl budget from each search engine.³ Bing and Yandex allocate crawl budget dynamically based on domain authority, crawl history, and server response consistency. A freshly registered domain that responds quickly and returns high-quality content on early crawls receives progressively higher crawl budget over weeks and months as the engine builds a positive crawl history for that host.

Maximizing early crawl budget means submitting only your highest-value pages during the trust-building period. Submitting an entire site catalog immediately after launch splits the limited early budget across hundreds or thousands of URLs, leaving many with low crawl priority. Submitting a small set of the most important pages via IndexNow in the first week, then gradually expanding submission volume as crawl rates increase, concentrates budget on content that establishes domain quality signals earliest.

Crawl budget and domain migration

Moving an established domain to a new CMS or URL structure resets the crawl budget allocated to the new URL patterns. Bing treats 301 redirects as signals to transfer crawl priority from old URLs to new ones, but the transfer is gradual and the new URL patterns start with lower crawl frequency than the old ones. Submitting the highest-traffic new URLs via IndexNow immediately after migration seeds the new URL patterns in Bing's priority queue. Monitoring the Crawl Information report in Bing Webmaster Tools in the first 30 days after migration reveals how quickly crawl frequency for the new patterns recovers to pre-migration levels.

Plan the migration submission batch around your busiest content first, because the new URL patterns that carry the most traffic need the crawl priority the most during the recovery window. Pair the IndexNow batch with a verified redirect map so Bingbot never hits a soft 404 on a moved path, since those misses waste the scarce early budget on errors instead of useful pages. Recheck the Crawl Information report at the two-week and thirty-day marks rather than only once, because the recovery curve is rarely linear and a mid-window dip often self-corrects without intervention.

Try in the tool

HTTP 202 Accepted: What It Means for IndexNow

For IndexNow URL submissions, a 200 response from api.indexnow.org is the expected success confirmation: it means the API endpoint accepted your submitted URLs and queued them for distribution to participating search engines.¹ A 202 Accepted response means the URL was received but IndexNow key validation is still pending; this is common on first requests with a new key and typically resolves to 200 after subsequent submissions once verification completes.² The actual engine fetch happens asynchronously after the success response, at each engine's own priority timeline.

The distinction between 200 OK and 202 Accepted matters for IndexNow. A 200 response indicates the key is verified and the submission entered the distribution queue. A 202 response indicates the server accepted the request but is still verifying the key. Because IndexNow distribution to multiple engines is asynchronous, both responses indicate the endpoint acknowledged receipt without waiting for each participating engine to confirm the crawl.

What is 202 Accepted?

202 vs 200 in IndexNow responses

The IndexNow API specification lists both 200 OK and 202 Accepted as valid success responses. A 200 response means the key is verified and the submission entered the distribution queue. A 202 response means the URL was received but key validation is still pending; this is common on first requests with a new key and typically resolves to 200 after the key is verified. Both responses indicate successful receipt; neither is more authoritative than the other in terms of confirming eventual crawl.

CapyToolkit's IndexNow URL Submitter displays the HTTP response code for each submission, so you can see whether the endpoint returned 200 or 202 for your request. Both codes indicate success; 4xx codes indicate errors requiring investigation. The important practical takeaway is that neither 200 nor 202 guarantees the URL has been crawled yet; it only confirms the submission entered the distribution queue, and the actual crawl happens on each engine's own priority schedule after that point.

Error codes vs 202

Understanding the error response codes helps distinguish between a successful 200/202 and a failed submission. A 400 Bad Request indicates malformed JSON, missing required fields, or an invalid URL format in the submission body. A 403 Forbidden indicates the key file at the domain root does not match the submitted key value, or the key file is not reachable at the expected path. A 422 Unprocessable Entity indicates the URL submitted does not match the host specified in the request. A 429 Too Many Requests indicates the submission rate exceeded the engine's rate limit for that domain.

A 200 or 202 response means none of these errors occurred: the URL format is valid, the submission entered the distribution queue, and key verification either passed (200) or is still pending (202). Receiving a 200 or 202 rather than an error code confirms that the IndexNow infrastructure accepted your payload and queued it for fan-out to all participating search engines, which then independently schedule the actual crawl of each submitted URL based on their own priority queue dynamics and your domain's crawl trust level.

Handling 429 Too Many Requests with retry logic

A 429 Too Many Requests response from the IndexNow endpoint means your submission rate exceeded the engine's per-domain rate ceiling.⁴ Bing's approximate rate limit is 600 URL submissions per minute per host; submitting larger batches without pauses reliably triggers 429 responses. When a 429 occurs, no URLs from that request enter the crawl queue; the entire batch is dropped and must be resubmitted.

Exponential backoff is the standard retry pattern for rate-limited APIs. After a first 429 response, wait 5 seconds before retrying. After a second 429, wait 10 seconds. After a third, wait 20 seconds. This pattern prevents overwhelming the endpoint during transient rate-limit periods while still completing the submission within a predictable time window. For batch submission scripts, build this retry logic into the submission loop rather than relying on manual resubmission after failures.

Choosing batch sizes to stay under rate limits

Submitting 50 URLs per request with a 6-second pause between requests keeps submission volume at approximately 500 URLs per minute, well under Bing's 600-per-minute ceiling. Over one hour this approach handles approximately 30,000 URL submissions without triggering rate-limit responses. For smaller submission pipelines, batching at 10 to 20 URLs per request provides even more headroom. The IndexNow API accepts up to 10,000 URLs per POST, but submitting at that ceiling without pauses consistently produces 429 responses for the excess. Treat the per-request URL limit as a protocol maximum, not a recommended batch size.¹

Logging IndexNow response codes in a submission pipeline

Recording the HTTP response code for every IndexNow submission creates an audit trail that reveals patterns invisible from individual submissions. A pipeline that logs URL, submission timestamp, response code, and retry count for each URL lets you identify which domains or URL patterns consistently produce 403 or 422 errors, diagnose rate-limit incidents with exact timing, and confirm that a bulk submission batch completed without silent failures.

A minimal log entry needs four fields: the submitted URL, the Unix timestamp, the HTTP response code, and the request host. Storing these in a database or appending to a CSV file supports offline analysis without requiring a dedicated logging platform. After each bulk submission, filter the log for non-2xx response codes and categorize the failures: 403s point to key file issues, 422s point to URL-host mismatches, and 429s point to rate-limit incidents requiring batch-size adjustments.

Alerting on sustained 403 failures

A sustained run of 403 responses across an entire submission batch indicates a key file problem rather than a transient error. Key files can become unreachable due to web server configuration changes, CDN cache purges that accidentally clear the key file path, or HTTPS certificate renewals that briefly cause certificate errors on the key file URL. Set up an alert in your logging pipeline that triggers when the 403 error rate for a domain exceeds 20% over a five-minute window. This alert fires early enough to investigate the key file issue before the entire day's submissions are lost to silent rejection.

Diagnosing 422 Unprocessable Entity in IndexNow submissions

A 422 Unprocessable Entity response from the IndexNow endpoint means the API understood the request format but rejected it because the URL does not belong to the declared host. The most common cause is a mismatch between the host field in the request body and the domain portion of URLs in the urlList. If the host field declares example.com but a URL in the list points to www.example.com, the API rejects the submission with 422.

Fixing a 422 requires inspecting the request body for domain consistency. The host field must match the domain without https:// and without a trailing slash, matching exactly the domain used in all submitted URLs. If your site uses both the apex domain and www, decide which canonical form to use and ensure all submitted URLs and the host field use the same form consistently.

Cross-domain URL filtering before submission

Large submission pipelines that aggregate URLs from multiple sources (sitemaps, CMS webhooks, analytics exports) sometimes include URLs from external domains or CDN-served asset URLs. Filtering the URL list to include only URLs that match the declared host before calling the IndexNow API eliminates 422 errors at the source. A string filter that checks whether each URL starts with https:// followed by the declared host removes any URLs from mismatched domains before the batch reaches the endpoint.

Run the same host filter on the host field itself as a sanity check, because a single stray character or a copied trailing slash in the host value triggers 422s across the whole batch even when every URL is correct. Validate both the host string and the URL list against the same canonical form before sending, so the request body is internally consistent rather than relying on one side matching the other. Catching a malformed host at filter time avoids a submission round-trip that would otherwise fail every URL in the payload.

Try in the tool

What Is an IndexNow Verification Key File?

An IndexNow verification key file is a plain text file placed at the root of your domain that proves you own or control the domain before you can submit URLs via IndexNow.¹ The file is named using your API key value (e.g. a1b2c3d4.txt) and contains only that key value as its content. When you submit URLs from that domain, the IndexNow endpoint fetches the key file and verifies that the file contents match the key in your submission before distributing the URLs to participating engines.

The key file is a one-time setup requirement. Once placed correctly and confirmed accessible via HTTPS, the same key file authorizes all future submissions from that domain indefinitely, unless you rotate the key. Rotating the key requires updating the key file name, its contents, and the key value you send in submission requests simultaneously.

What is IndexNow Key File?

Setting up the key file

Generate a key value: any string between 8 and 128 characters using lowercase alphanumeric characters, uppercase characters, and hyphens.² Many CMS plugins generate this automatically. Create a plain text file named {key}.txt (for example, if your key is abc123, the file is abc123.txt). Place the file at the root of your domain so it is reachable at https://yourdomain.com/abc123.txt. The file should contain only the key value as the entire file content, with no extra whitespace or newlines.

After placing the file, confirm it is reachable by opening the URL in a browser or using a command line HTTP request. The response should return 200 OK with the plain text key value as the body. Once confirmed, you can begin making IndexNow submissions using that key and key file location.

Key reuse across subdomains

The same key value can be used across multiple domains or subdomains, but each domain still requires its own key file at its root.¹ If you manage example.com, blog.example.com, and shop.example.com, all three can share the same key value, but each needs the file at https://example.com/{key}.txt, https://blog.example.com/{key}.txt, and https://shop.example.com/{key}.txt respectively.

This shared-key approach simplifies key management: rotating the key only requires updating the key value across all files and updating the submission configuration. Building on this, using the same key across all properties gives you a consistent operator identity across participating engines. The trade-off is that a single compromised key affects all domains simultaneously, so teams managing client sites or high-value properties may prefer per-domain keys to limit the blast radius of any single key exposure.

Troubleshooting a 403 Forbidden response after key file setup

A 403 response after placing the IndexNow key file almost always comes from one of five causes: the file is not publicly accessible, the server returns the wrong content type, the file contains extra characters beyond the key value, the filename does not match the key exactly, or a CDN is serving a cached error response for the path.²

Verify public accessibility first. Open https://yourdomain.com/{key}.txt in a browser while logged out of all accounts, or use curl -I to confirm the URL returns a 200 status with no authentication redirect. If the file returns 200 in your browser but the IndexNow submission still returns 403, the response content type may be wrong. Servers that map .txt files to text/html rather than text/plain cause 403 failures because the validation check expects plain text. Check your web server's MIME type configuration for .txt files.

Byte-order mark and trailing whitespace failures

A byte-order mark (BOM) added by some text editors at the start of a UTF-8 file causes silent 403 failures. The key file must contain only the key value as raw ASCII text with no BOM. Creating the file with echo -n "yourkey" > yourkey.txt on Linux or Mac, or using a code editor such as VS Code with the BOM option disabled, produces a clean file. Trailing newlines are less likely to cause failures, but to match the protocol specification exactly, create the file without any trailing newline or whitespace. Recheck after every web server configuration change that might affect how .txt files are served.

Hosting the key file on CDN-served or platform-restricted infrastructure

Platform-restricted hosting environments (Webflow, Squarespace, Shopify, Wix) prevent direct file uploads to the domain root, which is the standard location for the IndexNow key file.³ Cloudflare Workers and Pages, AWS S3, and GitHub Pages serve as external key file hosts; you then configure a redirect rule on the platform to route /{key}.txt requests to the external host URL.

In Cloudflare Pages, create a _redirects file in your project root with a rule mapping /{key}.txt to the S3 or GitHub Pages URL serving the file. Alternatively, a Cloudflare Worker listening on the key file path returns the key value directly without needing an external host: the Worker responds with return new Response(keyValue, { headers: { 'content-type': 'text/plain' } }). This approach keeps the key file entirely within your Cloudflare account without additional storage configuration.

CDN cache invalidation after key file placement

CDN-cached 404 responses for the key file path are a frequent setup blocker. If you placed the key file after your CDN cached a 404 for that path, the cached response continues serving until the TTL expires or you manually purge the cache. In Cloudflare, purge the specific URL under Caching → Cache Purge → Purge By URL after placing the key file redirect. After purging, wait one to two minutes for CDN edge propagation before retrying the IndexNow submission. A direct curl request with a custom host override to hit the origin IP confirms the file is correctly placed at the origin before verifying that the CDN serves it correctly.

Key rotation: updating the key value across all submission configs

Key rotation is the process of replacing a compromised or expired IndexNow key with a new value. The IndexNow protocol does not enforce a rotation schedule, but security policies at organizations handling high-volume submissions often require annual or semi-annual rotation. Rotating a key requires updating the key file, the submission configuration, and any CMS plugin settings simultaneously; a mismatch between these three components causes 403 failures on all submissions until the inconsistency is corrected.³

Start rotation by generating a new key value and placing the new key file at the domain root before making any changes to submission configs. Both the old and new key files can coexist at the domain root; the old file remains authoritative until submission configs switch to the new key. After the new file is confirmed reachable via curl, update the key value in your CMS plugin settings or submission pipeline, then delete the old key file. This sequence ensures continuous submission without a validation gap.

Rotation across multi-domain deployments

For deployments using the same key value across multiple domains, rotation requires updating the key file on every domain simultaneously. Automating this step prevents a partial rotation where some domains switch to the new key while others still serve the old key file. A deployment script that pushes the new key file to all domains in a single operation and verifies each domain's response before updating submission configs ensures clean rotation. For WordPress multi-site or Drupal multi-site deployments, update the IndexNow plugin API key in the network settings rather than per-subsite to propagate the new key across all subsites at once.

Verify the rotation from a staging domain first when one is available, because a script that pushes the new key to all production domains at once leaves no easy rollback if the push fails partway through. Confirm the new key file returns 200 on a single domain, switch its submission config, and submit a test URL that returns 200 rather than 403 before rolling the change out to the remaining domains. This staged approach turns a high-risk all-at-once operation into a sequence of small, observable steps where a failure affects one property instead of the whole fleet.

Try in the tool

What Is Yandex Webmaster?

Yandex Webmaster is the official webmaster platform for Yandex Search, the largest search engine in Russia¹ and prominent across Eastern Europe and Central Asia. Similar in function to Google Search Console, Yandex Webmaster provides tools for submitting sitemaps, monitoring crawl status, reviewing indexed pages, diagnosing crawl errors, and viewing search performance data for a site's Yandex presence.

For sites targeting Yandex-heavy markets, Yandex Webmaster is a required part of the SEO toolkit. Yandex applies its own crawl algorithms, ranking factors, and index quality criteria that differ from Google's and Bing's, meaning technical SEO decisions (URL structure, sitemap format, canonical tags) should be verified in Yandex Webmaster rather than assumed to behave identically to Google Search Console.

What is Yandex Webmaster?

Key features of Yandex Webmaster

Sitemap submission accepts sitemaps in the standard sitemaps.org XML format, the same format used by Google and Bing. Once submitted, Yandex Robots (the Yandex crawler) fetches the sitemap on its own schedule and queues listed URLs for crawling. The Dashboard shows index status, crawl errors, and a count of indexed pages, providing a quick health check for the site's Yandex presence.

The Search Queries report shows which queries triggered Yandex results for the site, along with impressions, clicks, and position data similar to Google Search Console's Performance report. Furthermore, Yandex Webmaster includes a URL Diagnosis tool that checks an individual URL for crawl accessibility, canonical tag recognition, and index status. The URL Diagnosis tool is particularly valuable for Russian-language sites because it reveals exactly how YandexBot interprets your page structure, including whether the engine correctly identifies the canonical URL, reads the hreflang annotations, and processes the Russian-language content signals that Yandex's quality filter evaluates before admitting a page into the active search index.

IndexNow and Yandex Webmaster

Yandex participates in IndexNow, so submitting URLs via IndexNow sends an immediate crawl signal to Yandex alongside Bing and other participants.² For Yandex-targeted content, this means publishing a page and immediately submitting via IndexNow delivers the URL to Yandex Robots for priority processing without requiring a manual Yandex Webmaster submission.

Yandex Webmaster's sitemap submission remains valuable alongside IndexNow: it provides the full URL inventory for scheduled discovery and the monitoring tools to verify that Yandex is actually crawling and indexing submitted content. Building on this, the URL Diagnosis tool in Yandex Webmaster can confirm whether a specific IndexNow-submitted URL was crawled and indexed, serving as a verification step that IndexNow's API response alone does not provide.

Crawl error reporting and the Pages Excluded report

Yandex Webmaster runs daily site diagnostics that surface indexing errors and configuration recommendations under Website optimization → Site diagnostics, so you can address server errors, robots.txt blocks, and other failure types before they affect indexing. For Yandex-targeted sites, diagnosing these errors in Yandex Webmaster is more reliable than inferring YandexBot behavior from Google Search Console data, because Yandex and Googlebot interpret robots.txt wildcard patterns differently in several edge cases.

The Pages section under Site Statistics adds another layer of crawl visibility. It displays a breakdown of indexed pages, excluded pages, and pages excluded due to duplicates. Yandex's duplicate detection groups pages with nearly or completely identical text content that share the same domain but have different URLs and excludes all but one from active results.³ Reviewing this report before a bulk IndexNow submission surfaces URLs Yandex has already decided to exclude, so you can fix the underlying issue rather than wasting crawl budget on known rejections.

Running URL Diagnostics before an IndexNow submission

Yandex Webmaster's URL Diagnostics tool accepts a single URL and reports its crawl accessibility, HTTP status code as seen by YandexBot, canonical tag value, and current index status. Run any URL through Diagnostics before submitting via IndexNow to confirm YandexBot can fetch it without errors. A URL returning 200 in your browser but showing an access error in Diagnostics typically has a conditional redirect that responds differently to YandexBot's user agent string. Fixing these issues before submission avoids silent rejection and protects your domain's crawl quota.

Yandex's behavioral ranking signals and search performance data

Yandex incorporates behavioral signals such as click-through rate, dwell time, and bounce rate from search results into its ranking algorithm, and these signals influence how pages are ranked for similar queries over subsequent crawl cycles. Optimizing for Yandex therefore includes writing Russian-language titles and meta descriptions that match Yandex search query intent precisely, because a low click-through rate in Yandex results can depress rankings for the affected queries.

The Search Queries report in Yandex Webmaster shows impressions and clicks at the query level, including position data for recent periods. Filtering the report by position to find queries where your page ranks between positions 4 and 10 with low CTR identifies optimization targets specific to Yandex. Yandex Webmaster also lets you filter the same report by device type, which helps isolate desktop-specific and mobile-specific CTR issues.

Structured data in Yandex results

Yandex supports Schema.org structured data alongside its own Yandex Turbo pages format. Adding Organization and BreadcrumbList Schema.org markup improves rich snippet eligibility in Yandex results similarly to how it affects Google.⁴ Yandex Webmaster's Site Appearance section previews how your pages appear in Yandex results and surfaces structured data validation errors that prevent rich snippets from appearing. Check this section after adding new Schema.org markup to your pages before submitting updated URLs via IndexNow.

Domain ownership verification methods in Yandex Webmaster

Yandex Webmaster offers three domain ownership verification methods: HTML file upload, meta-tag insertion, and TXT record via DNS.⁵ The HTML file method is the most reliable for standard hosted infrastructure: upload the provided verification file to your domain root so it is reachable at https://yourdomain.com/{verification-file}.html, then click Verify in the Yandex Webmaster dashboard. This file can remain permanently in place without affecting your site.

The meta-tag method inserts a <meta name="yandex-verification"> tag in the <head> of your homepage. It is convenient for platforms that restrict root-level file uploads, such as Wix hosted sites or managed e-commerce platforms. Removing the meta tag after verification revokes access without warning, so leave it in place for the duration you manage the property.

Subdomain and regional domain verification

Yandex Webmaster requires separate verification for each subdomain and each country-code top-level domain. Verifying example.com does not grant access to data from m.example.com or example.ru; each requires its own verification step. For sites serving Russian, Belarusian, and Kazakh markets on separate domains or subdomains, verify each property individually. Yandex Webmaster groups primary and alternate site addresses so their search performance data is attributed to a single property, which simplifies monitoring IndexNow submission results across a multi-domain publishing setup.

Set a recurring reminder to re-verify properties after any domain migration or hosting change, because a move that changes the document root can silently invalidate the HTML file method and revoke your access without sending an alert. Keep the verification file and meta tag in place even after verification succeeds, since removing either one immediately cuts off your view into Yandex's crawl behavior. When a property loses verification, resubmitting IndexNow URLs becomes guesswork, because you can no longer confirm whether Yandex actually crawled the pages you pushed.

Try in the tool