Submitting a Sitemap to Search Engines

Submitting a Sitemap to Search Engines

A sitemap describes your entire URL inventory for crawl-based discovery, while IndexNow sends real-time change signals for individual URLs. Sitemaps and IndexNow solve different problems. Both serve the same indexing goal by different mechanisms, and using both together provides faster and more complete search engine coverage than either tool alone.

Submitting a sitemap to Google Search Console and Bing Webmaster Tools is the standard baseline for any website. For Google, it is also one of the few direct signals you can send to Googlebot, since Google does not support IndexNow.¹ For Bing, sitemaps provide the historical discovery layer that IndexNow's real-time signals extend; Bing processes sitemap submissions on a scheduled crawl basis, while IndexNow bypasses that schedule for newly changed URLs.²

How to submit a sitemap to Google and Bing

Google Search Console accepts sitemap submissions under Index → Sitemaps. Navigate to the Sitemaps section, enter the sitemap URL (typically /sitemap.xml or /sitemap_index.xml), and click Submit. Google fetches and parses the sitemap on its own schedule, usually within hours of the first submission and regularly thereafter.³ After submitting, the Sitemaps report shows the submission date, last read date, and the number of URLs Google discovered, which gives you a quick way to confirm that Google successfully fetched and parsed your sitemap and that the discovered URL count matches your expectations.

Submitting to Google and Bing

For Bing, submit the sitemap under Sitemaps in Bing Webmaster Tools. Bing also accepts a Sitemap directive in your robots.txt file (Sitemap: https://yourdomain.com/sitemap.xml), which Bingbot discovers automatically during its robots.txt fetch without manual Webmaster Tools submission.⁴ Submitting through both the Bing Webmaster Tools interface and the robots.txt directive simultaneously provides redundancy: if Bingbot fetches your robots.txt before you have manually submitted the sitemap URL, it discovers the sitemap path automatically, and if the robots.txt directive is temporarily unreachable due to a server issue, the manually submitted sitemap URL in Bing Webmaster Tools ensures the sitemap remains registered. This dual-submission approach is the recommended best practice because it eliminates the single point of failure that would exist if you relied on only one discovery method, and it takes less than two minutes to complete both submissions during initial site setup.

What sitemaps cover that IndexNow does not

Sitemaps describe the full URL inventory of your site, including pages that may not trigger a CMS publish event (static pages, archived content, category pages). IndexNow covers only URLs you explicitly submit at the point of change. Building on this, sitemaps allow you to specify a lastmod date for each URL, giving engines a signal about content freshness without requiring a separate push notification for each page.

For large sites with thousands of pages, sitemaps ensure that every URL is eventually discoverable even if individual pages are never submitted via IndexNow. Conversely, IndexNow ensures recently changed pages receive priority attention before the sitemap's next crawl cycle. A practical way to think about the division of labor is that sitemaps answer the question "what pages exist on this site?" while IndexNow answers the question "which pages have changed right now and need immediate crawl attention?", and both questions must be answered for complete search engine coverage.

When a sitemap alone is not enough

Sitemaps are a request to crawl, not a guarantee of prompt indexing. For time-sensitive content like news articles, product launches, or event announcements, the gap between sitemap submission and the engine's next scheduled fetch can be hours or days. For Bing and other participating engines, IndexNow indexes pages between sitemap crawls by delivering an immediate priority crawl signal.

Furthermore, newly launched domains often experience slow sitemap processing because search engines apply lower crawl priority to domains without established trust signals. Combining IndexNow submission with a sitemap submission helps new domains accumulate crawl history faster, because IndexNow signals quality intent while the sitemap provides the discovery inventory. During the first two to four weeks after a new domain launches, prioritizing IndexNow submissions for every new page while keeping the sitemap updated gives the domain the fastest possible path to building the crawl trust that eventually increases the engine's daily crawl allocation and sitemap fetch frequency.

Sitemap format requirements: encoding, URL limits, and compression

The sitemaps.org protocol specifies exact format requirements. Sitemaps must use UTF-8 encoding without a byte-order mark. Each sitemap file may contain a maximum of 50,000 URL entries and must not exceed 50 MB in size when uncompressed; files that exceed either limit are rejected by the parser without processing any URLs.⁵ The <lastmod> date element, when present, must use ISO 8601 format (either YYYY-MM-DD or the full datetime form YYYY-MM-DDTHH:MM:SS+HH:MM); informal date formats like MM/DD/YYYY cause the element to be silently ignored rather than raising a validation error.

Gzip compression is supported and recommended for large sitemaps. Google and Bing both accept .xml.gz files submitted by sitemap URL, reducing bandwidth and crawl overhead for files with tens of thousands of URL entries. Compress the file after generation and before submitting; the HTTP response from your server should include Content-Encoding: gzip and the URL itself should end in .xml.gz to avoid content-type confusion. If your server adds a .gz extension to a file not actually gzip-compressed, the parser fails silently.

Special characters in URLs require XML entity escaping

URLs containing ampersands, angle brackets, apostrophes, or quotation marks in the <loc> element must use XML entity escaping: & becomes &, < becomes <, > becomes >. URLs containing spaces must be percent-encoded as %20 before inclusion in the sitemap. Most sitemap generation libraries handle this automatically, but manually assembled sitemaps or sitemaps exported from spreadsheets frequently contain unescaped characters that silently break XML parsing for all URLs appearing after the first malformed <loc> entry.

Using the robots.txt Sitemap directive for automatic discovery

Adding a Sitemap: directive to your robots.txt file lets both Bingbot and Googlebot discover your sitemap without requiring a manual submission to Bing Webmaster Tools or Google Search Console. The directive appears on its own line and provides the full absolute URL of your sitemap: Sitemap: https://yourdomain.com/sitemap.xml. Both engines read this directive during the robots.txt fetch that precedes every crawl session, which means any engine crawling your site discovers the sitemap automatically.⁴

You can include multiple Sitemap directives in a single robots.txt file to reference multiple sitemap files individually or to reference a sitemap index file alongside separate specialty sitemaps. Adding both Sitemap: https://yourdomain.com/sitemap.xml and Sitemap: https://yourdomain.com/sitemap-news.xml simultaneously registers both files with all discovering crawlers. The Google Search Console and Bing Webmaster Tools sitemap submission interfaces show sitemaps discovered via robots.txt alongside manually submitted ones, providing a unified view of all registered sitemaps regardless of how they were registered.

Verifying robots.txt is valid before adding the Sitemap directive

A robots.txt file with a syntax error stops engines from processing all directives below the error. Before adding the Sitemap directive, validate your robots.txt file using Google Search Console's robots.txt tester or the free robotstxt.io validator. A malformed Disallow directive above the Sitemap line can prevent the sitemap directive from being read entirely. After adding the Sitemap line, reload the robots.txt tester to confirm the directive is parsed correctly and the sitemap URL resolves to a 200 response. This validation step is especially important for sites that have accumulated robots.txt rules over years of development, because a single misplaced colon or missing slash in an early Disallow rule can silently invalidate every directive that follows it, including your Sitemap line.

Re-validate the robots.txt each time you edit it, not only when you first add the Sitemap line, because a later change to a Disallow rule is just as capable of breaking the directive as the original mistake was. Treat the robots.txt tester as a pre-deploy check in the same way you would treat a build step, and run it after every edit whether or not you touched the Sitemap line. This habit catches the silent failures that would otherwise hide until a crawler skipped your entire sitemap for days without any visible error in the console.

When to use this

Submit your sitemap to both Google Search Console and Bing Webmaster Tools as a baseline one-time setup, then update whenever your URL structure changes significantly. Use IndexNow in parallel for real-time change signals on individual pages, covering the gap between sitemap crawl cycles.

Examples

Sitemap submitted; new articles appear in Bing results 6-12 hours after publish

Add IndexNow submission at publish; new articles appear in Bing results within minutes (approximate).