URL Canonicalization and Normalization

URL Canonicalization and Normalization

URL canonicalization is the process of converting a URL into a standardized form so that two URLs referring to the same resource produce the same string. Without canonicalization, a CDN cache may store the same resource at five different keys, a web crawler may index the same page dozens of times, and a signature verification system may reject a valid request because the URL was normalized differently by the client and server.

RFC 3986 Section 6 defines several normalization steps, including case normalization, percent-encoding normalization, and path segment compression.¹ Beyond these, applications apply additional rules: stripping the default port, sorting query parameters, removing tracking parameters, and preferring trailing-slash or no-trailing-slash forms.

RFC 3986 normalization steps

Case normalization lowercases the scheme and host: HTTP://EXAMPLE.COM/path becomes http://example.com/path. Percent-encoding normalization decodes unreserved characters (%41 → A) and uppercases hex digits in remaining encoded triplets (%2f → %2F). Path compression removes dot-segments: /a/b/../c becomes /a/c, and /./path becomes /path.¹ Consequently, most URL parsers apply case normalization automatically, so url.hostname in JavaScript always returns lowercase. Path compression requires an explicit step: URL constructors resolve dot-segments, but strings with dot-segments passed to HTTP clients may or may not be compressed before the request is sent. These three steps together cover the majority of normalization cases that applications encounter in practice, and most modern URL libraries implement them by default.

Application-level normalization

Beyond RFC 3986, applications apply their own rules. Default port removal strips :80 from http:// URLs and :443 from https:// URLs because their presence or absence should not affect caching. Query parameter sorting produces a deterministic key for the same logical URL: ?b=2&a=1 and ?a=1&b=2 refer to the same page, so sorting alphabetically by key name produces a stable cache key. Building on this, trailing slash normalization must be consistent across the application since redirecting /about to /about/ (or the reverse) prevents duplicate content. For SEO, use a rel='canonical' link element and a 301 redirect to enforce the preferred form.²

Canonicalization in practice

For cache key generation, apply: scheme lowercasing, host lowercasing, default port removal, percent-encoding normalization, and path compression. Optionally sort query parameters and remove known tracking parameters (utm_source, fbclid, gclid). Consequently, a canonical URL function in JavaScript can be written as: function canonical(raw) { const u = new URL(raw); u.searchParams.sort(); return u.origin + u.pathname + (u.search || ''); }. Building on this, for request signing (e.g., AWS Signature V4), use the exact canonicalization algorithm specified by the signing protocol because ad-hoc normalization will break signature verification. The AWS documentation specifies every encoding rule down to the hex digit case, so deviating from it in any way causes 403 errors that are difficult to diagnose without comparing the canonical request string byte by byte.

Canonical URLs for SEO and content deduplication

Search engines treat different URL representations of the same content as separate pages unless you signal the preferred form. A product page accessible at /product/123, /product/123?ref=google, and /product/123?utm_source=twitter may be indexed as three separate pages, diluting ranking signals across duplicates. The rel="canonical" link element in your HTML head tells search engines which URL represents the original: <link rel="canonical" href="https://example.com/product/123" />.² Google follows canonical hints but may ignore them if the signals are contradictory; use consistent internal linking to reinforce the canonical choice.

When to use 301 redirects versus canonical tags

A 301 redirect physically moves all traffic from the old URL to the new one, consolidating all signals, while a canonical tag keeps the old URL accessible but tells search engines to attribute signals to the new URL. Knowing when to use a 301 over a canonical tag comes down to whether the old URL should disappear: redirect it if so, or add a canonical tag when multiple URLs serve the same content for functional reasons like tracking parameters, session IDs, or print versions. For pagination, Google recommends either a view-all page with canonical pointing to it, or self-referencing canonical tags on each page in the sequence rather than canonical-all to page one.

URL normalization in web crawling and deduplication

Web crawlers that encounter the same resource at multiple URLs must normalize before storing to avoid fetching the same page twice. A crawler processing 1 million URLs may encounter 30% duplicates after normalization: different tracking parameters, session IDs, and casing variations all resolve to the same content. Apply these normalization steps before deduplication: lowercase the scheme and host, remove default ports, resolve dot-segments, percent-encoding normalization (decode unreserved characters, uppercase remaining encoded triplets), and optionally sort query parameters.

Bloom filters for URL deduplication at scale

A crawler processing billions of URLs cannot store all seen URLs in a hash set; the memory requirement is prohibitive. A Bloom filter provides a space-efficient probabilistic data structure that answers "have I seen this URL before?" with no false negatives and a configurable false-positive rate.³ For a smaller-scale crawler processing millions of URLs, a Redis-backed Bloom filter (using the RedisBloom module) gives you distributed deduplication without managing in-memory data structures.

The trade-off for that memory saving is the false-positive rate: a Bloom filter may occasionally report that a URL was seen when it was not, which means a genuinely new URL can be skipped as a duplicate. Pick the filter size and hash count so the false-positive rate stays low enough that rare skips do not matter for your crawl. Because the structure has no false negatives, you never mistakenly re-fetch a URL you already stored, which is the property that matters most for crawler efficiency.

Request signature canonicalization in AWS and OAuth

Amazon Web Services requires a canonical request string as input to the SigV4 signing algorithm. The canonical request includes: the HTTP method, the canonical URI (percent-encoded path), the canonical query string (sorted by parameter name, then by value), canonical headers (sorted by header name, lowercase), signed headers list, and the hashed payload.⁴ Each component has its own encoding rules: the path is percent-encoded with uppercase hex digits, spaces in query values are encoded as %20 (not +), and the empty string for a missing query component is still included. A single deviation at any step produces a signature that does not match.

OAuth 1.0 signature base string construction

OAuth 1.0 requires constructing a signature base string from the request method, the base URL (without query parameters), and the normalized parameter string.⁵ The base URL is the scheme, host, port, and path; query parameters and fragments are excluded. Parameters are collected from both the query string and the POST body (if form-encoded), sorted by name, concatenated with &, and percent-encoded using a specific encoding function (RFC 3986 with uppercase hex). The resulting base string is METHOD&encoded_base_url&encoded_parameters, which is then hashed with the client secret. Mismatches in any normalization step fail signature verification, making OAuth 1.0 notoriously difficult to implement without a library.

When to use this

Apply URL canonicalization when implementing cache keys, deduplicating URLs in a web crawler, generating consistent API request signatures, or solving duplicate content issues for SEO.

Examples