URL Parser Reference

What Is a Query String?

Because URLs must carry data without breaking their structure, the query string provides a dedicated location for key-value pairs that communicate intent, state, or filtering criteria to the server. Found after the ? character and before the # fragment¹, the query string is part of every URL that passes parameters, from search queries to OAuth callbacks to pagination state.

What is a query string?

Structure and encoding

Each key-value pair in a query string follows the pattern key=value. Multiple pairs are joined by & characters: ?name=alice&role=admin&page=2 contains three parameters. Consequently, values that contain &, =, or # must be percent-encoded to prevent them from being misread as delimiters. The + character has a special meaning in query strings: in the application/x-www-form-urlencoded format used by HTML forms, + represents a space. In RFC 3986 query strings, + is a literal plus sign, so use %20 for spaces in RFC-compliant contexts². Choosing the right encoding convention for your context prevents subtle bugs where a space is interpreted as a plus or vice versa by the receiving server.

How servers and browsers use query strings

Web servers receive the query string as part of the request path and expose it to application code as a parsed data structure. In PHP, $_GET contains the parsed parameters³. In Python WSGI, environ['QUERY_STRING'] holds the raw string⁴. Consequently, the query string is sent to the server on every request, unlike the fragment (#), which is processed entirely by the browser and never transmitted to any backend. Building on this, query strings appear in server logs, proxy caches, and HTTP history, so they are not a secure location for sensitive data like passwords or API keys.

Repeated keys and empty values

The query string specification does not prohibit repeated keys: ?tag=js&tag=python is valid and conveys an array of two values for the tag key. How servers handle repeated keys depends on the framework: PHP collects them into an array when the key ends with []. Python's parse_qs always returns lists. JavaScript's URLSearchParams preserves all values and returns them via getAll()⁵. Furthermore, a key may appear with no value (?debug) or with an empty value (?debug=), and these are distinct states that may carry different meanings in your application. Always normalize how you treat missing versus empty values so that your API does not silently conflate "parameter absent" with "parameter present but blank."

Query strings in form submissions and the encoding boundary

HTML forms submit data using the query string format by default. A form with method="GET" serializes its fields into the URL's query string: <form method="GET" action="/search"><input name="q" value="hello world"></form> submits to /search?q=hello+world. The browser encodes spaces as + and special characters as %XX, following the application/x-www-form-urlencoded format². When the same form uses method="POST", the same encoding applies but the data goes in the request body, not the URL. This encoding difference matters when your server-side code expects one format but receives the other.

How multipart form data differs from query strings

Forms with enctype="multipart/form-data" (used for file uploads) do not use query string encoding at all. The browser sends each form field as a separate MIME part with its own headers, and the server parses the multipart boundary rather than splitting on & and =. If your API accepts both form submissions and query string parameters, document which encoding each endpoint expects. A POST endpoint that reads from $_GET in PHP will not see multipart form data; it reads from $_POST or $_FILES instead.

Because multipart encoding includes a boundary delimiter in the Content-Type header, the request body cannot be parsed as a simple key-value string. Each part carries its own Content-Disposition header naming the field, and file parts include a filename and content type. This structure makes multipart essential for binary data but adds overhead for simple key-value pairs where application/x-www-form-urlencoded is more compact.

Query string parsing in serverless and edge functions

Serverless functions and edge runtimes (Cloudflare Workers, Vercel Edge, AWS Lambda@Edge) receive the query string as part of the request URL. In Cloudflare Workers, new URL(request.url).searchParams gives you the same WHATWG API as in a browser. Vercel Edge Functions expose the query object directly from the request context. The parsing behavior is identical to browser and Node.js implementations, but cold start time means you should avoid constructing URL objects in hot paths for high-throughput edge middleware.

Caching implications of query strings in CDNs

CDN caching behavior varies by provider, but most CDNs treat the full URL including the query string as the cache key. A request to /api/data?page=1 and /api/data?page=2 are cached as separate entries. Some CDNs allow you to configure which query parameters are included in the cache key: Cloudflare's Cache Keys feature and Fastly's VCL both support stripping or including specific parameters. For APIs where the query string contains session tokens or tracking IDs that do not affect the response, configure your CDN to ignore those parameters and avoid cache fragmentation that reduces your hit rate.

Query strings versus request bodies for API design

The choice between query strings and request bodies depends on the semantics of the data. Query strings are appropriate for parameters that identify or filter a resource: search terms, pagination offsets, sort fields, and filter criteria. Request bodies (JSON, form data) are appropriate for data that creates or modifies a resource: user profiles, document content, or complex nested structures. Consequently, a GET /search?q=term&page=2 endpoint uses query strings because the parameters describe how to retrieve the resource. A POST /users endpoint sends the user data in the body because it creates a new resource.

When query strings become too complex

If your query string grows beyond five or six parameters, or if any parameter value is a complex nested object, switch to a request body. Some proxy servers and WAFs reject URLs longer than 8 KB, and deeply nested filter structures are easier to express in JSON than in flat key-value pairs. GraphQL APIs solve this by always using POST with a JSON body, even for read-only queries, specifically to avoid query string complexity limits.

Try in the tool

What Is a URL Scheme?

Because different resources require different access methods, the URL scheme specifies the protocol or mechanism used to retrieve the resource. The scheme is the first component of a URL, everything before the colon, and determines what the rest of the URL means and how it should be interpreted.¹

What is a URL scheme?

Common URL schemes

The http scheme identifies resources accessible via the Hypertext Transfer Protocol, while https adds TLS encryption. Consequently, ftp identifies resources on File Transfer Protocol servers, largely replaced by HTTPS for file delivery but still used in legacy systems. The mailto scheme forms email addresses: mailto:[email protected] opens the user's mail client. The file scheme accesses local files: file:///etc/hosts reads a local file path with three slashes (two for the authority separator and one for the path). Building on this, data URIs embed content directly: data:text/plain;base64,SGVsbG8= embeds a base64-encoded string.² The tel scheme links telephone numbers (tel:+1-555-0100) and the sms scheme composes SMS messages, both handled by the operating system rather than the browser.

Custom and private URI schemes

Operating systems and applications register custom URI schemes to handle protocol intents: slack:// opens the Slack app, vscode:// opens VS Code, and myapp:// launches a mobile app. These are registered either with IANA (for public schemes) or at the operating system level (for app-specific schemes). Consequently, browsers pass URIs with registered custom schemes to the operating system's URL handler rather than loading them as web pages. Building on this, web applications can register as handlers for custom schemes via the registerProtocolHandler() browser API, allowing web apps to intercept custom URI navigations.³ The web+ prefix convention prevents conflicts with OS-registered schemes: a handler registered for web+myapp cannot collide with a native app that registers myapp:// at the OS level.

Security implications of scheme validation

The scheme determines what action the browser or server takes with a URL. Allowing user-supplied URLs to contain any scheme is dangerous: javascript:alert(1) executes code when used as an anchor href, data:text/html,<script>... embeds executable HTML, and file:/// accesses the local filesystem. Consequently, when accepting user-supplied URLs, always validate that the scheme is one of the expected values, typically http or https, after parsing the URL with a standards-compliant parser. String-prefix checking (url.startsWith('https')) is insufficient because of percent-encoding and whitespace handling.⁴ CapyToolkit runs all URL parsing examples locally in your browser, so scheme validation happens client-side without sending any input to a server.

WebSocket and real-time communication schemes

The ws:// and wss:// schemes identify WebSocket endpoints, with wss:// providing TLS encryption analogous to the https:// upgrade. A WebSocket URL like wss://api.example.com/realtime follows the same authority and path structure as HTTP URLs, but the browser handles the scheme differently: instead of rendering a page, it opens a persistent bidirectional connection via the WebSocket API. The scheme comparison in JavaScript is straightforward: new URL(socketUrl).protocol === "wss:" tells you whether the connection will be encrypted.⁵

How browsers handle unknown schemes

When a browser encounters a URL with an unregistered scheme (one it cannot render or delegate to an OS handler), it shows an error page or prompts the user to choose an application. This behavior is a security boundary: the browser will not silently pass an unknown-scheme URL to arbitrary code. For web applications that generate URLs with custom schemes, test the fallback experience for users who do not have the corresponding app installed. A common pattern is to redirect to an app store or provide a web-based alternative when the custom scheme fails.

Because the fallback is entirely browser-dependent, there is no standard way to detect scheme support from JavaScript before navigation. Some frameworks attempt a timer-based approach: navigate to the custom scheme, then fall back to a web URL if the page remains visible after a short delay. This heuristic is fragile and can trigger false positives on slow devices, so graceful degradation through explicit user action (a "Open in App" button that navigates to the app store on failure) remains the most reliable pattern.

IANA scheme registration and the difference between standards-tree and private-use schemes

The Internet Assigned Numbers Authority maintains the Uniform Resource Identifier (URI) Schemes registry, which lists every officially registered scheme. Standards-tree schemes (http, https, ftp, mailto) are defined by IETF RFCs and available for general use. Private-use schemes should use the x- prefix per RFC 2396, though many widely adopted schemes (slack, vscode, steam) omit the prefix and register directly with IANA or the relevant platform. The registration process requires an RFC or equivalent specification for standards-tree schemes; vendor-tree schemes need only a contact and description.

Scheme length and character constraints

RFC 3986 Section 3.1 restricts schemes to a letter followed by any combination of letters, digits, plus signs, hyphens, and periods. A scheme like web+myapp: is valid; 3com: is not, because it starts with a digit. The WHATWG URL Standard enforces this rule strictly: new URL("3com://example.com") throws a TypeError.⁶ When designing a custom scheme for an application, start the name with a letter and keep it short; long scheme names are harder to read and more likely to collide with future IANA registrations.

Scheme-relative URLs and protocol-relative link loading

A scheme-relative URL (starting with //) inherits the scheme from the context in which it is loaded. The reference //cdn.example.com/script.js loaded from an HTTPS page becomes https://cdn.example.com/script.js; loaded from an HTTP page, it becomes http://cdn.example.com/script.js. This pattern was widely used for CDN resources before HTTPS-everywhere became the norm, allowing a single reference to work on both HTTP and HTTPS pages without mixed-content warnings.

Why protocol-relative URLs are now deprecated

The protocol-relative pattern is now considered an anti-pattern by the W3C and most web performance guides. A page served over HTTPS that loads a resource via // inherits HTTPS, but a man-in-the-middle attacker on the initial HTTP connection can rewrite the reference to point to a malicious server. Modern best practice is to always use explicit https:// URLs for external resources. The upgrade-insecure-requests Content Security Policy directive can automatically upgrade HTTP references to HTTPS, but explicit https:// URLs are more reliable.⁷ When you encounter // references in legacy code, replace them with https:// during your next maintenance pass.

Try in the tool

What Is Percent-Encoding?

Because URLs must be transmitted as ASCII text without ambiguity, characters that are not allowed in a specific URL component are converted to a hexadecimal representation prefixed by a percent sign. This conversion is called percent-encoding (also URL encoding), and it is what turns a space into %20 or a forward slash inside a query value into %2F.¹

What is percent-encoding?

Which characters to encode and when

Unreserved characters (A–Z, a–z, 0–9, -, ., _, ~) are never encoded because they are safe in every URL position. Reserved characters (:/?#[]@!$&'()*+,;=) have structural meaning in URLs and must be encoded when used as data values rather than delimiters. Consequently, the exact set of characters that require encoding differs by component: a forward slash / is valid in a path but must be encoded as %2F inside a path segment that represents a single resource identifier. Building on this, context-aware encoding functions (encodeURIComponent in JavaScript, quote() in Python) handle these rules automatically.² The encodeURI() function in JavaScript preserves reserved characters since it targets full URLs, while encodeURIComponent() encodes everything except unreserved characters since it targets a single query parameter value.

Encoding versus decoding

Encoding converts a plain character to its %XX form: a space becomes %20, & becomes %26. Decoding reverses the process: %20 becomes a space. Built-in URL parsers decode percent-encoded characters automatically when exposing component values. The call url.searchParams.get('q') returns the decoded string, not the raw encoded form, so you rarely need to call decodeURIComponent() yourself when working with parsed URL objects. Consequently, double-encoding occurs when an already-encoded value is encoded again, turning %20 into %2520 (% encodes to %25, then 20 follows). Always decode a value before re-encoding it for a new URL context. The same double-encoding trap applies when you pass a fully encoded URL as a query parameter value to another URL: encode the value once, and verify the receiving endpoint does not decode it twice.

Percent-encoding versus form encoding

RFC 3986 percent-encoding and HTML form encoding (application/x-www-form-urlencoded) are related but different. Form encoding encodes spaces as + rather than %20, and is the default format for HTML form GET and POST submissions. Building on this, most form submission parsers (PHP's $_GET, Python's parse_qs) decode + as a space. RFC 3986-compliant URLs should use %20 for spaces since a + in a path segment or query string that follows RFC 3986 is a literal plus sign, not a space. Use %20 in API URLs; use + only in form-encoded query strings.² When you parse a URL that contains + in a query value, check whether the sending context was a form submission before deciding whether to decode it as a space.

Percent-encoding in HTTP headers and request bodies

HTTP headers that carry URL values (Location, Content-Location, Referer) must contain properly percent-encoded URLs.³ A Location header with a redirect target containing spaces or non-ASCII characters will cause parsing failures in strict HTTP clients. The correct form is Location: https://example.com/search?q=hello%20world, not Location: https://example.com/search?q=hello world. Most HTTP libraries handle encoding automatically when you pass a URL object, but raw header construction requires manual encoding.

How JSON APIs handle percent-encoded values

When a URL appears as a string value in a JSON API payload, the percent-encoding is preserved as-is in the JSON string. The JSON serializer does not decode %XX sequences; it treats them as literal characters. A response like {"url": "https://example.com?q=hello%20world"} contains the encoded form, and the client must decode the parameter value after extracting it from the JSON. This two-layer encoding (percent-encoding inside JSON) is a common source of bugs when developers forget to call decodeURIComponent() or unquote() on the extracted value.

Because JSON strings must be valid UTF-8, any percent-encoded bytes that would form invalid UTF-8 sequences (such as an isolated %80) must either be decoded before JSON serialization or the entire value must be re-encoded as a base64 string. Most modern frameworks handle this transparently, but custom middleware that logs or transforms request URLs before they reach the JSON serializer can inadvertently produce malformed output if they treat the percent-encoded string as raw text.

Rawurlencode versus urlencode in PHP and Go

PHP provides two encoding functions that differ in space handling: urlencode() encodes spaces as + (form format), while rawurlencode() encodes spaces as %20 (RFC 3986 format).⁴ For path segments, always use rawurlencode(); for query parameter values, either function works as long as the receiving parser matches. Go follows the same pattern: url.QueryEscape() encodes spaces as +, while url.PathEscape() encodes spaces as %20.⁵ The WHATWG URL Standard's encodeURIComponent() in JavaScript always uses %20, making it safe for both path and query contexts.

The safe parameter in Python's quote function

Python's urllib.parse.quote() accepts a safe parameter that specifies characters to leave unencoded. The default safe value is /, meaning forward slashes are preserved in the output. For encoding a path segment (not a full path), pass safe='' to encode slashes as %2F: quote("path/segment", safe='') produces "path%2Fsegment". For query parameter values, the default safe='/' is usually fine since slashes in query values are literal data. Getting the safe parameter wrong is one of the most common sources of double-encoding bugs in Python URL handling.⁶ When you need to encode a full path that contains slashes between segments, call quote() on each segment individually and join them with /, rather than encoding the slashes that separate segments.

Signature verification and canonical request signing

Request signing protocols (AWS Signature V4, OAuth 1.0, HMAC-based API authentication) require precise percent-encoding of the URL components before computing the signature. AWS Signature V4 specifies its own encoding rules: encode every byte except unreserved characters, use uppercase hex digits (%2F not %2f), and encode slashes in the path. A URL like https://example.com/path/to+file.txt encodes the + as %2B in the canonical request, because + is not an unreserved character in the AWS scheme. Using the wrong encoding function or the wrong case for hex digits produces a signature mismatch that is difficult to debug.

Debugging signature mismatches caused by encoding

When a signed request fails verification, the most common cause is an encoding difference between the client and server. Log the canonical request string on both sides and compare character by character. Common mismatches include: the client encoding a space as %20 while the server expects +, the client encoding slashes in the path while the server does not, or the client using lowercase hex digits while the server uses uppercase. Tools like AWS's SigV4 test suite let you verify your canonical request construction against known-good examples. CapyToolkit runs all URL parsing examples locally in your browser, which follows the WHATWG URL Standard encoding rules.

Try in the tool

What Is a URL Fragment?

Because some resources, like HTML documents, contain multiple addressable sections, the URL fragment provides a way to identify a specific portion of the resource after it has been retrieved. Everything after the # in a URL is the fragment, and unlike every other URL component, the fragment is handled entirely in the client and never transmitted to the server.¹

What is a URL fragment?

How browsers handle fragments

When a browser navigates to a URL with a fragment, it first loads the full resource (making an HTTP request without the #section part), then scrolls the viewport to the element whose id or name attribute matches the fragment. Consequently, changing only the fragment by clicking an anchor like <a href="#installation"> does not trigger a new HTTP request. The hashchange event fires in JavaScript when the fragment changes, and window.location.hash returns the fragment including the leading #. Building on this, fragment-only navigation records a new history entry, allowing the back button to return to the previous fragment.² If no element on the page matches the fragment, the browser scrolls to the top of the page and no hashchange event fires for the same fragment value.

Fragment routing in single-page applications

Before the HTML5 History API, SPAs used the fragment as a client-side router: the route /products was stored as /#/products so the page could navigate between views without a server request. Building on this, the server always serves the same HTML file regardless of the fragment, and JavaScript reads window.location.hash to render the correct view. Consequently, modern SPAs use the History API (pushState/replaceState) instead, which changes the full URL path without the # prefix and produces cleaner URLs without the hash, at the cost of requiring server-side configuration to serve the app shell for all paths. The trade-off is that fragment-based routing requires no server changes while History API routing needs a catch-all rewrite rule.

Fragment in APIs and security contexts

OAuth 2.0 implicit grants deliver the access token in the fragment: https://app.example.com/callback#access_token=abc123. Because the fragment is never sent to the server, the token is accessible only to JavaScript on the client, which reduces server-side token exposure. Consequently, this also means the token is visible to any JavaScript on the page, including third-party scripts. The implicit grant is deprecated in favor of the authorization code flow with PKCE for this reason.³ Building on this, fragments are also used in magic link authentication tokens, where the link goes to /verify#token=xyz and the JavaScript sends only the token to the backend, not the full URL.

The fragment and the History API in modern browsers

The HTML5 History API (pushState, replaceState) supersedes fragment-based routing by letting JavaScript change the full URL path without triggering a page reload. history.pushState({}, '', '/dashboard') changes the URL from / to /dashboard without a navigation event. The server is never contacted; the browser simply updates the address bar and history stack. When the user hits Back, the popstate event fires with the previous URL, and your router renders the matching view.⁴ Consequently, the server must be configured to serve your SPA's index.html for all paths the client-side router handles, or users refreshing a deep link will get a 404.

Scroll restoration with the History API

Modern browsers implement automatic scroll restoration for Back and Forward navigation. When using pushState for client-side routing, set history.scrollRestoration = 'manual' in JavaScript to disable the browser's default behavior and implement your own scroll-to-top logic on route change. Without this, the browser restores the previous scroll position when the user navigates Back, which feels jarring in a single-page app where the content has changed. React Router and Vue Router both handle this automatically with their <ScrollRestoration> and scrollBehavior options.

Fragments for deep linking and text fragment URLs

Fragments enable deep linking into specific content within a page without requiring server-side support. A documentation site that renders sections as tabs can store the active tab in the fragment: https://docs.example.com/guide#installation. JavaScript reads window.location.hash to activate the correct tab on page load, and updateFragment('#configuration') pushes a new history entry when the user switches tabs. The server serves the same HTML for all fragment variations; the client-side code reads the fragment and renders the matching section.

Preserving fragment state across page loads

When a user refreshes the page or bookmarks a URL that contains a fragment, the browser preserves that fragment value across the reload, so your JavaScript can read window.location.hash on the next load and restore the previous view without any server round-trip. This behavior makes fragments a lightweight alternative to storing UI state in localStorage or a session cookie for single-page flows where the state is shareable via a link. The trade-off is that a long or opaque fragment value makes the address bar harder to read, and any analytics tool that truncates URLs may group distinct fragment states under the same base path unless you configure it to include the hash.

Because the fragment is purely client-side, it survives not only reloads but also browser restarts when the session is restored. This means a user who bookmarks a specific tab in a documentation site will return to that exact tab days later without any server-side session tracking. For applications that need this persistence without the overhead of a backend, fragments provide a zero-infrastructure solution that works across all browsers without configuration.

Text fragment URLs for linking to specific text

Chrome and Firefox support text fragment URLs that scroll to and highlight specific text on a page: https://example.com/page#:~:text=specific%20text%20to%20highlight. The #:~:text= syntax after the fragment tells the browser to scroll to the first occurrence of the specified text and apply a highlight style.⁵ This feature requires no JavaScript on the page; it is handled entirely by the browser. For pages where you control the content, text fragment URLs provide a way to link to specific paragraphs without adding named anchors or IDs to every element. CapyToolkit runs all URL examples locally in your browser where text fragment support is available.

Try in the tool

What Is URL Authority?

Because a URL must identify both what resource to retrieve and where to retrieve it from, the authority component specifies the server (or other resource owner) responsible for the resource. The authority follows // in the URL and ends at the first / in the path, and it contains the host that the client must connect to.¹

What is a URL authority?

The three sub-components

Userinfo precedes the @ sign and carries credentials: user:password@. RFC 3986 deprecated embedding passwords in URLs because the practice exposes credentials in logs, history, and referrer headers. Consequently, the host sub-component is a registered domain name (example.com), an IPv4 address (192.168.1.1), or an IPv6 address in brackets ([::1]). The port follows a colon after the host: :8080. Omitting the port means the default port for the scheme applies, 80 for http:// and 443 for https://. URL parsers strip default ports from the authority string during normalization.² When you parse a URL that includes userinfo, always verify that the credentials are intentional before forwarding the URL to another service, since leaking credentials through accidental forwarding is a common source of security incidents.

How parsers expose the authority

Different parsers expose the authority and its sub-components differently. In JavaScript, url.host returns host:port (e.g., "example.com:8080"), while url.hostname returns just the domain name without the port, and url.port returns the port as a string (empty for the default port). Consequently, url.origin combines scheme and authority: "https://example.com:8080". In Python's urlparse, the netloc field contains the full authority including userinfo and port; the .hostname, .port, .username, and .password properties extract each sub-component. Building on this, Go's url.URL struct provides .Host (hostname+port as one string) and .Hostname(), .Port() methods.³ When you build logic that compares authorities across different platforms, always normalize the port first since one parser may strip a default port while another preserves it, producing string mismatches that are difficult to debug.

Authority in security contexts

The authority is the primary determinant of the same-origin policy in browsers, so two URLs are same-origin if and only if their scheme, host, and port all match.⁴ Consequently, https://example.com and http://example.com are cross-origin because the scheme differs. https://example.com and https://www.example.com are cross-origin because the host differs. Building on this, the authority is also what makes URL parsing security-critical: a URL like https://[email protected]/ has evil.com as the host and safe.com as userinfo, so a string check for "safe.com" in the URL would pass, but the connection goes to evil.com. Always parse user-supplied URLs with a standards-compliant parser and inspect the host property directly, rather than searching for domain names in the raw string.

Authority in CORS and cross-origin resource sharing

Cross-origin resource sharing relies on the authority as its primary comparison key. When a browser sends a cross-origin request, the server's Access-Control-Allow-Origin response header must match the request's origin, which is the scheme plus authority. A header value of https://example.com does not match https://example.com:443 even though they refer to the same default port, because the origin comparison is string-based after normalization. Wildcard values in Access-Control-Allow-Origin cannot be used with credentials; the server must echo the exact origin back to the browser.⁵

How preflight requests use the authority

For requests that trigger a CORS preflight (PUT, DELETE, or requests with custom headers), the browser sends an OPTIONS request with an Origin header containing the full authority. The server must respond with Access-Control-Allow-Origin matching that exact authority, plus any additional permitted methods and headers. A misconfigured server that returns Access-Control-Allow-Origin: https://example.com when the actual origin is https://www.example.com causes the preflight to fail silently, and the browser blocks the actual request without surfacing the reason in most developer tools.

Default ports and authority normalization across parsers

Default port handling varies across URL parsing libraries in ways that affect authority comparison. JavaScript's URL class strips the default port from url.host and url.origin: new URL("https://example.com:443/path").origin returns "https://example.com" without the port. Python's urlparse preserves the port in netloc even when it is the default: urlparse("https://example.com:443/path").netloc returns "example.com:443".⁶ Go's url.URL.Host includes the port only when it is explicitly present in the input string, regardless of whether it is the default.

Why port normalization matters for URL comparison

When comparing two URLs for equality, the port handling difference between parsers can produce false negatives. A URL constructed with an explicit :443 and one without it refer to the same resource, but a naive string comparison of the authority component will disagree. Normalize both URLs before comparison: strip default ports (80 for HTTP, 443 for HTTPS) from the authority, then compare the remaining strings. Most URL libraries provide a normalized form, but the normalization rules are not identical across languages, so document which parser your comparison logic uses.

This normalization is especially important in security-sensitive contexts like CORS origin validation and CSP source matching, where a port mismatch can incorrectly block a legitimate request or allow an unauthorized one. For example, a CSP directive of script-src https://example.com should match both https://example.com and https://example.com:443, but a parser that preserves the explicit port will fail the match unless normalization is applied. Building a small canonicalization helper that strips default ports based on the scheme ensures consistent behavior across your application regardless of which parser produced the URL object.

Internationalized domain names in the authority component

Internationalized domain names introduce encoding complexity into the authority component. A URL like https://münchen.de/path stores the Unicode hostname in the authority, but DNS resolution requires the ASCII-compatible encoding (xn--mnchen-3ya.de). The IDNA 2008 standard (RFC 5891) defines the conversion algorithm, and most URL parsers apply it automatically when you access the hostname property.⁷ JavaScript's new URL("https://münchen.de").hostname returns "xn--mnchen-3ya.de" in the Punycode form.

IDNA 2003 versus IDNA 2008 differences

Not all parsers use the same IDNA standard. Python's encodings.idna module implements IDNA 2003, which converts the German sharp s (ß) to "ss". The WHATWG URL Standard and IDNA 2008 preserve ß as a distinct character in the encoded domain. This means https://straße.de encodes differently depending on the parser: IDNA 2003 produces xn--strae-oqa.de while IDNA 2008 produces xn--strae-oqa.de (the same in this case, but different for other characters). For URL comparison across systems, normalize both hostnames using the same IDNA standard before comparing. CapyToolkit runs all URL parsing examples locally in your browser, which follows the WHATWG URL Standard. When your application accepts user-supplied URLs with internationalized hostnames, always normalize the hostname to a consistent IDNA form before storing or comparing it, since the same visual domain can produce different encoded forms across parsers.

Try in the tool