URL Parser & Inspector: Code Examples

URL Parsing in JavaScript

JavaScript's built-in URL constructor parses any valid URL into its components. URLSearchParams handles query strings with methods like get(), getAll(), and append(). No external libraries needed.¹

Core parsing API

JavaScript's URL constructor accepts any valid URL string and returns an object with 12 named properties. Passing https://api.example.com:8080/v1/users?page=2#top gives you .protocol ("https:"), .hostname ("api.example.com"), .port ("8080"), .pathname ("/v1/users"), .search ("?page=2"), and .hash ("#top"). The .origin property combines scheme, hostname, and port, which makes it useful for same-origin comparisons in security-sensitive code. The .href property returns the normalized full URL, which the parser may have canonicalized slightly from the input by lowercasing the scheme and host or by resolving dot-segments in the path. For same-origin checks, compare .origin values rather than reconstructing strings from individual components, because subtle encoding differences can otherwise cause two equivalent URLs to look unequal.²

Query string and parameter handling

Accessing url.searchParams returns a live URLSearchParams object that stays synchronized with the URL. Get a single parameter with .get("key"), which returns null if the key is absent. For repeated keys, such as ?tag=js&tag=astro, use .getAll("key") to retrieve all values as an array. The .set() method replaces every existing value for a key with a single new value, while .append() adds another value to the list without removing the ones already there. Call .toString() to serialize the params back to a query string without the leading question mark, which you can then assign directly to url.search or concatenate onto an existing URL. Iterating with for...of yields every key-value pair in insertion order, including duplicates that would collapse in a plain object.³

Edge cases, encoding, and pitfalls

Relative URLs require a base: new URL("/about") throws a TypeError, but new URL("/about", "https://example.com") resolves correctly. Protocol-relative inputs like //example.com/path are not valid, because the constructor refuses to guess which scheme you intended and throws instead. The URL constructor handles percent-encoding automatically: %20 in a pathname is preserved as-is, while a literal space in the input is encoded to %20 on the normalized .href output. For non-ASCII hostnames, the URL API applies UTS #46 IDNA Compatibility Processing, which converts names like münchen.de to their ASCII-compatible punycode encoding. Paths containing non-ASCII characters are percent-encoded following the WHATWG URL Standard rather than RFC 3986, so expect subtle differences when interoperating with older parsers.⁴

Building URLs from components with the URL constructor

The URL constructor accepts only complete URL strings, not individual components. To build a URL from parts, construct the full string yourself: const url = new URL(https://${host}${path}). Consequently, the constructor normalizes the result immediately: it lowercases the scheme and host, removes default ports, and resolves dot-segments in the path. For dynamic query parameters, mutate url.searchParams after construction rather than concatenating query strings by hand. Calling url.searchParams.set('key', 'value') and then url.href gives you a correctly encoded URL without manual percent-encoding.

Validating URLs before use in fetch requests

Wrapping new URL(input) in a try/catch is the standard validation pattern, but it does not guarantee the URL points to an allowed host. For fetch requests built from user input, parse the URL first, then check the hostname against an allowlist before calling fetch. A URL that parses successfully can still point to an internal IP address or a domain you did not intend to reach. Adding a hostname check after parsing prevents accidental requests to internal services when the input comes from untrusted sources.

A practical pattern is to keep an allowlist of permitted hosts in a configuration file and load it at startup so the check stays current without code changes. Combine the allowlist with a logged rejection so that unexpected redirect targets are visible during incident review. For applications that accept URLs from many sources, store the allowlist in a shared configuration service so every service validates against the same rules. This keeps the security boundary consistent and makes it easier to audit which hosts your code is allowed to contact.

URLSearchParams iteration and serialization patterns

URLSearchParams is iterable, which means for (const [key, value] of url.searchParams) gives you every key-value pair in insertion order. Building on this, Object.fromEntries(url.searchParams) converts the params to a plain object, but repeated keys collapse to the last value only. For APIs that send repeated keys as arrays, use url.searchParams.getAll('key') instead. When serializing back to a string, url.searchParams.toString() produces a properly encoded query string without the leading question mark, ready for concatenation into a full URL. Converting the entries to an array with Array.from(url.searchParams) preserves every duplicate pair, which is useful when you need to round-trip query parameters through a form or a log without losing repeated values.⁵

Preserving URLSearchParams through JSON serialization

URLSearchParams objects do not survive JSON.stringify and JSON.parse. If you need to store or transmit parsed query parameters as JSON, convert them first: const data = Object.fromEntries(params) for single-value access, or const data = Array.from(params) for a full list of pairs including duplicates. Reconstruct with new URLSearchParams(data) after parsing the JSON. This pattern is common in analytics pipelines where query parameters are extracted, serialized to a message queue, and reparsed by a downstream consumer.⁶

The WHATWG URL Standard versus RFC 3986 in JavaScript

JavaScript's URL class implements the WHATWG URL Standard, not RFC 3986 directly. The differences matter in edge cases: WHATWG percent-encodes the path more aggressively, rejects some characters that RFC 3986 allows in the query component, and applies Unicode IDNA Compatibility Processing (UTS #46) to internationalized hostnames rather than IDNA 2003 or IDNA 2008. For most HTTPS URLs the two standards produce identical results, but URLs with non-ASCII paths, unusual schemes, or legacy encoding may parse differently. When interoperability with RFC 3986-based systems matters (such as OAuth signature generation), test your URL handling against the specific standard the other system implements.³

Handling file:// and custom scheme URLs

The URL constructor handles file:// URLs without special treatment: new URL('file:///etc/hosts') parses with protocol 'file:', hostname '', and path '/etc/hosts'. Custom schemes like myapp://resource/123 parse the same way, giving you access to .hostname, .pathname, and .searchParams. However, the URL class does not register custom scheme handlers; it only parses the string. For Electron or Node.js applications that need to handle custom protocol URLs, parse the URL to extract the resource path and dispatch to your handler based on the scheme value.

Notes

new URL("https://example.com/path?key=value#hash") gives you .hostname, .pathname, .search, .hash, .origin, and more. Use .searchParams.get("key") to read individual query parameters. The URL API is supported in all modern browsers and Node.js.

Examples

const url = new URL("https://example.com:8080/path/page?q=hello#section");
console.log(url.hostname);   // "example.com"
console.log(url.pathname);   // "/path/page"
console.log(url.search);     // "?q=hello"
console.log(url.hash);       // "#section"
console.log(url.port);       // "8080"

const params = new URLSearchParams("?page=1&size=20");
console.log(params.get("page"));   // "1"
params.set("size", "50");
params.append("sort", "name");
console.log(params.toString());    // "page=1&size=50&sort=name"

const base = "https://example.com/blog/post";
const resolved = new URL("../about", base);
console.log(resolved.href);
// "https://example.com/about"

const url = new URL("https://cdn.example.com/images/photo.jpg");
const segments = url.pathname.split("/");
const filename = segments[segments.length - 1];
console.log(filename);  // "photo.jpg"

const url = new URL("https://example.com:8080/path/page?q=hello#section");
console.log(url.hostname);   // "example.com"
console.log(url.pathname);   // "/path/page"
console.log(url.search);     // "?q=hello"
console.log(url.hash);       // "#section"
console.log(url.port);       // "8080"

URL Parsing in Python

Python's urllib.parse module provides urlparse(), parse_qs(), and urlunparse() to decompose and reconstruct URLs. It handles all standard URL formats including query strings with multiple values.¹

Core parsing API

Calling urlparse("https://user:[email protected]:8080/path?q=1#section") returns a ParseResult named tuple with six fields: scheme, netloc, path, params, query, and fragment. The netloc field contains the full authority including userinfo and port; to extract just the hostname, use the .hostname property or split netloc manually on the colon. The params field captures the semicolon-separated parameter segment of old-style URLs, which is almost never used in practice because modern query strings use the ? delimiter instead. For most code, urlsplit offers a cleaner five-field alternative that merges params into the path, and it is the safer default unless you specifically need to detect the rare semicolon parameter syntax.²

Query string and parameter handling

Parse a query string into a dict of lists using parse_qs(parsed.query). Because a URL can repeat the same key multiple times, such as ?tag=js&tag=python, parse_qs always returns lists, so params["tag"] yields ["js", "python"]. The parse_qs function skips keys with empty values by default; pass keep_blank_values=True to retain them, which is useful when a form field submits an empty string you want to distinguish from a missing field. For iteration that preserves duplicates, parse_qsl returns a flat list of (key, value) pairs in insertion order. Rebuilding a query string from a dict uses urlencode({'key': 'val', 'page': 1}), which percent-encodes unsafe characters automatically and handles spaces as plus signs by default.³

Edge cases, encoding, and pitfalls

Resolving a relative URL requires urljoin: urljoin("https://example.com/blog/post", "../about") returns "https://example.com/about". Passing a bare relative path to urlparse does not raise an error; it silently assigns everything to path, leaving scheme and netloc empty, which is a common source of bugs when code later assumes parsed.scheme is set. Percent-encoding and decoding live in separate functions: quote(text) encodes a string for safe use in a URL segment, unquote(text) decodes it back, and quote_plus uses + for spaces instead of %20 to match the application/x-www-form-urlencoded format used by HTML form submissions. Always decode percent-encoded values before displaying them to users, because showing raw %2E or %20 sequences in a UI confuses anyone who is not reading URLs at the byte level.⁴

Reconstructing URLs with urlunparse and urlunsplit

urlunparse takes a six-element tuple (scheme, netloc, path, params, query, fragment) and returns a complete URL string. Building on this, urlunsplit takes a five-element tuple without the params field, matching urlsplit's output. Both functions handle empty components gracefully: passing an empty string for the query produces a URL without a question mark, while omitting the query entirely in the tuple has the same effect. For programmatic URL construction, build a list or tuple of components, modify the relevant index, and pass it to urlunparse. This avoids string concatenation bugs where a missing slash or extra colon breaks the URL structure.²

When to use urlencode versus manual query string building

urllib.parse.urlencode converts a dict or list of tuples into a percent-encoded query string. Passing doseq=True handles list values correctly: urlencode({'tag': ['js', 'python']}, doseq=True) produces 'tag=js&tag=python'. Without doseq, the list is stringified as "['js', 'python']", which is almost never what you want. For query strings that require a specific parameter ordering (some APIs sort parameters for signature generation), use a list of tuples instead of a dict to preserve insertion order. Manual string building with f-strings is error-prone: you must remember to encode each value, handle the & separator, and deal with empty values.³

Another pitfall is mixing urlencode output with already encoded values, which produces a double encoded string that the server decodes incorrectly. Prefer a single code path for query construction so that every value passes through urlencode exactly once. When the same query string is built in more than one place, extract the logic into a shared helper and call it from every endpoint. This removes the chance that one code path forgets to encode and silently ships a malformed parameter to the server.

Internationalized domain names and IDNA encoding

Python's urllib.parse does not automatically apply IDNA encoding to internationalized domain names. Passing 'https://münchen.de/path' to urlparse returns the Unicode hostname as-is, which cannot be used directly for DNS resolution. To convert to ASCII-compatible encoding, call hostname.encode('idna').decode('ascii'), which produces the ASCII punycode form that always begins with a four-character ace prefix followed by a hyphen. For URLs already in the URL string, extract the hostname first, encode it, and reconstruct the URL with the ASCII hostname. The encodings.idna module handles this in older Python versions, but the built-in str.encode('idna') method is the standard approach in Python 3.⁵

Handling data: and javascript: scheme URLs

urlparse parses data: and javascript: scheme URLs without validation. A string like 'javascript:alert(1)' parses with scheme 'javascript' and path 'alert(1)', which looks harmless if you only inspect the path. When accepting user-supplied URLs, always validate the scheme after parsing by checking that parsed.scheme is in an allowlist of permitted schemes such as 'http' and 'https'. String-prefix checks are insufficient because a URL like 'javascript%3Aalert(1)' might pass a naive string check but parse differently after decoding. Parse first, then validate the parsed scheme, because relying on string prefixes alone leaves a window for encoded payloads to slip through.⁶

Thread safety and performance of urllib.parse

All urllib.parse functions are pure: they take string inputs and return new objects without modifying any module-level state. This makes them safe to call from multiple threads simultaneously without locks. The module is implemented in pure Python, but parses thousands of URLs per second in typical CPython workloads and is fast enough for most applications. The main performance consideration is object allocation: each urlparse call creates a new ParseResult named tuple. If you only need one component (such as the hostname), extracting it from the full ParseResult still allocates the entire tuple. For extreme performance, a compiled regex or a C extension that extracts only the needed component avoids the allocation overhead.¹

Testing URL parsing logic with edge case inputs

Unit tests for URL parsing should cover: URLs with no scheme (urlparse assigns everything to path), URLs with an empty host (http:///path), URLs with percent-encoded characters at segment boundaries, and URLs with query strings containing encoded ampersands and equals signs. Python's unittest or pytest parametrize decorator makes it easy to run the same parsing logic against a table of input URLs and expected component values. Include at least one IDN hostname, one IPv6 address, and one URL with a non-standard port in your test suite. These edge cases catch the most common parsing bugs before they reach production.

Notes

Use urlparse() to split a URL into a named tuple with scheme, netloc, path, params, query, and fragment. Use parse_qs() for query parameters as a dict of lists. Use urlunparse() to rebuild a URL from components. For Python 3, import from urllib.parse.

Examples

from urllib.parse import urlparse
parsed = urlparse("https://example.com:8080/path/page?q=hello#section")
print(parsed.scheme)   # https
print(parsed.netloc)   # example.com:8080
print(parsed.path)     # /path/page
print(parsed.query)    # q=hello
print(parsed.fragment) # section

from urllib.parse import urlparse, parse_qs
parsed = urlparse("https://example.com?name=alice&name=bob&page=2")
params = parse_qs(parsed.query)
print(params["name"])  # ["alice", "bob"]
print(params["page"])  # ["2"]
# parse_qs returns a dict of lists (handles multi-value params)

from urllib.parse import urlunparse
parts = ("https", "example.com", "/new/path", "", "key=value", "bottom")
url = urlunparse(parts)
print(url)
# https://example.com/new/path?key=value#bottom

from urllib.parse import quote, unquote
encoded = quote("hello world & goodbye")
print(encoded)   # hello%20world%20%26%20goodbye
decoded = unquote(encoded)
print(decoded)   # hello world & goodbye

from urllib.parse import urlparse
parsed = urlparse("https://example.com:8080/path/page?q=hello#section")
print(parsed.scheme)   # https
print(parsed.netloc)   # example.com:8080
print(parsed.path)     # /path/page
print(parsed.query)    # q=hello
print(parsed.fragment) # section

URL Parsing in Go

Go's net/url package provides robust URL parsing through the url.URL struct. Parse full URLs or relative references, access individual components, build query strings, and handle percent-encoding correctly.¹

Core parsing API

Calling url.Parse(rawURL) returns a *url.URL struct and an error. The struct exposes Scheme, Host (hostname plus port), Path (decoded), RawPath (percent-encoded original), RawQuery (raw query string), and Fragment. The Opaque field captures the scheme-specific portion of URIs like mailto:[email protected] where there is no authority component to parse into separate host and path fields. For most HTTP URLs, accessing u.Hostname() returns just the host without the port, and u.Port() returns the port number as a string that is empty when the standard port for the scheme is in use. Always check the error return value before dereferencing the pointer, because url.Parse returns a nil URL when the input cannot be parsed as a valid URI.²

Query string and parameter handling

Calling u.Query() parses the raw query string into a url.Values, which is a map[string][]string. Get the first value with q.Get("key"), which returns an empty string if the key is absent rather than a nil slice. All values for a key are stored as a slice, so q["tag"] returns []string{"js", "astro"} for a URL with two tag parameters. To build a query string from code, create a url.Values with url.Values{"key": {"val"}}, call .Set() or .Add() to modify it, then assign its .Encode() result to u.RawQuery before calling u.String(). The .Encode() method percent-encodes every value and sorts the keys alphabetically, which makes the output deterministic and safe to cache or compare.³

Edge cases, encoding, and pitfalls

Path contains the decoded path, but the original encoding is lost unless RawPath is non-empty, because url.Parse sets RawPath only when the path contains a percent-encoded segment that differs from the decoded form. Setting u.Path = "/new path" is safe, because calling u.String() re-encodes it correctly and produces a properly escaped result without manual escaping on your part. For query parameter values, url.QueryEscape uses + for spaces while url.PathEscape uses %20, and mixing them in the wrong context produces subtle bugs when a server expects one format but receives the other. Resolving a relative URL uses u.ResolveReference(rel), which mutates nothing and returns a new *url.URL that follows the RFC 3986 Section 5.2 algorithm exactly.⁴

Building URLs with the url.URL struct and url.Values

Construct a url.URL struct with the fields you need, then call .String() to serialize the complete URL including any modifications you have made to its components. For query parameters, build a url.Values map, populate it with .Set() and .Add(), then assign its .Encode() result to the RawQuery field. Building on this, the difference between .Set() and .Add() matters for multi-value parameters: .Set('tag', 'go') replaces all existing values, while .Add('tag', 'web') appends a new value alongside existing ones. Calling .Encode() on a url.Values with multiple tag values produces 'tag=go&tag=web', which is the correct format for APIs that accept repeated keys. Always assign to RawQuery rather than trying to build the query string manually, because manual construction easily misses edge cases like encoding reserved characters or handling empty values correctly.³

Escaping and unescaping path segments correctly

url.PathEscape encodes a string for safe use in a URL path segment, converting spaces to percent-encoded sequences and preserving reserved characters that are legal inside paths. The complementary url.PathUnescape reverses the encoding, restoring the original string from its percent-encoded form. When building a URL from user-supplied file names or dynamic path segments, always run each segment through url.PathEscape before assigning it to u.Path, because a segment containing a slash or a percent sign that is not escaped will change the structure of the resulting URL in unexpected ways. For query parameter values, use url.QueryEscape instead, which encodes spaces as plus signs to match the application/x-www-form-urlencoded convention that most form submissions rely on.

A frequent mistake is reusing url.QueryEscape for path segments because the resulting plus signs change path meaning. Keep the two escape functions separate and apply each only to the component it was designed for. When you build a URL from many parts, encode each segment with the matching function before you assemble the final string. This discipline prevents the subtle class of bug where an encoded space becomes a literal plus and the server reads two values instead of one.

Handling opaque URIs and non-HTTP schemes

Go's url.URL struct handles schemes without an authority component through the Opaque field. A URI like 'mailto:[email protected]' has Scheme 'mailto' and Opaque '[email protected]' with no Host, Path, or Port. Calling .String() on an opaque URI reconstructs it as 'mailto:[email protected]' without adding a // authority separator. For 'urn:isbn:0451450523', the Opaque field holds 'isbn:0451450523'. Building on this, when your code handles arbitrary URIs (not just HTTP URLs), check the Opaque field before accessing Host or Path. A nil pointer dereference on .Host for a mailto: URL is a common panic in Go code that assumes all URLs have an authority component.²

Comparing URLs for equality in Go

Two *url.URL values that look identical when printed may differ internally because one has a Host field while the other stores the same authority in the Opaque field. The simplest way to compare two URLs is to call u.String() on each and compare the resulting strings, because the serialized form normalizes scheme casing, host casing, and default port omission. For comparisons that need to treat equivalent hosts as equal regardless of representation, extract u.Hostname() and u.Port() separately and compare those values after canonicalizing the port to the scheme default when it is empty. Be aware that url.URL structs produced by url.Parse may retain fields like Userinfo or ForceQuery that affect equality checks even when they are not visible in casual inspection of the .String() output.

Parsing URLs from HTTP request contexts in Go

When handling HTTP requests in Go, the net/http server parses the request URL into a url.URL struct available as req.URL. This URL contains only the path and query from the request line, because the scheme and host come from the TLS connection and the Host header rather than from req.URL. To reconstruct the full URL a client used, combine the scheme (from req.TLS != nil), the host (from req.Host), and the path and query (from req.URL).

Reconstructing the full request URL from scheme and host

Building on this, req.URL.RawQuery contains the raw encoded query string, while req.URL.Query() returns a parsed url.Values. For proxy servers that forward requests, the incoming request URL may contain an absolute target (https://example.com/path) per HTTP proxy protocol, so check req.URL.IsAbs() and handle the absolute form differently from the relative path form. The net/http package also exposes req.RequestURI, which holds the unmodified request-target from the request line and is useful for logging and debugging the original client intent. To reconstruct the full URL a client used, combine the scheme (from req.TLS != nil), the host (from req.Host), and the path and query (from req.URL).¹

Testing URL parsing in Go with table-driven tests

Go's testing package makes table-driven tests the standard approach for URL parsing validation. Define a slice of structs with input URL strings and expected component values, then loop through them in a single test function. Building on this, cover cases like IPv6 hosts in brackets, internationalized domain names, URLs with empty query values, and URLs containing encoded slashes in the path. The net/url package is thoroughly tested in the Go standard library itself, but your application logic around parsed URLs (such as hostname allowlisting or path normalization) needs its own test coverage. Use t.Run() for subtests so failures report the specific input URL that caused the error.

Notes

Use url.Parse() to parse a URL string into a url.URL struct. Access .Scheme, .Host, .Path, .RawQuery, .Fragment. Use .Query() to get a url.Values map of parsed query parameters. Use .String() to reconstruct the full URL. PathEscape() and PathUnescape() handle percent-encoding for path segments.

Examples

package main

import (
	"fmt"
	"net/url"
)

func main() {
	u, _ := url.Parse("https://example.com:8080/path/page?q=hello#section")
	fmt.Println("Scheme:", u.Scheme)
	fmt.Println("Host:", u.Host)
	fmt.Println("Path:", u.Path)
	fmt.Println("RawQuery:", u.RawQuery)
	fmt.Println("Fragment:", u.Fragment)
}

u, _ := url.Parse("https://example.com?name=alice&page=2")
q := u.Query()
fmt.Println(q.Get("name"))    // alice
fmt.Println(q.Get("page"))     // 2
fmt.Println(q["name"])         // [alice] — direct access returns slice

u := &url.URL{
	Scheme: "https",
	Host:   "example.com",
	Path:   "/search",
}
q := u.Query()
q.Set("q", "golang url parsing")
q.Add("page", "1")
u.RawQuery = q.Encode()
fmt.Println(u.String())

package main

import (
	"fmt"
	"net/url"
)

func main() {
	path := url.PathEscape("files/my document.pdf")
	fmt.Println(path)  // files/my%20document.pdf
	original, _ := url.PathUnescape(path)
	fmt.Println(original)  // files/my document.pdf
}

package main

import (
	"fmt"
	"net/url"
)

func main() {
	u, _ := url.Parse("https://example.com:8080/path/page?q=hello#section")
	fmt.Println("Scheme:", u.Scheme)
	fmt.Println("Host:", u.Host)
	fmt.Println("Path:", u.Path)
	fmt.Println("RawQuery:", u.RawQuery)
	fmt.Println("Fragment:", u.Fragment)
}

URL Structure: RFC 3986 Explained

RFC 3986 defines the generic syntax for Uniform Resource Identifiers. Every URL follows the same structure: scheme://authority/path?query#fragment. Understanding these components helps with parsing, building, and debugging URLs correctly.¹

Authority and host components

Inside the authority component, the structure is userinfo@host:port. The userinfo sub-component, rarely seen in practice, carries credentials before the @ sign. RFC 3986 deprecated embedding passwords in URLs for security reasons, but the format remains valid. Consequently, the host sub-component is either a domain name, an IPv4 address like 192.168.1.1, or an IPv6 address enclosed in brackets like [::1]. The port is numeric and optional; parsers strip default ports (80 for HTTP, 443 for HTTPS) during normalization. A parser that needs to handle both IPv4 and IPv6 hosts must account for the bracket notation that IPv6 addresses require.²

Path, query, and fragment

Path segments are separated by forward slashes and are hierarchical. Dot-segments, . (current) and .. (parent), are resolved by the reference resolution algorithm in Section 5.2: /a/b/../c becomes /a/c. Building on this, the query component follows the ? and has no defined internal structure in RFC 3986; the application-level convention of key=value pairs separated by & is specified by the HTML and WHATWG URL Standards, not RFC 3986 itself. The fragment, following #, is processed entirely client-side and never sent to the server. When building URLs with user-supplied paths, always validate that dot-segment resolution does not escape the intended directory boundary.³

Percent-encoding and normalization

Percent-encoding represents a byte as %XX where XX is the byte's hexadecimal value. Unreserved characters (A-Z, a-z, 0-9, hyphen, period, underscore, tilde) must never be encoded; reserved characters must be encoded when used as literal data rather than delimiters. RFC 3986 Section 6 defines normalization: case-normalize percent-encoded triplets to uppercase, decode unreserved characters, and remove default ports. Consequently, two URLs that look different may be equivalent after normalization, so comparison should be done on the normalized forms. Implementing all three normalization steps ensures that your URL comparison logic treats semantically equivalent URLs as equal, which is essential for caching, deduplication, and link resolution in web crawlers.¹

URI reference types and resolution contexts

RFC 3986 defines four types of URI reference: absolute URIs (with scheme), scheme-relative references (starting with //), relative references (starting with / or a path segment), and empty references. A parser must determine the type to apply the correct resolution algorithm. Building on this, Section 5.2 defines the reference resolution algorithm: parse the base URI and the reference, then merge components according to the reference type. An absolute reference replaces the entire base. A scheme-relative reference inherits only the scheme. A path-relative reference is merged at the path level after removing the base URI's last path segment. Understanding these four types explains why urljoin in Python and new URL in JavaScript produce different results for the same inputs: they implement the same algorithm but may classify edge-case references differently.²

Percent-encoding normalization in practice

Percent-encoding normalization goes beyond simply encoding reserved characters. Section 6 of RFC 3986 defines three normalization steps that make equivalent URLs compare as equal. First, percent-encoded triplets are case-normalized to uppercase, so %2f and %2F are treated identically. Second, unreserved characters that were unnecessarily percent-encoded are decoded, since %41 and A represent the same octet. Third, default ports are removed from the authority component, because https://example.com:443/ and https://example.com/ identify the same resource. Implementing all three steps ensures that your URL comparison logic treats semantically equivalent URLs as equal, which is essential for caching, deduplication, and link resolution in web crawlers.

These normalization steps are safe to apply only when both URLs are controlled by the same authority or trust boundary. Skipping port removal or case folding can cause a crawler to revisit the same page or miss a genuinely distinct resource. Before you collapse two URLs into one canonical form, confirm that they resolve to the same server and the same content. Applying aggressive normalization across unrelated hosts can merge pages that should remain separate in your index.

Same-document references and the empty fragment

A URI reference of # (empty fragment) refers to the current document with the fragment cleared. A reference of '' (empty string) refers to the current document entirely, per Section 5.3. Building on this, these edge cases matter in web applications: setting window.location.hash = '' in JavaScript navigates to the same page without a fragment, while setting it to '#' navigates to the same page with an empty fragment. Both trigger a hashchange event. For server-side redirect handling, a Location header value of '#' is technically valid but meaningless; most servers should return an absolute URL instead. When implementing a redirect endpoint, validate that the target URL is absolute and has a valid scheme before issuing the redirect.⁴

Scheme-specific syntax and the hier-part

The hier-part of a URI follows the scheme and colon and takes one of two forms: //authority/path for URIs with an authority component, or path for scheme-specific URIs without one. The authority component is the most common case for web URLs and contains the userinfo, host, and port sub-components. For schemes like mailto, tel, and data, the hier-part is scheme-specific and does not use the double-slash prefix. A parser must not assume that every URI has an authority component, because attempting to extract a host from a mailto: URI would produce incorrect results. When building a URL, choose the hier-part form based on the scheme: use //authority for http, https, ftp, and ws, and use the scheme-specific path form for mailto, tel, data, and other non-authority schemes.

Registered names, IP addresses, and future IP versions

RFC 3986 Section 3.2.2 defines the host sub-component as either a registered name (domain), an IPv4 address, or an IPv6 address (or future IP version) enclosed in brackets. The bracket notation for IPv6 addresses prevents ambiguity with the port separator colon: [::1]:8080 has host ::1 and port 8080. Building on this, the RFC reserves the bracket syntax for future IP versions (IPv7 and beyond), so parsers should accept any content within brackets as a valid host without requiring it to be a specific IP format.

Internationalized domain names and IDNA encoding

For internationalized domain names, RFC 3986 does not define IDNA encoding; that is handled by separate standards (RFC 5891 for IDNA 2008). A parser that needs to resolve hostnames must apply IDNA encoding after parsing and before DNS lookup. For URL comparison, normalize the host to lowercase and apply IDNA encoding before comparing. When your application accepts user-supplied hostnames with non-ASCII characters, always apply IDNA encoding before DNS lookup to prevent homograph attacks that use visually similar characters from different scripts.⁵

Notes

The generic syntax is: scheme ":" hier-part [ "?" query ] [ "#" fragment ]. The authority component contains userinfo@host:port. Percent-encoding uses %XX where XX is the hexadecimal byte value. Reserved characters (:/?#[]@!$&'()*+,;=) have special meaning and must be encoded when used as data. Unreserved characters (A-Z a-z 0-9 - . _ ~) never need encoding.

Examples

https://user:[email protected]:8080/path/to/page?search=term&page=2#section

Scheme:     https
Authority:  user:[email protected]:8080
  Userinfo:  user:pass
  Host:      example.com
  Port:      8080
Path:       /path/to/page
Query:      search=term&page=2
Fragment:   section

Space → %20
Exclamation → %21
Hash → %23
Dollar → %24
Ampersand → %26
Apostrophe → %27
Left paren → %28
Right paren → %29
Asterisk → %2A
Plus → %2B
Comma → %2C
Slash → %2F
Colon → %3A
Semicolon → %3B
Equals → %3D
Question → %3F
At sign → %40
Bracket → %5B
Right bracket → %5D

Reserved (have special meaning in URIs):
  : / ? # [ ] @ ! $ & \' ( ) * + , ; =
Unreserved (never need encoding):
  A-Z a-z 0-9 - . _ ~
All other characters must be percent-encoded.

Original:  https://example.com:443/../path/../other/../file?q=1
Normalized: https://example.com/file?q=1
Rules: Remove default port (:443 for https, :80 for http). Resolve dot-segments (/../ and /./). Remove empty query parameters.

https://user:[email protected]:8080/path/to/page?search=term&page=2#section

Scheme:     https
Authority:  user:[email protected]:8080
  Userinfo:  user:pass
  Host:      example.com
  Port:      8080
Path:       /path/to/page
Query:      search=term&page=2
Fragment:   section

URL Parsing in PHP

PHP parses URLs with parse_url(), a function that returns an associative array of components.¹ Inside that array you find the scheme, host, port, user, pass, path, query, and fragment keys, and only the keys present in the URL are returned. Parsing query strings into variables uses parse_str(), which writes directly into a named array.² Together with http_build_query() for encoding and urlencode() or rawurlencode() for individual values, PHP gives you the full URL manipulation toolkit without any extensions or Composer packages.³

Core parsing API

Calling parse_url("https://user:[email protected]:8080/path?q=1#top") returns an associative array: ["scheme"=>"https", "host"=>"example.com", "port"=>8080, "user"=>"user", "pass"=>"pass", "path"=>"/path", "query"=>"q=1", "fragment"=>"top"]. Passing a component constant as the second argument returns just that part, so parse_url($url, PHP_URL_HOST) returns the host string directly. The port key holds an integer, not a string, and is the only key that is not a string in the returned array.¹

When you only need one part of a URL, pass the appropriate PHP_URL_* constant as the second argument to parse_url. This avoids destructuring the full array and makes the intent of your code clearer. The available constants include PHP_URL_SCHEME, PHP_URL_HOST, PHP_URL_PORT, PHP_URL_USER, PHP_URL_PASS, PHP_URL_PATH, PHP_URL_QUERY, and PHP_URL_FRAGMENT. Each returns the corresponding component as a string, except PHP_URL_PORT which returns an integer. If the component is missing from the URL, parse_url returns null for that constant, so you can detect its absence with a simple null check.

parse_url is lenient by design and parses partial URLs without raising errors. Passing "example.com/path" without a scheme returns ["path"=>"example.com/path"] with no host key, because PHP interprets the entire string as a relative path. For seriously malformed URLs like "http://foo:bar:baz", parse_url returns false instead of an array. Always check the return value before accessing array keys, and consider validating the result with filter_var when the URL comes from user input.

Query string and parameter handling

Given the query string from parse_url(), pass it to parse_str($queryString, $params) to decode it into a PHP array. PHP supports bracket notation natively: the query string filter[status]=active&filter[role]=admin decodes into $params["filter"]["status"] and $params["filter"]["role"]. For multiple values with the same key, append empty brackets: ?tag[]=js&tag[]=php decodes into an array $params["tag"]. Rebuilding a query string from an array uses http_build_query($params), which handles nested arrays, percent-encoding, and the & separator automatically.²

PHP offers two encoding functions for URL components, and choosing the wrong one can break your URLs. urlencode() uses the application/x-www-form-urlencoded format, which encodes spaces as plus signs (+). rawurlencode() follows RFC 3986, which encodes spaces as %20. For query string values where the receiving system expects the + convention, use urlencode(). For path segments where %20 is the correct encoding, use rawurlencode(). Mixing these two formats is a common source of bugs when a URL is encoded with one function and decoded with the other.

http_build_query accepts an associative array and returns a URL-encoded query string. It handles nested arrays by using bracket notation in the generated string, and it percent-encodes keys and values automatically. The function accepts optional arguments for encoding type, numeric prefix, and arg_separator.output, which let you customize the output format. When rebuilding a query string after modifying parameters, always pass the full modified array to http_build_query rather than concatenating new values onto an existing query string, because the existing string may contain stale or duplicate keys.

When a query parameter value itself contains special characters like ampersands or equals signs, http_build_query encodes them automatically so the resulting string is safe to append to a URL. For APIs that expect repeated keys rather than bracket notation, pass a flat array with duplicate keys and set the encoding type to PHP_QUERY_RFC1738. Building on this, test your generated URLs against the receiving system's parser before deploying to production, because subtle encoding differences can cause silent data corruption.

Edge cases, encoding, and pitfalls

parse_url() is lenient by design and does not validate the URL.⁴ Passing "example.com/path" without a scheme returns ["path"=>"example.com/path"] with no host, because PHP interprets the entire string as a relative path. Yet parse_url() returns false for seriously malformed URLs like "http://foo:bar:baz". For encoding, urlencode() uses the application/x-www-form-urlencoded format (+ for spaces), while rawurlencode() follows RFC 3986 (%20 for spaces)⁵. Use rawurlencode() for path segments and urlencode() for query values when you need the + convention.

This difference matters when you decode a URL that was encoded with urlencode() using a RFC 3986 decoder, because the + signs will be treated literally rather than as spaces. To avoid this mismatch, use rawurlencode() for path segments that may be decoded by RFC 3986-based parsers, and use urlencode() only for query values that stay within PHP's own encoding ecosystem. Building on this, be consistent with your encoding choice throughout the codebase so that decoding logic does not need to guess which format was used.

parse_url() accepts relative URLs without raising errors, but the returned array contains only the keys present in the input. A URL like "/about?q=1" returns ["path"=>"/about", "query"=>"q=1"] with no scheme or host. When your application needs to resolve relative URLs against a known base, store the base URL separately and combine the components yourself, because parse_url does not perform relative resolution. For user-supplied input, validate that the URL includes a scheme before treating it as an absolute URL.

parse_url() also accepts URLs with unusual but technically valid structures, such as "http://user:pass@host:port/path" where the userinfo component contains special characters. For URLs with non-ASCII hostnames, parse_url returns the raw Unicode string without converting it to IDNA encoding, so you need idn_to_ascii for internationalized domain names. When processing URLs from untrusted sources, always validate the scheme against an allowlist of expected schemes (http, https) to prevent unexpected protocol handlers from being invoked.

Validating URLs with filter_var and its limitations

PHP's filter_var($url, FILTER_VALIDATE_URL) checks that a URL string follows the expected format.³ It validates the scheme (must be present), the host (must be non-empty for http/https URLs), and the overall structure. Building on this, FILTER_VALIDATE_URL does not check that the hostname resolves, that the TLD exists, or that the URL is reachable. It accepts URLs with invalid TLDs like 'https://example.invalidtld' because it only checks syntax, not semantics. For stricter validation, combine filter_var with a DNS check: checkdnsrr(parse_url($url, PHP_URL_HOST), 'A') verifies the hostname has a DNS A record. This adds a network call, so use it only when the URL will actually be fetched.

Sanitizing and escaping URLs for output in HTML

When a URL appears in an HTML href attribute or as plain text on a page, you need to escape it for the context. Use htmlspecialchars($url, ENT_QUOTES, 'UTF-8') to convert characters like &, <, >, and quotes into their HTML entities. This prevents cross-site scripting when user-supplied URLs are rendered on a page. For URLs inside href attributes, validate the scheme first to avoid javascript: or data: injection, then escape the remaining characters. Always treat URLs from user input as untrusted and apply context-appropriate escaping before output.

Validation and escaping solve different problems, so apply both rather than choosing one. Validate the scheme to block dangerous protocols, then escape the remaining characters so the URL cannot break out of the HTML attribute context. When the URL is built from several user supplied pieces, escape the final assembled string rather than each piece in isolation. This catches cases where two safe fragments combine into an unsafe whole that neither piece revealed on its own.

Working with parse_str and the bracket notation pitfall

parse_str($query, $output) decodes a query string into an associative array. PHP's bracket notation creates nested arrays from parameter names like 'filter[status]=active': $output['filter']['status'] equals 'active'. Building on this, a security consideration: parse_str writes directly into the variable you pass, and a query string with a key like 'output[secret]=value' can overwrite existing array keys in $output.⁶ Always pass an empty array as the second argument to contain the parsed parameters. For untrusted query strings, consider parsing manually with explode('&', $query) and validating each key against an allowlist before inserting into your data structure. The bracket notation is powerful but can be exploited to inject unexpected keys if the input is not controlled.

Building URLs without a dedicated URL builder

PHP does not include a single URL builder class, so you construct URLs from components by concatenating strings. A reliable pattern is to parse an existing URL with parse_url, modify the parts you need, then reassemble with a helper function that joins scheme, host, port, path, query, and fragment. For query parameters, http_build_query handles nested arrays and percent-encoding automatically, so you do not need to manually encode each value. When rebuilding, check whether the port is the default for the scheme (80 for http, 443 for https) and omit it to keep the URL clean. Building URLs from user input requires validating each component before concatenation to avoid injection of unexpected schemes or hosts.⁷

Handling multibyte and IDN URLs in PHP

PHP's parse_url handles ASCII URLs correctly but does not automatically convert internationalized domain names to their ASCII-compatible encoding. A URL like 'https://münchen.de/path' parses with host 'münchen.de' (the raw Unicode string). To convert to the IDNA encoding, use idn_to_ascii('münchen.de', IDNA_DEFAULT, INTL_IDNA_VARIANT_UTS46), which produces 'xn--mnchen-3ya.de'.⁸ The idn_to_ascii function requires the intl extension. Building on this, for URLs with non-ASCII path segments, rawurlencode encodes each UTF-8 byte as a percent-encoded triplet. PHP's rawurlencode operates on bytes, not characters, so it handles multibyte strings correctly without special configuration.⁵ Always use rawurlencode for path segments and urlencode for query values to match the expected encoding format for each component.

Encoding multibyte path segments with rawurlencode

When a URL path contains non-ASCII characters like Chinese, Arabic, or Cyrillic script, rawurlencode converts each UTF-8 byte to a percent-encoded triplet, producing a fully ASCII-safe path. This is the correct approach for paths that will be consumed by systems expecting RFC 3986 encoding. For query string values, urlencode is appropriate when the receiving system uses the application/x-www-form-urlencoded format. Always encode path segments before constructing the full URL, and avoid double-encoding by checking whether a value is already percent-encoded before applying rawurlencode again.

Notes

parse_url($url) returns an array; missing components are absent (not null). Use PHP_URL_HOST, PHP_URL_PATH etc. as the second argument to extract one component. parse_str($query, $out) fills $out with decoded key-value pairs; bracket notation creates nested arrays. http_build_query($array) encodes back to a query string. filter_var($url, FILTER_VALIDATE_URL) validates format.

Examples

<?php
$url = 'https://user:[email protected]:8080/path?q=hello#section';
$parts = parse_url($url);
echo $parts['scheme'];    // https
echo $parts['host'];      // example.com
echo $parts['port'];      // 8080
echo $parts['path'];      // /path
echo $parts['query'];     // q=hello
echo $parts['fragment'];  // section

<?php
$url = 'https://api.example.com/v1/users?page=2';
$host  = parse_url($url, PHP_URL_HOST);
$path  = parse_url($url, PHP_URL_PATH);
$query = parse_url($url, PHP_URL_QUERY);
echo $host;   // api.example.com
echo $path;   // /v1/users
echo $query;  // page=2

<?php
$url = 'https://example.com/search?q=php+url&page=1';
parse_str(parse_url($url, PHP_URL_QUERY), $params);
echo $params['q'];       // php url
$params['page'] = 2;
$params['sort'] = 'date';
$newQuery = http_build_query($params);
echo $newQuery;
// q=php+url&page=2&sort=date

<?php
$filter = ['filter' => ['role' => 'admin', 'page' => 2]];
$query  = http_build_query($filter);
$url    = 'https://example.com/users?' . $query;
echo $url;
// https://example.com/users?filter%5Brole%5D=admin&filter%5Bpage%5D=2

<?php
$url = 'https://user:[email protected]:8080/path?q=hello#section';
$parts = parse_url($url);
echo $parts['scheme'];    // https
echo $parts['host'];      // example.com
echo $parts['port'];      // 8080
echo $parts['path'];      // /path
echo $parts['query'];     // q=hello
echo $parts['fragment'];  // section

URL Parsing in Ruby

Ruby's standard library includes the URI module, which parses URLs into typed objects with named accessor methods for each component.¹ URI.parse returns a URI::HTTP or URI::HTTPS object exposing scheme, host, port, path, query, and fragment. For more complex needs such as template URIs, IDN domains, or flexible query handling, the Addressable gem extends the built-in module with a superset API.² Both libraries handle encoding and decoding, support URI normalization, and allow modifying individual components without reconstructing the full URL string by hand.

Core parsing API

Calling URI.parse("https://example.com:8080/path?q=hello#section") returns a URI::HTTPS object. Access components via .scheme, .host, .port, .path, .query, and .fragment. The .host method returns just the hostname without the port, so it never includes the port number even when it is non-standard.¹ For the full authority including port, use .authority (Ruby 3.2+) or format it as "#{uri.host}:#{uri.port}". The .to_s method reconstructs the full URL from the parsed components. Building on this, Ruby 3.1 introduced URI::File for file:// URIs, and the library distinguishes between URI::HTTP for http://, URI::HTTPS for https://, and URI::Generic for everything else. Knowing the returned class matters because HTTPS enforces port and scheme validation that Generic does not. For most applications, checking .is_a?(URI::HTTPS) before calling .request_uri or .path confirms you are dealing with a web URL before issuing HTTP requests.

Query string and parameter handling

URI.parse returns the raw query string via .query, and parsing it into key-value pairs requires URI.decode_www_form(uri.query). This method returns an array of two-element arrays: [["key", "value"], ...]. Building on this, convert to a hash with URI.decode_www_form(uri.query).to_h, but note that duplicate keys lose all but the last value. For repeated keys, keep the array and iterate it directly. Rebuild a query string with URI.encode_www_form({key: 'value', page: 2}), which percent-encodes unsafe characters using the application/x-www-form-urlencoded format.³ When the query string contains encoded ampersands or plus signs, decode_www_form handles the percent-decoding automatically, so you do not need to call CGI.unescape yourself. For applications that parse query strings repeatedly, memoizing the decoded pairs avoids re-parsing the same raw string on every request.

Edge cases, encoding, and pitfalls

URI.parse raises URI::InvalidURIError for malformed URLs, so wrap it in a rescue block when parsing user-supplied input. Yet it is lenient about relative paths: URI.parse("/path?q=1") succeeds, returning a URI::Generic with an empty scheme and host. For encoding, URI.encode_www_form_component(str) encodes a single value, and URI.decode_www_form_component(str) decodes it. The Addressable::URI.parse method is more permissive and handles IRIs (internationalized resource identifiers) natively, making it the right choice for user-facing URL handling.² A common pitfall is assuming URI.parse validates the scheme strictly; in practice, it accepts any well-formed URI including custom schemes like myapp://, which is useful for deep linking but means you must validate the scheme yourself if your application only accepts http and https URLs. For defensive parsing, combine a regex scheme check with URI.parse to reject unexpected schemes before extracting components.

Modifying and reconstructing URLs with URI

URI objects in Ruby are mostly read-only: the accessor methods return values, but there is no built-in setter for individual components. To modify a URL, read the components you need, change the relevant ones, and reconstruct with URI::HTTPS.build(host: 'example.com', path: '/new', query: 'key=val'). The build class method accepts a hash of components and returns a new URI object. Building on this, for query parameter modifications, decode the query with URI.decode_www_form, manipulate the resulting array of pairs, and rebuild with URI.encode_www_form. Assign the result to a new URI object via the build method. This immutable approach prevents accidental mutation of shared URI objects in multi-threaded Rails applications.

Resolving relative URLs with URI.merge versus URI.join

URI.merge and URI.join handle relative references differently. URI.merge follows RFC 3986's resolution algorithm precisely, replacing path segments and resolving dot-segments such as ".." and "." as a browser would. URI.join, by contrast, concatenates the base path with the reference and is useful when you want simple path concatenation without full resolution logic. For link resolution in parsers and crawlers, merge is the correct choice because it handles edge cases like trailing slashes and query-only references. For building paths where you control both inputs, join is simpler and more predictable.

The choice also affects how trailing slashes and empty path references are handled, which matters for link graphs that must stay consistent. When in doubt, prefer merge for crawlers and join only for simple path building where you control both inputs. Document the resolver you chose in the module that performs link extraction so future maintainers do not switch strategies by accident. A consistent resolver keeps the discovered URLs stable across crawls and avoids duplicate entries that differ only by a trailing slash.

Building query strings from hashes

When constructing a URL from user input, start with a hash of parameters and pass it to URI.encode_www_form, which handles the percent-encoding and joining in one call. This approach avoids manual string interpolation that could leave unencoded reserved characters in the query string. For nested parameters such as filters[category]=ruby, encode_www_form passes them through as-is, and Rack parses them into nested hashes on the server side. Always build the query string before assigning it to uri.query rather than concatenating strings yourself.

Using Addressable::URI for advanced URL handling

The Addressable gem extends Ruby's URI module with IRI support, URI templates (RFC 6570), and more permissive parsing.⁴ Addressable::URI.parse handles Unicode characters in paths and hostnames without raising URI::InvalidURIError, which the built-in URI module raises for non-ASCII input.⁵ For internationalized domain names, Addressable normalizes the host using IDNA encoding automatically.⁶ Building on this, Addressable::URI supports URI templates: uri = Addressable::URI.parse('https://api.example.com/{?q,page}'); uri.expand(q: 'ruby', page: 2) produces 'https://api.example.com/?q=ruby&page=2'. This is the only Ruby library that implements RFC 6570 templates. For standard HTTPS URL parsing without templates, the built-in URI module avoids an extra dependency, but it cannot handle the edge cases that Addressable normalizes silently.

Encoding differences between URI.encode_www_form and manual encoding

URI.encode_www_form follows the application/x-www-form-urlencoded convention, encoding spaces as "+" and percent-encoding reserved characters using uppercase hex digits. Manual encoding with URI.encode_www_form_component applies the same percent-encoding but encodes spaces as "%20" instead, making it suitable for encoding individual query parameter values where the "+" character would be misinterpreted as a literal plus sign. For constructing query strings from a hash, encode_www_form is the correct choice because it handles the joining and encoding in one step. Use encode_www_form_component when encoding a value that will be assembled into a URL by hand to avoid ambiguity in how spaces appear in the final string. For applications that accept user-provided URLs and need to normalize them before storage, Addressable's normalization behavior is more predictable than the built-in URI module, which can raise on input that browsers accept without issue.

Handling URLs in Rails applications

Rails provides several URL helpers that wrap Ruby's URI module. The URI module is available in all Rails apps without additional gems. For parsing incoming request URLs, Rails exposes the parsed URL through request.original_url, request.path, and request.query_parameters. The query_parameters hash is already decoded by Rack, so you do not need to call URI.decode_www_form yourself. Building on this, for URL generation, use Rails route helpers (users_path, user_path(@user)) rather than constructing URLs manually. These helpers respect the application's default_url_options host and protocol settings, ensuring generated URLs match the deployment environment. When you need to parse an external URL (from an API response or user input), use URI.parse or Addressable::URI.parse in a service object, not in a view template. CapyToolkit offers a URL parser tool that lets you paste any URL and inspect its components interactively, which is useful when debugging URL handling in a Rails request cycle.

Safe URL parsing with error handling in Ruby

URI.parse raises URI::InvalidURIError for malformed input, making a rescue block the standard error-handling pattern. For user-supplied URLs, wrap the call in a begin/rescue/end block and return nil or a default value on failure. Building on this, for applications that need to parse many URLs without individual exception handling, define a helper method: def safe_parse(input) = URI.parse(input) rescue nil. This returns nil for any malformed input, letting the caller handle the failure case. When using Addressable::URI, the same pattern applies: Addressable::URI.parse returns nil for unparseable strings rather than raising, which simplifies the calling code. Choose the approach that matches your error-handling strategy at the application level.

Notes

URI.parse returns a typed object (URI::HTTP, URI::HTTPS, URI::Generic). Access .scheme, .host, .port, .path, .query, .fragment. Parse query strings with URI.decode_www_form(uri.query) → array of pairs. Build query strings with URI.encode_www_form(hash). Use Addressable::URI for IRI support and URI templates.

Examples

require 'uri'

uri = URI.parse('https://example.com:8080/path/page?q=hello#section')
puts uri.scheme    # https
puts uri.host      # example.com
puts uri.port      # 8080
puts uri.path      # /path/page
puts uri.query     # q=hello
puts uri.fragment  # section

require 'uri'

uri = URI.parse('https://example.com/search?q=ruby+url&page=2&tag=parsing&tag=uri')
pairs = URI.decode_www_form(uri.query)
# => [["q", "ruby url"], ["page", "2"], ["tag", "parsing"], ["tag", "uri"]]
pairs.each { |k, v| puts "#{k}: #{v}" }

hash = pairs.to_h
puts hash['page']  # 2 (last value for 'tag' only)

require 'uri'

uri = URI.parse('https://example.com/search?q=ruby&page=1')
params = URI.decode_www_form(uri.query).to_h
params['page'] = '2'
params['sort'] = 'date'
uri.query = URI.encode_www_form(params)
puts uri.to_s
# https://example.com/search?q=ruby&page=2&sort=date

require 'uri'

base = URI.parse('https://example.com/blog/post')
relative = URI.parse('../about')
resolved = base.merge(relative)
puts resolved.to_s
# https://example.com/about

require 'uri'

uri = URI.parse('https://example.com:8080/path/page?q=hello#section')
puts uri.scheme    # https
puts uri.host      # example.com
puts uri.port      # 8080
puts uri.path      # /path/page
puts uri.query     # q=hello
puts uri.fragment  # section

URL Parsing in Rust

Rust URL parsing centers on the url crate, which provides the Url type: a fully parsed, validated URL that enforces correctness at construction time.¹ Url::parse() returns a Result<Url, ParseError>, making malformed URLs impossible to use without explicit error handling. Once parsed, the Url type exposes scheme(), host_str(), port(), path(), query(), and fragment() methods for reading individual components. For query strings, query_pairs() returns an iterator of (Cow<str>, Cow<str>) pairs without heap allocation per iteration, making it efficient for large parameter sets. The url crate implements the WHATWG URL Standard for parsing and normalization behavior.²

Core parsing API

Calling Url::parse("https://example.com:8080/path?q=hello#section") returns a Result. Pattern-matching on Ok(url) gives you a Url value: url.scheme() returns "https", url.host_str() returns Some("example.com"), url.port() returns Some(8080), and url.path() returns "/path". The url crate stores the URL as a single contiguous string internally and returns slices into that buffer, so most accessor methods return &str rather than String, avoiding allocation.¹ For owned strings, call .to_owned() on the returned slice. This design means that Url values hold ownership of the underlying string, and the borrowed &str slices cannot outlive the Url value itself, which is why methods like .to_string() exist for callers that need an independent copy.

Query string and parameter handling

Iterating query parameters uses url.query_pairs(), which returns an iterator of (Cow<str>, Cow<str>) tuples. The iterator handles percent-decoding automatically, so %20 becomes a space in the returned Cow value. For mutation, url.query_pairs_mut() returns a serializer that appends or replaces parameters: .append_pair("key", "value") adds a pair, and .clear() removes all pairs before rebuilding. Calling url.set_query(None) removes the query string entirely. The url crate does not provide a HashMap-style API, so use .query_pairs().collect() to build one when random access is needed.³ Building on this, the order of query parameters is preserved by the parser, which matters for APIs that include repeated keys like ?tag=rust&tag=url and rely on the original ordering for correct interpretation.

Edge cases, encoding, and pitfalls

Relative URLs cannot be parsed alone because Url::parse("/path") returns Err(RelativeUrlWithoutBase). To resolve a relative reference, call base_url.join("/path"), which follows the WHATWG URL Standard resolution algorithm.⁴ Base URL joining replaces only the path and below, not the scheme or host, so the resulting URL always inherits the base's origin. For percent-encoding a path segment, use the percent_encoding crate with the appropriate AsciiSet constant for the encoding context. The url crate normalizes URLs on parse: it converts the scheme to lowercase, removes the default port, and resolves dot-segments in the path. For internationalized domain names, the url crate applies IDNA encoding, so münchen.de becomes xn--mnchen-3ya.de in the host component.⁵

Mutation patterns with query_pairs_mut

The query_pairs_mut method returns a UrlQuery serializer that supports append_pair, clear, and extend_pairs. Building on this, to replace a single parameter while preserving others, collect the pairs you want to keep: let pairs: Vec<_> = url.query_pairs().filter(|(k, _)| k != "page").collect(); url.query_pairs_mut().clear().extend_pairs(pairs); url.query_pairs_mut().append_pair("page", "3"). The clear call removes all existing parameters before the new set is written. For applications that build URLs from templates, construct the base URL first, then use query_pairs_mut to add parameters in a loop. This avoids string concatenation and ensures every value is percent-encoded correctly by the serializer.

The percent_encoding crate and custom AsciiSet definitions

The percent_encoding crate provides finer control over which characters get percent-encoded than the url crate, which applies a fixed encoding set depending on the URL component. When encoding a query value, the url crate uses QUERY_ENCODE_SET, which preserves the "/" and "?" characters. For encoding a path segment, it uses PATH_ENCODE_SET, which preserves "/" but encodes spaces. If your application needs a different set, such as encoding a literal "+" in a query value without it being decoded as a space on the server, define a custom AsciiSet using percent_encoding::AsciiSet and pass it to the encode function. This is useful when integrating with legacy APIs that expect non-standard encoding behavior.

Defining a custom set keeps the rest of the serializer untouched, so only the characters you care about change encoding. This avoids the risk of a global encoding change that breaks other components in the same URL. When a downstream system needs a non standard encoding, create the set once and reuse it for every request to that system. Reusing a single definition also makes the behavior easy to test and keeps the encoding rule in one place instead of scattered through the codebase.

Error handling patterns for URL parsing in Rust

Url::parse returns a Result<Url, ParseError>, which forces the caller to handle the error case at compile time.² Building on this, the ParseError enum has variants for missing scheme, invalid port, invalid IP address, and other specific failure modes. For user-facing applications, map ParseError to a user-friendly message: match Url::parse(input) { Ok(url) => url, Err(ParseError::RelativeUrlWithoutBase) => { /* provide a base URL hint */ }, Err(_) => { /* generic invalid URL message */ } }. For web servers that parse URLs from request paths, use Url::parse and propagate the error as a 400 Bad Request response. The Result type makes it impossible to accidentally use an unparsed URL, which is a significant safety advantage over languages where URL parsing returns null or throws unchecked exceptions.

The url crate and WHATWG URL Standard compliance

The url crate follows the WHATWG URL Standard for parsing and serialization, which means its behavior matches what browsers do when resolving and normalizing URLs. This includes converting the scheme and host to lowercase, removing default ports (443 for https, 80 for http), and applying IDNA encoding to internationalized domain names. The WHATWG standard also defines how percent-encoding is applied in each URL component, so the url crate's output is consistent with browser behavior when the same URL is parsed by Chrome or Firefox. For Rust applications that need to replicate browser URL resolution logic, the url crate is the most reliable choice because it is maintained by the Servo browser engine team.

Using Url as a key in HashMap and HashSet

The Url type implements Hash, Eq, PartialEq, and Ord, making it usable as a key in HashMap and HashSet. Building on this, the Hash implementation is based on the normalized string representation: two Url values that normalize to the same string hash to the same bucket.⁴ This means Url::parse('HTTPS://EXAMPLE.COM:443/path') and Url::parse('https://example.com/path') are considered equal as HashMap keys because the url crate normalizes the scheme to lowercase and removes the default port.

When to use a newtype wrapper

For applications that need case-sensitive or port-preserving comparison, wrap the Url in a newtype that implements Hash and Eq based on the raw string representation instead. This is useful when your application treats http://example.com and http://example.com:80 as distinct endpoints, which the default Hash implementation does not preserve. For cache deduplication, the default behavior is correct: two URLs that normalize to the same string represent the same resource.

Notes

Add url = "2" to Cargo.toml. Url::parse() returns Result<Url, ParseError>. Use .scheme(), .host_str(), .port(), .path(), .query() for components. Iterate query params with .query_pairs(). Mutate with .query_pairs_mut().append_pair("key", "val"). Resolve relative URLs with base.join("../other"). Most accessors return &str (no allocation).

Examples

use url::Url;

fn main() {
    let url = Url::parse("https://example.com:8080/path?q=hello#section")
        .expect("invalid URL");
    println!("{}", url.scheme());     // https
    println!("{:?}", url.host_str()); // Some("example.com")
    println!("{:?}", url.port());     // Some(8080)
    println!("{}", url.path());       // /path
    println!("{:?}", url.fragment()); // Some("section")
}

use url::Url;

fn main() {
    let url = Url::parse("https://example.com/search?q=rust+url&page=2&tag=systems&tag=web")
        .unwrap();
    for (key, value) in url.query_pairs() {
        println!("{} = {}", key, value);
    }
    // q = rust url
    // page = 2
    // tag = systems
    // tag = web
}

use url::Url;

fn main() {
    let mut url = Url::parse("https://example.com/search?q=rust").unwrap();
    url.query_pairs_mut()
        .append_pair("page", "2")
        .append_pair("sort", "date");
    println!("{}", url);
    // https://example.com/search?q=rust&page=2&sort=date
}

use url::Url;

fn main() {
    let base = Url::parse("https://example.com/blog/post").unwrap();
    let resolved = base.join("../about").unwrap();
    println!("{}", resolved);
    // https://example.com/about
}

use url::Url;

fn main() {
    let url = Url::parse("https://example.com:8080/path?q=hello#section")
        .expect("invalid URL");
    println!("{}", url.scheme());     // https
    println!("{:?}", url.host_str()); // Some("example.com")
    println!("{:?}", url.port());     // Some(8080)
    println!("{}", url.path());       // /path
    println!("{:?}", url.fragment()); // Some("section")
}

URL Parsing in Java

Java offers two standard classes for URL handling: java.net.URI, which parses and represents a URI according to RFC 2396¹, and the legacy java.net.URL, which adds network connectivity via .openConnection(). For modern code, java.net.URI is the right choice because java.net.URL is considered legacy and its .equals() method performs DNS lookups², making it unsuitable for use in collections. OkHttp's HttpUrl class provides a fluent API with builder pattern, handles edge cases more robustly, and is the standard choice in Android and OkHttp-based server code.

Core parsing API

Calling URI.create("https://example.com:8080/path?q=hello#section") returns a URI object. URI.create() throws the unchecked IllegalArgumentException for malformed input, while new URI(string) throws the checked URISyntaxException¹, so choose based on your error-handling strategy. Access components via .getScheme(), .getHost(), .getPort() (returns -1 for the default port), .getPath(), .getRawQuery(), and .getFragment(). The .getRawQuery() method returns the query string before percent-decoding, so use .getQuery() when you need the decoded form. The URI class is immutable, which makes it safe to store in collections and share across threads without synchronization, and its .toString() method reconstructs the full URL from the parsed components. The .normalize() method resolves dot-segments in the path, which is useful when comparing URLs that may use relative path references internally.

Query string and parameter handling

java.net.URI provides no built-in method for parsing query parameters into a map, which is a notable gap in the standard library. Building on this, the standard pattern is to split the raw query string manually: Arrays.stream(uri.getRawQuery().split("&")).map(p -> p.split("=", 2)) gives a stream of key-value arrays. For production code, Apache HttpComponents offers URLEncodedUtils.parse(uri, StandardCharsets.UTF_8), returning a List<NameValuePair>³ with proper handling of encoded characters and duplicate keys. OkHttp's HttpUrl provides .queryParameter(name) and .queryParameterValues(name) as a cleaner alternative. When building your own parser, remember that the raw query string is percent-encoded, so you must decode each key and value with URLDecoder.decode after splitting, and you should handle the edge case where the query string contains a key without an equals sign by treating the value as an empty string.

Edge cases, encoding, and pitfalls

java.net.URL's .equals() method performs a DNS lookup to compare hosts, which blocks the calling thread and causes two URLs pointing to the same server via different hostnames to compare as equal. Never use URL in HashMap or HashSet; use URI instead. For percent-encoding, java.net.URLEncoder.encode(value, "UTF-8") encodes a value for query strings (+ for spaces)⁴, while the URI multi-argument constructor accepts decoded component strings and handles encoding internally. Building on this, URI.create() requires a fully pre-encoded string, so use the multi-argument constructor when building URLs from decoded component parts. The distinction between these two decoding strategies matters because mixing them leads to double-encoding bugs where spaces appear as "%2520" instead of "%20" in the final URL.

The URI multi-argument constructor in detail

The five-argument URI constructor accepts (scheme, userInfo, host, port, path, query, fragment), applying encoding rules specific to each component. The host argument is validated for legal characters, the path is percent-encoded per path-segment rules, and the query string is expected to already be pre-encoded. This means you should pass decoded values for scheme, host, and path, but a pre-encoded query string. Passing a decoded query string results in the constructor percent-encoding the special characters in the query, which changes the semantics of operators like & and =.

Treat the query argument as already encoded data and never pass a decoded string into the constructor for that field. Doing so lets the constructor encode the ampersand and equals signs again, which corrupts the parameter boundaries your server expects. When you build a URL from decoded components, encode each value yourself before assembling the query and pass the encoded result to the constructor. This keeps the responsibility for encoding in your code where it is visible and testable rather than hidden inside the parser.

OkHttp HttpUrl for fluent URL building and parsing

OkHttp's HttpUrl class provides a builder pattern for constructing URLs and a rich query parameter API for reading them. Building on this, HttpUrl.parse(input) returns null for invalid input instead of throwing, making it safe for user-supplied URLs. The builder accepts path segments individually: new HttpUrl.Builder().scheme('https').host('api.example.com').addPathSegment('v1').addPathSegment('users').addQueryParameter('page', '2').build(). Each addPathSegment call encodes the segment and inserts slashes correctly. For query parameters, addQueryParameter appends a value, and queryParameterValues('key') returns all values for a repeated key as a list. This is more ergonomic than java.net.URI for HTTP-specific URL handling because HttpUrl understands URL semantics natively rather than treating the URL as a generic structured string.²

Apache HttpComponents URLEncodedUtils for query parsing

Apache HttpComponents provides the URLEncodedUtils.parse utility, which converts a query string into a list of NameValuePair objects with proper percent-decoding applied to both keys and values. This is the standard choice for applications that already depend on the Apache HttpClient library because it handles edge cases like empty values, missing equals signs, and encoded ampersands correctly. The returned list preserves the original ordering of parameters, which matters for APIs that include repeated keys. For applications that do not already use Apache HttpClient, the OkHttp HttpUrl API provides similar functionality with a smaller dependency footprint.

Building URLs with the multi-argument URI constructor

The URI class provides a five-argument constructor that accepts decoded component strings and handles encoding internally: new URI('https', 'user:pass', 'example.com', 8080, '/path', 'q=hello', 'section'). Building on this, each argument is encoded according to its component's rules: the path is percent-encoded, the query string is expected to be pre-encoded, and the host is validated. This constructor is the correct way to build a URL from decoded component parts without manually encoding each field. For query strings, you still need to encode the query parameter values yourself or pass a pre-encoded string. The constructor throws URISyntaxException (a checked exception), so wrap it in a try/catch or declare the exception in your method signature.

URL handling in Spring Boot applications

In Spring Boot, URL handling typically involves parsing the incoming request URL from the HttpServletRequest object and constructing redirect or forward URLs using Spring's UriComponentsBuilder. The UriComponentsBuilder class provides a fluent API for building URLs from template variables, handling encoding correctly, and appending query parameters without manual string manipulation. For REST API endpoints, Spring's @RequestParam annotation automatically extracts and decodes query parameters, so you rarely need to call URI.create() yourself in controller code. When building redirect URLs with redirect:, Spring's RedirectView uses UriUtils.encode to handle the encoding, which prevents open-redirect vulnerabilities by validating the target URL against trusted hosts.

Internationalized domain names and IDNA in Java

Java's java.net.URI does not automatically apply IDNA encoding to internationalized domain names. A URI created from 'https://münchen.de/path' stores the Unicode hostname as-is. To convert to ASCII-compatible encoding, use java.net.IDN.toASCII('münchen.de'), which produces 'xn--mnchen-3ya.de'. Building on this, for URLs parsed from user input, extract the hostname with uri.getHost(), convert with IDN.toASCII, and reconstruct the URI with the ASCII hostname. The IDN class implements IDNA 2003 (RFC 3490)⁵, which differs from IDNA 2008 (RFC 5891)⁶ used by the WHATWG URL Standard. For most domain names the two standards produce the same encoding, but a few characters (notably the German sharp s ß) differ. When interoperability with WHATWG-based systems matters, test your IDNA encoding against the expected output.

Notes

Prefer java.net.URI over java.net.URL. URI.create(str) throws unchecked IllegalArgumentException; new URI(str) throws checked URISyntaxException. .getPort() returns -1 for default port. No built-in query param parsing, so use Apache HttpComponents URLEncodedUtils or OkHttp HttpUrl. Never use URL in collections due to DNS-lookup equals().

Examples

import java.net.URI;

public class UrlParsing {
    public static void main(String[] args) {
        URI uri = URI.create("https://example.com:8080/path?q=hello#section");
        System.out.println(uri.getScheme());    // https
        System.out.println(uri.getHost());      // example.com
        System.out.println(uri.getPort());      // 8080
        System.out.println(uri.getPath());      // /path
        System.out.println(uri.getRawQuery());  // q=hello
        System.out.println(uri.getFragment());  // section
    }
}

import java.net.URI;
import java.util.*;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;

public class QueryParser {
    public static Map<String, String> parseQuery(URI uri) {
        Map<String, String> params = new LinkedHashMap<>();
        if (uri.getRawQuery() == null) return params;
        for (String pair : uri.getRawQuery().split("&")) {
            String[] kv = pair.split("=", 2);
            String key = URLDecoder.decode(kv[0], StandardCharsets.UTF_8);
            String val = kv.length > 1
                ? URLDecoder.decode(kv[1], StandardCharsets.UTF_8) : "";
            params.put(key, val);
        }
        return params;
    }
}

import okhttp3.HttpUrl;

public class BuildUrl {
    public static void main(String[] args) {
        HttpUrl url = new HttpUrl.Builder()
            .scheme("https")
            .host("api.example.com")
            .addPathSegment("v1")
            .addPathSegment("users")
            .addQueryParameter("role", "admin")
            .addQueryParameter("page", "2")
            .build();
        System.out.println(url);
        // https://api.example.com/v1/users?role=admin&page=2
    }
}

import java.net.URI;

public class ResolveRelative {
    public static void main(String[] args) {
        URI base = URI.create("https://example.com/blog/post");
        URI relative = URI.create("../about");
        URI resolved = base.resolve(relative);
        System.out.println(resolved);
        // https://example.com/about
    }
}

import java.net.URI;

public class UrlParsing {
    public static void main(String[] args) {
        URI uri = URI.create("https://example.com:8080/path?q=hello#section");
        System.out.println(uri.getScheme());    // https
        System.out.println(uri.getHost());      // example.com
        System.out.println(uri.getPort());      // 8080
        System.out.println(uri.getPath());      // /path
        System.out.println(uri.getRawQuery());  // q=hello
        System.out.println(uri.getFragment());  // section
    }
}

URL Parsing in C#

C# parses URLs with System.Uri, which accepts an absolute or relative URI string and exposes typed properties for each component¹. Passing a URL string to the Uri constructor validates and normalizes it immediately, so no separate parse step is needed. For building and modifying URLs, UriBuilder provides writable properties that reconstruct the URL via its Uri property. ASP.NET Core adds QueryHelpers.ParseQuery(), which parses a query string into a Dictionary<string, StringValues>², correctly handling repeated keys and empty values in server-side code.

Core parsing API

new Uri("https://example.com:8080/path?q=hello#section") constructs a Uri object. Properties include .Scheme, .Host, .Port (returns 443 for HTTPS even when omitted), .AbsolutePath, .Query (with the leading ?), and .Fragment (with the leading #)¹. The .Authority property combines host and port, returning "example.com:8080" when the port is non-default, or just "example.com" for the default port. The .IsDefaultPort property tells you whether the URL uses the scheme's standard port, allowing you to strip it from display strings. The Uri class is immutable, so parsing once and reusing the Uri instance across your application is safe and avoids repeated validation overhead. The .MakeRelativeUri method computes the relative path between two URIs with the same scheme and authority, which is useful for generating breadcrumb navigation or computing download links.

Query string and parameter handling

System.Uri provides no built-in query parameter parsing, so it returns .Query as a raw string including the leading ?. For ASP.NET Core projects, Microsoft.AspNetCore.WebUtilities.QueryHelpers.ParseQuery(uri.Query) returns a Dictionary<string, StringValues>² where StringValues handles repeated keys as an array. Building on this, QueryHelpers.AddQueryString(url, key, value) appends a parameter to a URL string safely. For non-ASP.NET projects, System.Web.HttpUtility.ParseQueryString(uri.Query) returns a NameValueCollection with similar capabilities. When building URLs from user input, always use AddQueryString or ParseQuery to construct the query string rather than concatenating raw strings, because these helpers handle percent-encoding and prevent injection of additional query parameters through unencoded ampersands. The StringValues struct is a zero-allocation wrapper that avoids copying strings when only a single value is present, which makes it efficient for the common case of unique keys.

Edge cases, encoding, and pitfalls

Uri normalizes URLs on construction: it lowercases the scheme and host, resolves dot-segments in the path, and removes the default port from the Authority. Yet it does not validate the host as a real domain, so an invalid hostname like "example..com" may still parse without error. For encoding, Uri.EscapeDataString(value) encodes per RFC 3986 (%20 for spaces)³, while Uri.EscapeUriString(input) is less aggressive and is not suitable for encoding individual component values. Use EscapeDataString for query parameter values. The distinction matters because EscapeUriString preserves characters like / and ? that have structural meaning in a URL, so passing a user-supplied value through EscapeUriString and then concatenating it into a URL allows the injected value to alter the URL's structure rather than being treated as a single component value.

When to use EscapeDataString versus UrlEncode

Uri.EscapeDataString follows RFC 3986 and encodes spaces as %20, making it the correct choice for encoding individual query parameter values and path segments. System.Net.HttpUtility.UrlEncode uses the application/x-www-form-urlencoded format and encodes spaces as +, which is what HTML form submissions expect. Use UrlEncode when building form data for application/x-www-form-urlencoded POST bodies, and EscapeDataString when constructing URL components for APIs that expect RFC 3986 encoding. Mixing the two leads to servers interpreting %20 as a literal "%20" or + as a literal "+" rather than a space.

UriBuilder for modifying existing URLs

UriBuilder wraps a Uri with mutable properties: Scheme, Host, Port, Path, Query, and Fragment. Setting any property and reading .Uri or .ToString() reconstructs the URL with the updated values. Building on this, UriBuilder.Query accepts a query string with or without the leading question mark: both 'key=value' and '?key=value' produce the same result⁴. To modify query parameters, parse the existing query with QueryHelpers.ParseQuery, update the dictionary, and reassign: builder.Query = QueryHelpers.ParseQuery(uri.Query).ToDictionary(k => k.Key, k => k.Value.ToString()). This round-trip through ParseQuery handles encoding and decoding correctly, which matters when the existing query contains percent-encoded characters that you want to preserve unchanged. For applications that frequently modify URLs, UriBuilder is more readable than string manipulation and less error-prone than manual concatenation.

QueryHelpers.AddQueryString for safe parameter appending

QueryHelpers.AddQueryString is the safest way to add a query parameter to an existing URL in ASP.NET Core because it handles encoding and prevents parameter injection. The method accepts a URL string, a key, and a value, then returns a new URL with the parameter appended and correctly percent-encoded. When the URL already contains a query string, AddQueryString appends the new parameter with an ampersand; when the URL has no query string, it adds a leading question mark. For multiple parameters, the overload accepts an IEnumerable<KeyValuePair<string, string>> and appends each one in order. This is preferable to string concatenation because it prevents bugs where an unencoded ampersand in a user-supplied value creates a spurious extra parameter.

When you already have a collection of parameters instead of one key at a time, the IEnumerable overload handles the entire set without manual formatting. Because it iterates the collection directly, the calling code stays free from ampersand and question-mark logic, and every value still goes through the same percent-encoding pipeline. This is useful for filter queries, pagination controls, or any UI where the parameter set is assembled dynamically at runtime.

Comparing URIs with Uri.Equals and canonical forms

Uri.Equals compares URIs by their normalized string representation: scheme and host are compared case-insensitively, default ports are removed, and percent-encoded unreserved characters are decoded before comparison³. Building on this, new Uri('HTTP://EXAMPLE.COM/path').Equals(new Uri('http://example.com/path')) returns true. For custom comparison logic (such as ignoring the query string or comparing only the path), extract the relevant properties and compare them directly. The Uri.Compare method provides more control, allowing you to specify which components to compare and whether to ignore the case. For cache key generation, use the normalized Uri.ToString() as the key, which applies all RFC 3986 normalization steps that the Uri class implements.

Handling relative URIs in .NET

The Uri class supports relative URIs through the UriKind.Relative parameter and the constructor that accepts a base Uri and a relative reference. new Uri(baseUri, "../about") resolves the relative reference against the base following RFC 3986's resolution algorithm, which matches browser behavior for resolving href attributes. The base Uri must be absolute, and the relative reference can be a path-only reference like "/path?q=1" or a full relative URL with dot-segments. For parsing both absolute and relative URLs from user input, use Uri.TryCreate with UriKind.RelativeOrAbsolute, which accepts either form and sets the Kind property accordingly. This is useful in ASP.NET Core middleware that handles redirect URLs where the input may be either a full URL or a relative path.

Uri.EscapeDataString encoding behavior and limits

Uri.EscapeDataString encodes all characters except unreserved characters (letters, digits, hyphen, period, underscore, tilde). It encodes spaces as %20, not +, making it suitable for RFC 3986 encoding of individual component values. Building on this, the Uri constructor itself rejects strings longer than 65,519 characters with a UriFormatException⁵, so be aware of this ceiling when passing large strings through Uri construction pipelines. For form-encoded query strings (spaces as +), use System.Net.Http.FormUrlEncodedContent or HttpUtility.UrlEncode instead. The choice between EscapeDataString and UrlEncode depends on the receiving server's expectations: modern APIs typically accept both, but signature verification code may require a specific encoding format.

Parsing URLs in .NET with Uri.TryCreate for safe validation

Uri.TryCreate provides a safe way to parse and validate URLs in a single call without exception handling. It returns a bool indicating success and outputs the parsed Uri via an out parameter: if (Uri.TryCreate(input, UriKind.Absolute, out var uri)) { /* use uri */ }⁵. Building on this, combine TryCreate with scheme validation for a complete check: verify that uri.Scheme is either Uri.UriSchemeHttp or Uri.UriSchemeHttps. For ASP.NET Core applications that accept user-supplied URLs, this pattern avoids the performance cost of exception-based validation on invalid input. Use Uri.TryCreate rather than wrapping the Uri constructor in a try/catch when the input may frequently be malformed, such as in form validation or API input parsing.

Notes

new Uri(str) validates and normalizes immediately. .Host gives hostname without port; .Port always returns a number (443 for default HTTPS). .Query includes the leading ?. Use UriBuilder for modifications. Use QueryHelpers.ParseQuery() (ASP.NET Core) or HttpUtility.ParseQueryString() (.NET Framework) for query param parsing. Use Uri.EscapeDataString() not EscapeUriString() for encoding individual values.

Examples

using System;

var uri = new Uri("https://example.com:8080/path?q=hello#section");
Console.WriteLine(uri.Scheme);       // https
Console.WriteLine(uri.Host);         // example.com
Console.WriteLine(uri.Port);         // 8080
Console.WriteLine(uri.AbsolutePath); // /path
Console.WriteLine(uri.Query);        // ?q=hello
Console.WriteLine(uri.Fragment);     // #section

using Microsoft.AspNetCore.WebUtilities;

var uri = new Uri("https://example.com/search?q=csharp+url&page=2&tag=dotnet&tag=web");
var query = QueryHelpers.ParseQuery(uri.Query);
Console.WriteLine(query["q"]);    // csharp url
Console.WriteLine(query["page"]); // 2
foreach (var tag in query["tag"]) {
    Console.WriteLine(tag); // dotnet  then  web
}

using System;
using System.Collections.Generic;
using System.Net.Http;
using Microsoft.AspNetCore.WebUtilities;

var builder = new UriBuilder("https://example.com/search");
var q = new Dictionary<string, string> {
    ["q"] = "csharp",
    ["page"] = "2",
};
builder.Query = await new FormUrlEncodedContent(q).ReadAsStringAsync();
Console.WriteLine(builder.Uri);
// https://example.com/search?q=csharp&page=2

using System;

var base2 = new Uri("https://example.com/blog/post");
var relative = new Uri("../about", UriKind.Relative);
var resolved = new Uri(base2, relative);
Console.WriteLine(resolved);
// https://example.com/about

using System;

var uri = new Uri("https://example.com:8080/path?q=hello#section");
Console.WriteLine(uri.Scheme);       // https
Console.WriteLine(uri.Host);         // example.com
Console.WriteLine(uri.Port);         // 8080
Console.WriteLine(uri.AbsolutePath); // /path
Console.WriteLine(uri.Query);        // ?q=hello
Console.WriteLine(uri.Fragment);     // #section

URL Parsing in Swift

Swift URL parsing uses two complementary types from the Foundation framework. URL parses and stores a URL value, providing access to the raw string and scheme, while URLComponents breaks the URL into mutable, individually writable properties¹ including scheme, host, port, path, and queryItems. URLQueryItem represents a single query parameter as a name-value pair, which replaces the error-prone manual string splitting used in other languages. Together they cover the full URL lifecycle: parsing, reading individual components, modifying them safely, and reconstructing the final URL string.

Core parsing API

Constructing a URL from a string uses URL(string: rawURL), which returns an Optional<URL> that is nil for invalid input. Consequently, for user input, URLComponents(string:) is safer because it also returns Optional and provides writable component properties. Once parsed, URLComponents exposes .scheme, .host, .port (Int?), .path, .query (raw query string), .fragment, and .queryItems, where the last property is a [URLQueryItem]? that parses key-value pairs automatically¹. Call components.url to reconstruct the URL from the current property values. For parsing user-supplied URLs, prefer URLComponents over URL because its component-level access lets you inspect and sanitize each part before reconstruction. This matters most when the input comes from a text field or a deep link, where malformed scheme or host values are common.

Query string and parameter handling

Accessing .queryItems on a URLComponents value returns [URLQueryItem]?, which is an array of name-value pairs where each URLQueryItem has a .name and an optional .value². For reading, filter the array: components.queryItems?.first(where: { $0.name == "page" })?.value. Building on this, modify query items by assigning a new array: components.queryItems = [URLQueryItem(name: "q", value: "swift url"), URLQueryItem(name: "page", value: "2")]. Call components.url to reconstruct the URL with the new items percent-encoded correctly. When a parameter name appears more than once, first(where:) returns only the first match, so use filter combined with compactMap to collect every value associated with that name.

Edge cases, encoding, and pitfalls

URLComponents percent-encodes query item names and values automatically when you assign to .queryItems, but it does not double-encode values that are already encoded. Yet if you assign directly to .query (the raw string), no encoding is applied, so that assignment is for pre-encoded query strings only. For relative URL resolution, use URL(string: relative, relativeTo: base), which returns an Optional<URL> and resolves the reference following RFC 3986³. Call .absoluteURL on the result to get the fully resolved URL. For encoding path segments, use addingPercentEncoding(withAllowedCharacters: .urlPathAllowed)³, and avoid .urlQueryAllowed here because that character set permits symbols that are not valid inside a path segment.

Reading individual query parameters from URLComponents

URLComponents.queryItems returns an array of URLQueryItem, each with a .name and an optional .value. To find a specific parameter, filter the array: components.queryItems?.first(where: { $0.name == "page" })?.value. Building on this, the return type is Optional<String> because the parameter may be absent or may have no value (for a parameter like ?debug, the URLQueryItem has name "debug" and value nil). For repeated keys, filter all matching items: components.queryItems?.filter { $0.name == "tag" }.compactMap { $0.value } gives you an array of all tag values. This functional approach is idiomatic Swift and avoids manual string splitting. For parameters that may appear multiple times, always use filter rather than first(where:) to avoid silently dropping duplicate values.

Building URLs safely with URLComponents and URLQueryItem

Building a URL from scratch starts with an empty URLComponents to which you assign .scheme, .host, .path, and an array of URLQueryItem values. This approach guarantees that each component is percent-encoded according to its own rules, so a query value containing an ampersand or equals sign will not corrupt the surrounding query string. Call .url on the populated components to produce the final Optional<URL>, and always handle the nil case rather than force-unwrapping, since an invalid scheme or empty host can cause the reconstruction to fail.

If you need to update an existing query without reconstructing the entire URL, assemble a new array of URLQueryItem values and assign it to .queryItems. Preserve the parameters you want to keep, substitute the ones you want to change, and let URLComponents re-encode everything in order. This round-trip through .queryItems is safer than string concatenation, especially when the URL contains repeated keys or values with special characters.

Resolving relative URLs and handling file URLs in Swift

URL(string: relative, relativeTo: base) resolves a relative reference against a base URL following RFC 3986. Building on this, the method returns an Optional<URL> that is nil if the relative string is not a valid reference. For file URLs, URL(string: "file:///Users/name/Documents/file.txt") parses with scheme "file", empty host, and path "/Users/name/Documents/file.txt". The isFileURL property returns true for file scheme URLs⁴. For iOS and macOS apps that handle file URLs from document pickers or drag-and-drop, use URL.startAccessingSecurityScopedResource() before reading the file, and call stopAccessingSecurityScopedResource() when done. This security-scoped access is required for files outside the app's sandbox, and forgetting to balance the start and stop calls will leak the security assertion.

URL encoding edge cases with emojis and multibyte characters

Emoji and multibyte characters in a query value must be UTF-8 percent-encoded before transmission, and URLComponents handles this automatically when you assign through .queryItems. The encoded form expands each byte into a percent-escaped triplet, so a single emoji can become nine or more characters in the final query string. When you need to encode a path segment that already contains non-ASCII text, use addingPercentEncoding(withAllowedCharacters: .urlPathAllowed) on the raw string before assigning it, since URLComponents does not re-encode the .path property the way it does for query items.

Comparing URLs and using URL as a dictionary key

URL implements Equatable and Hashable in Swift, making it usable as a dictionary key. Building on this, the equality comparison is based on the normalized absolute string: two URL values are equal if their absoluteString values match. The URL class normalizes the scheme and host to lowercase and removes the default port, so URL(string: "HTTPS://EXAMPLE.com:443/path") == URL(string: "https://example.com/path") evaluates to true⁴.

Persisting URLs across app launches

For cache key generation, use url.absoluteString as the key. For case-sensitive comparison (rarely needed), compare the url.againstStandardRules property or the raw string representation. When storing URLs in Core Data or UserDefaults, use the absoluteString for persistence and reconstruct with URL(string:) on retrieval. This round-trip is safe because absoluteString captures the fully resolved, normalized form of the URL, so two equivalent URLs produce identical stored strings.

Notes

Use URLComponents(string:) for parsing, which is safer than URL(string:) for user input. .queryItems returns [URLQueryItem]?, so filter by .name to find a specific param. Assign a new [URLQueryItem] array to modify query params safely. .url reconstructs the URL with correct encoding. Resolve relative URLs with URL(string: relative, relativeTo: base).

Examples

import Foundation

if let comps = URLComponents(string: "https://example.com:8080/path?q=hello#section") {
    print(comps.scheme!)    // https
    print(comps.host!)      // example.com
    print(comps.port!)      // 8080
    print(comps.path)       // /path
    print(comps.query!)     // q=hello
    print(comps.fragment!)  // section
}

import Foundation

let url = "https://example.com/search?q=swift+url&page=2&tag=ios&tag=macos"
if let comps = URLComponents(string: url) {
    let page = comps.queryItems?.first(where: { $0.name == "page" })?.value
    print(page ?? "not found")  // 2

    let tags = comps.queryItems?.filter { $0.name == "tag" }.compactMap { $0.value }
    print(tags ?? [])  // ["ios", "macos"]
}

import Foundation

var comps = URLComponents(string: "https://example.com/search?q=swift")!
comps.queryItems = [
    URLQueryItem(name: "q", value: "swift url parsing"),
    URLQueryItem(name: "page", value: "2"),
    URLQueryItem(name: "sort", value: "date"),
]
print(comps.url!)
// https://example.com/search?q=swift%20url%20parsing&page=2&sort=date

import Foundation

let base = URL(string: "https://example.com/blog/post")!
let relative = URL(string: "../about", relativeTo: base)!
print(relative.absoluteURL)
// https://example.com/about

import Foundation

if let comps = URLComponents(string: "https://example.com:8080/path?q=hello#section") {
    print(comps.scheme!)    // https
    print(comps.host!)      // example.com
    print(comps.port!)      // 8080
    print(comps.path)       // /path
    print(comps.query!)     // q=hello
    print(comps.fragment!)  // section
}

URL Parsing in Kotlin

Kotlin URL parsing builds on the Java standard library, and java.net.URI is accessible from Kotlin with no extra imports needed.¹ Idiomatic Kotlin wraps URI parsing in extension functions and safe parsing patterns using runCatching and Result. For Ktor-based projects, the io.ktor.http package provides a dedicated Url class and ParametersBuilder for type-safe URL construction.² Kotlin Multiplatform code targeting non-JVM platforms can use Ktor's URL utilities, which implement the same API across Android, iOS, and Desktop without JVM dependencies.²

Core parsing API

Calling URI.create("https://example.com:8080/path?q=hello#section") in Kotlin returns a URI object whose accessors for scheme, host, port, path, rawQuery, and fragment are identical to the Java API.¹ Idiomatic Kotlin wraps this in a safe constructor so runCatching plus URI.create plus getOrNull returns null for malformed input instead of throwing an exception you would otherwise have to catch.

Parsing the components

Prefer rawQuery over query when you need the encoded string exactly as it was sent because query decodes percent-encoding while rawQuery preserves it. Ktor's Url class exposes protocol, host, port, fullPath, and parameters as a QueryParameters map that handles multi-value keys natively. You can read a single parameter with url.parameters indexed by page or fetch every value for a repeated key with url.parameters.getAll of page.

Ktor's QueryParameters map handles both single keys and repeated keys through the same accessor, which removes the special-case parsing you would otherwise need in Ktor code. The type-safe accessors also suppress any ambiguity about whether a parameter was absent or present with an empty value. This handling is built into the Ktor client and server APIs, so the same code reads parameters from an incoming request and from a URL you construct yourself.

Query string and parameter handling

Parsing query parameters from a java.net.URI is concise with Kotlin extension functions. Building on this: uri.rawQuery?.split("&")?.map { it.split("=", limit = 2) }?.associate { it[0] to it.getOrElse(1) { "" } } produces a Map<String, String> in one expression. The limit = 2 prevents splitting on = characters inside encoded values, which matters when a value itself contains an equals sign that has been percent-encoded by the sender. For Ktor, access url.parameters["key"] for a single value or url.parameters.getAll("key") for all values of a repeated key. Building URLs uses URLBuilder with .parameters.append("key", "value"), and you can call .parameters.clear() followed by a series of appends when you are constructing a URL dynamically in a loop.

Edge cases, encoding, and pitfalls

java.net.URL.equals() performs DNS lookups, and that JVM behavior does not change in Kotlin.³ Always use URI instead of URL for comparison and collection storage, since two URIs with the same host string compare equal without touching the network. For encoding, java.net.URLEncoder.encode(value, Charsets.UTF_8.name()) produces application/x-www-form-urlencoded output with + for spaces, which is intended for HTML form payloads.⁴ Ktor's encodeURLQueryComponent(value) produces RFC 3986 percent-encoding with %20 for spaces, which is what servers expect when decoding raw URL components.⁵ Pick the encoding that matches what the server decodes on the other side, since using the wrong one is a common source of bugs when a server receives a literal plus where your client meant a space.

Ktor URLBuilder and type-safe URL construction

Ktor's URLBuilder provides a type-safe way to construct URLs with explicit protocol, host, port, path segments, and query parameters. Each path segment is added with .path('segment1', 'segment2'), which handles encoding and slash insertion automatically. Building on this, .parameters.append('key', 'value') adds query parameters with correct encoding. The .build() method returns a Url object, and .toString() produces the full URL string. For Ktor client requests, you can pass a URLBuilder directly to the request builder: client.get(URLBuilder().apply { protocol = URLProtocol.HTTPS; host = 'api.example.com'; path('v1', 'users'); parameters.append('page', '2') }.build()). Building URLs piece by piece in a view model or repository layer keeps the construction logic testable, and you can unit-test a URLBuilder without spinning up an HTTP client.

Handling URL encoding differences between JVM and Ktor

The JDK's URLEncoder.encode produces application/x-www-form-urlencoded output, which encodes spaces as plus signs and is intended for form data rather than URL components. Ktor's encodeURLQueryComponent follows RFC 3986 by encoding spaces as percent-encoded %20, which is what most web servers expect in URL query strings and paths. When you are building a query string to send with Ktor client calls or to append to a URL path, prefer Ktor's component-level encoding helpers so that the encoding matches what the target server parses. Mixing the two encodings inside the same URL is a common source of subtle bugs, especially when the server decodes a plus sign as a literal plus rather than a space.

Kotlin Multiplatform URL parsing strategies

Kotlin Multiplatform projects face a challenge: java.net.URI is available on JVM but not on native or WASM targets. Ktor's io.ktor.http package provides Url, URLBuilder, and related utilities that work across all Kotlin targets. Building on this, for a multiplatform project, define an expect/actual interface for URL parsing: the JVM actual implementation delegates to java.net.URI, while the native actual implementation uses Ktor's Url. This gives you the best performance on JVM (native JDK implementation) and consistent behavior on other targets. For projects that already depend on Ktor for HTTP, using Ktor's URL utilities everywhere avoids the platform-specific branching entirely.⁶ On JVM the Ktor Url implementation itself delegates internally to java.net.URI, so you get the same parsing behavior on every platform while keeping a single import path in your common source set.

Safe URL parsing patterns with Kotlin null safety

Kotlin's null safety makes URL parsing safer than in Java by forcing callers to acknowledge failures explicitly. The idiomatic pattern is runCatching { URI.create(input) }.getOrNull(), which returns null for malformed URLs instead of throwing a checked exception you would otherwise have to catch. Calling code then uses a null check or a safe call like parsed?.host, which the type system enforces. On top of that, the Result returned from runCatching can be chained with .map() to extract a component, so the happy path stays on the left margin: runCatching { URI.create(input) }.map { it.host }.getOrNull() gives you the host or null in one expression. For Ktor's Url class you can wrap the parse step the same way, letting the type system track whether parsing succeeded.

Android deep links and URL parsing in intent filters

Android apps register deep link patterns in the manifest with intent filters. When the system delivers an intent, the URL is available as intent.data, a Uri object (Android's android.net.Uri, not java.net.URI). Building on this, Android's Uri class exposes .scheme, .host, .path, .getQueryParameter('key'), and .lastPathSegment(). The API is similar to java.net.URI but not identical: getQueryParameter returns null for missing keys, and getQueryParameterNames() returns a Set of all parameter names. For apps using Jetpack Navigation with deep links, the navigation framework parses the URL and extracts arguments defined in the navigation graph. Parse the URL once in your navigation handler and pass the extracted values to your screens as typed arguments rather than passing the raw URL string.

Notes

Use URI.create(str) (not URL) for parsing. Wrap in runCatching { } for safe parsing of user input since it returns null on failure. .rawQuery returns the unencoded query string. For multi-type support, use Ktor's io.ktor.http.Url and URLBuilder. java.net.URLEncoder.encode(val, Charsets.UTF_8.name()) for query encoding.

Examples

import java.net.URI

fun main() {
    val uri = URI.create("https://example.com:8080/path?q=hello#section")
    println(uri.scheme)    // https
    println(uri.host)      // example.com
    println(uri.port)      // 8080
    println(uri.path)      // /path
    println(uri.rawQuery)  // q=hello
    println(uri.fragment)  // section
}

import java.net.URI
import java.net.URLDecoder

fun URI.queryParams(): Map<String, String> =
    rawQuery?.split("&")
        ?.map { it.split("=", limit = 2) }
        ?.associate {
            URLDecoder.decode(it[0], "UTF-8") to
            URLDecoder.decode(it.getOrElse(1) { "" }, "UTF-8")
        } ?: emptyMap()

fun main() {
    val uri = URI.create("https://example.com/search?q=kotlin+url&page=2")
    val params = uri.queryParams()
    println(params["q"])     // kotlin url
    println(params["page"])  // 2
}

import io.ktor.http.*

fun main() {
    val url = URLBuilder().apply {
        protocol = URLProtocol.HTTPS
        host = "api.example.com"
        path("v1", "users")
        parameters.append("role", "admin")
        parameters.append("page", "2")
    }.build()
    println(url.toString())
    // https://api.example.com/v1/users?role=admin&page=2
}

import java.net.URI

fun parseUrl(input: String): URI? =
    runCatching { URI.create(input) }.getOrNull()

fun main() {
    val good = parseUrl("https://example.com/path")
    val bad  = parseUrl("not a url")
    println(good?.host)  // example.com
    println(bad)         // null
}

import java.net.URI

fun main() {
    val uri = URI.create("https://example.com:8080/path?q=hello#section")
    println(uri.scheme)    // https
    println(uri.host)      // example.com
    println(uri.port)      // 8080
    println(uri.path)      // /path
    println(uri.rawQuery)  // q=hello
    println(uri.fragment)  // section
}

URL Parsing in Bash

Bash has no built-in URL parser, but standard POSIX tools (grep, sed, awk, and parameter expansion) can extract every URL component from a shell script.¹ Pattern-matching with extended regex isolates the scheme, authority, path, query string, and fragment reliably. For scripts that run in environments with Python or Node.js available, delegating to those runtimes produces cleaner code. Yet for minimal containers and CI jobs where only coreutils are guaranteed, pure Bash parameter expansion handles the most common cases without any subprocess overhead.

Core parsing API

Bash parameter expansion strips URL components using prefix and suffix patterns without spawning subprocesses. Given url="https://user:[email protected]:8080/path?q=hello#top", extract the scheme with ${url%%:*} which strips from the first colon to the end, and strip the fragment with ${url##*#} which removes everything before the last number sign. The authority requires stripping the scheme first with ${url#*//}, then stripping everything from the first slash with ${no_scheme%%/*} to isolate the host and port. These pure-Bash patterns are fast, portable, and require no external binaries since they work in any POSIX shell, but they fail on edge cases like URLs without a host or with IPv6 addresses in brackets.¹

Query string and parameter handling

Extracting the raw query string uses ${url##*\?} to strip everything before the last question mark. Building on this, parse individual parameters by splitting the query string on ampersand: query="${url##*\?}" followed by IFS='&' read -ra pairs <<< "$query_string" gives you an array of key-value strings. Split each pair on the equals sign with IFS='=' read -r key val <<< "$pair", then decode the value with printf '%b' "${val//%/\\x}" to expand percent-encoded sequences into their original bytes.² For Unicode-aware decoding where the input contains multi-byte UTF-8 sequences like percent-encoded e-acute or CJK characters, python3 or perl with the right locale settings is more reliable than pure Bash printf.

Edge cases, encoding, and pitfalls

Parameter expansion patterns assume a simple URL structure and break when the input omits an authority, uses IPv6 literal addresses in square brackets, or embeds unescaped fragment markers inside the query portion. For the common CI use case of extracting a hostname or path from a well-formed API response URL, the patterns are fast and sufficient. When your script must handle untrusted or varied URLs, delegate to python3 or node via an argv argument rather than string interpolation. Writing python3 -c "from urllib.parse import urlparse; print(urlparse(sys.argv[1]).hostname)" "$URL" keeps the value in a positional parameter so shell metacharacters inside it are never reinterpreted as part of the command.

Parsing URLs from curl output and HTTP responses

Curl writes the effective URL to stderr after a redirect chain, but extracting specific components from curl output requires additional parsing. The -w flag with format variables gives you individual components: curl -s -o /dev/null -w '%{url_effective}' https://bit.ly/xxx returns the final resolved URL after all redirects. For response headers, curl -I fetches only headers, and you can grep for Location to extract redirect targets. Building on this, the -w flag also exposes %{redirect_url} for the immediate next hop, which is useful when you need to inspect a single redirect step rather than the full chain.

Extracting URLs from JSON API responses in Bash

APIs often return URLs as string values in JSON responses. jq extracts these without regex: curl -s https://api.example.com/endpoint | jq -r '.url' gives you the raw URL string. If the URL contains percent-encoded characters, jq preserves them as-is. Pipe the extracted value to a further parsing step: URL=$(curl -s ... | jq -r '.url') && python3 -c "from urllib.parse import urlparse; u=urlparse('$URL'); print(u.hostname)". This pattern of extracting with jq and parsing with Python works reliably in CI pipelines where coreutils and Python are available but nothing else is guaranteed.

When the API returns an array of objects rather than a single value, jq can emit one URL per line with .[].url or map(select(.url)|.url), and a while-read loop feeds each line to the Python parser. Because jq outputs newline-separated values, the loop stays simple and avoids the quoting traps that appear when URLs are embedded inside a larger shell string. This keeps the parsing step independent of how many URLs the endpoint returns.

Node.js as an inline URL parser in minimal containers

When Python is not available in a minimal container, Node.js can serve as an inline URL parser: node -e "const u=new URL(process.argv[1]); console.log(u.hostname)" "$URL". This works on any system with Node.js installed, which includes most CI runners (GitHub Actions, GitLab CI, CircleCI). The overhead of spawning a Node.js process for each URL makes this impractical for bulk parsing, but for one-off extractions in deployment scripts or health check commands, the approach is reliable and handles all edge cases the WHATWG URL Standard covers.³ Node scripts let you validate and resolve relative URLs in pure JavaScript without any native dependencies, which is handy in minimal container images where Python has been stripped out.

Building URLs from shell variables with jq encoding

Constructing URLs from shell variables requires percent-encoding variable values to prevent injection. A value like category="food & drink" breaks a URL if concatenated directly into the query string because the ampersand and space would be misinterpreted by the server. Use jq to encode the value safely: encoded=$(jq -rn --arg v "$category" '$v|@uri') produces food%20%26%20drink, where the @uri filter in jq follows RFC 3986 encoding rules and handles every special character including spaces, ampersands, and plus signs.⁴ Concatenate the encoded value into the full URL with a simple variable expansion: url="https://example.com/items?category=$encoded". This approach avoids external encoding tools and works in any environment where jq is installed alongside your shell scripts.

Parsing URLs in awk for log file analysis

Awk processes URL log files efficiently without spawning external processes per line. Use match() with a regex to extract components from each URL in an access log: match($0, /^https?:\/\/([^/:]+)(:[0-9]+)?(\/[^?#]*)?(\?([^#]*))?(#(.*))?/, m) populates array m with host, port, path, query, and fragment. This regex handles the most common HTTP(S) URL formats but does not validate; an invalid URL produces empty capture groups rather than an error. For logs with millions of lines, awk processes data orders of magnitude faster than a Bash loop that calls Python or Node.js per line.

GNU awk versus POSIX awk for URL parsing

GNU awk (gawk) supports the third-argument match() function for capture groups, which makes component extraction concise.⁵ POSIX awk does not support capture groups in match(), requiring substr() manipulation that is more verbose and error-prone.⁶ For portability across macOS (which ships BSD awk) and Linux (which ships gawk), test your awk scripts with both variants or require gawk explicitly. If your script runs in a CI environment where gawk is guaranteed, the capture group syntax is the cleanest approach for field extraction from URLs in log files.

Notes

No native URL parser in Bash. Use parameter expansion: ${url%%:*} for scheme, ${url##*#} for fragment. Extract query string with ${url##*\?}. Split query on & with IFS='&' read -ra pairs. Percent-decode with printf '%b' "${enc//%/\\x}". For edge cases, delegate to python3 or node with the URL as an argv argument (not interpolated).

Examples

url="https://user:[email protected]:8080/path?q=hello&page=2#section"

scheme="${url%%:*}"
echo "$scheme"      # https

no_scheme="${url#*//}"
authority="${no_scheme%%/*}"
echo "$authority"   # user:[email protected]:8080

with_path="${no_scheme#*/}"
path="/${with_path%%\?*}"
echo "$path"        # /path

query="${url##*\?}"
query="${query%%#*}"
echo "$query"       # q=hello&page=2

fragment="${url##*#}"
echo "$fragment"    # section

url="https://example.com/search?q=bash+url&page=2&sort=date"
query="${url##*\?}"

declare -A params
IFS='&' read -ra pairs <<< "$query"
for pair in "${pairs[@]}"; do
    IFS='=' read -r key val <<< "$pair"
    params["$key"]="$val"
done

echo "${params[q]}"     # bash+url
echo "${params[page]}"  # 2
echo "${params[sort]}"  # date

url="https://api.example.com/v1/users?role=admin&page=2"

# Safe: handles IPv6, encoded chars, edge cases
host=$(python3 -c "from urllib.parse import urlparse; print(urlparse(sys.argv[1]).hostname)" "$url")
path=$(python3 -c "from urllib.parse import urlparse; print(urlparse(sys.argv[1]).path)" "$url")

echo "host: $host"   # api.example.com
echo "path: $path"   # /v1/users

decode_url() {
    printf '%b' "${1//%/\\x}"
}

encoded="hello%20world%2C%20goodbye"
decoded=$(decode_url "$encoded")
echo "$decoded"  # hello world, goodbye

url="https://user:[email protected]:8080/path?q=hello&page=2#section"

scheme="${url%%:*}"
echo "$scheme"      # https

no_scheme="${url#*//}"
authority="${no_scheme%%/*}"
echo "$authority"   # user:[email protected]:8080

with_path="${no_scheme#*/}"
path="/${with_path%%\?*}"
echo "$path"        # /path

query="${url##*\?}"
query="${query%%#*}"
echo "$query"       # q=hello&page=2

fragment="${url##*#}"
echo "$fragment"    # section

URL Parsing in Node.js

Node.js provides two URL parsing APIs: the modern WHATWG URL class (available globally since Node 18) and the legacy url module with url.parse() and url.format(). The WHATWG URL class, identical to the browser's URL API, is the correct choice for all new code. It parses URLs into the same components as browser environments, handles query strings via URLSearchParams, and works without any import in modern Node.js. The legacy url.parse() is deprecated as of Node 18 and will eventually be removed.¹

Core parsing API

new URL("https://example.com:8080/path?q=hello#section") works identically in Node.js as in a browser, returning an object with .protocol, .hostname, .port, .pathname, .search, .hash, .origin, and .href. Consequently, in Node.js scripts running before v18, import the WHATWG URL explicitly: import { URL } from 'node:url' or const { URL } = require('node:url'). Since Node 18, URL is also available as a global, so no import is required². The .search property holds the raw query string with the leading question mark, while .href gives the full serialized URL after normalization. The legacy url.parse(rawUrl) returns a plain object with different property names (.query is a string, not an object) and is no longer maintained. Prefer the WHATWG URL class even when the legacy module still works, because the WHATWG API stays consistent across Node.js versions and browser environments.

Query string and parameter handling

Access url.searchParams to get a URLSearchParams object with .get(), .getAll(), .set(), .append(), .delete(), and .has() methods. Building on this, iteration with for (const [key, value] of url.searchParams) works identically to browser URLSearchParams. For standalone query string parsing, import { URLSearchParams } from 'node:url' and construct directly: new URLSearchParams("key=value&page=2"). The legacy querystring module is deprecated; use URLSearchParams instead³. Converting to a plain object: Object.fromEntries(url.searchParams) collapses repeated keys to the last value only. When you need to preserve every value for a repeated key, call url.searchParams.getAll("key") instead, which returns an array of every value that appeared in the query string. This matters when a query string contains multiple values for the same key, such as ?tag=javascript&tag=nodejs, because calling the get method for that key returns only the first value while the getAll method returns every value in an array.

Edge cases, encoding, and pitfalls

The WHATWG URL constructor throws TypeError for invalid URLs. Wrap it in try/catch when parsing user-supplied input⁴. Relative URLs require a second base argument: new URL("/path", "https://example.com") works, while new URL("/path") throws. Yet the base URL can use custom schemes, and file:// and other schemes work for validation purposes. The WHATWG URL class normalizes the URL on construction by lowercasing scheme and host, removing default ports from the origin, and percent-encoding the path if needed. Protocol-relative URLs like //example.com/path require a prepended scheme before parsing. The WHATWG URL class also percent-encodes non-ASCII characters in the path using UTF-8, so new URL("https://example.com/café") produces pathname "/caf%C3%A9" without any manual encoding step. This automatic encoding applies to every component, including query parameter values that you add through the searchParams set method, which handles the percent-encoding internally so you never have to call encodeURIComponent() yourself.

File URL handling and pathToFileURL in Node.js

Node.js provides two utility functions for converting between file system paths and file:// URLs: pathToFileURL and fileURLToPath, both from the node:url module. pathToFileURL("/Users/name/file.txt") returns a URL object with href "file:///Users/name/file.txt", correctly adding the three slashes and percent-encoding special characters. The reverse conversion, fileURLToPath(url), handles both file:/// and file://localhost/ forms and decodes percent-encoded characters back to their original form.

Why file URLs need special treatment

File URLs behave differently from HTTP URLs in several ways that trip up developers. The file:///etc/hosts URL has an empty hostname, so url.hostname returns an empty string and url.origin returns "null" in Node.js⁵. A file URL with a non-empty hostname like file://localhost/etc/hosts is equivalent to file:///etc/hosts, but the WHATWG URL Standard treats the hostname as significant. When your code handles both HTTP and file URLs, check url.protocol === "file:" before accessing hostname-dependent properties to avoid null-origin edge cases.

For converting between the two forms, use the dedicated helpers instead of string concatenation: pathToFileURL keeps the drive letter and percent-encodes spaces, while fileURLToPath reverses that process and decodes the result back to a system path. This round trip is the recommended way to pass a file path from your script into a URL-aware API such as a fetch call or a module loader, because it respects platform-specific path rules that manual string building would miss.

URL validation patterns for Express and Fastify middleware

Validating URLs in Node.js middleware requires handling the TypeError that new URL() throws for invalid input. A reusable validation helper wraps the constructor and returns a structured result: function validateUrl(input) { try { const u = new URL(input); return { valid: true, url: u }; } catch { return { valid: false, url: null }; } }. This pattern works in Express route handlers, Fastify preValidation hooks, and any middleware layer that needs to reject malformed URLs before they reach business logic.

Validating URLs in request headers and redirect targets

When your application receives URLs from request headers (Origin, Referer) or constructs redirect targets from user input, validate the URL before using it. A redirect endpoint that takes a ?url= parameter should parse the value, check that the protocol is http: or https:, and verify the hostname against an allowlist before issuing the 302 response. Without hostname validation, an attacker can craft a URL like /redirect?url=https://evil.com and use your domain as an open redirector, which damages your domain's reputation in email spam filters and phishing detection systems.

Performance of WHATWG URL versus legacy url.parse()

The WHATWG URL class is slower than the legacy url.parse() function because it performs full validation, IDNA host checking, and normalization on every construction call. Benchmarks on Node 20 show url.parse() processing approximately 2 million URLs per second, while new URL() processes around 500,000 per second⁶. For applications that parse millions of URLs per second, such as log processors that parse access logs in real time or web crawlers that extract links from billions of pages, the 4x performance difference between the two parsers adds up to measurable infrastructure cost. However, url.parse() returns a plain object with no validation, meaning malformed URLs produce silently wrong results rather than throwing.

When to use each parser in production code

Use new URL() for all user-facing code: API route handlers, middleware, configuration parsing, and any context where a malformed URL should produce an error. Reserve url.parse() for internal batch processing of known-good URLs where you control the input format and need maximum throughput. Node.js documentation marks url.parse() as legacy, so any new code using it will need migration before the function is eventually removed. The CapyToolkit URL Parser tool uses the WHATWG URL Standard in the browser, which is functionally identical to Node.js's new URL() implementation.

Notes

Use the WHATWG URL class (global in Node 18+ or from 'node:url'). url.parse() is deprecated since Node 18. Access .searchParams for query parameters. Import URLSearchParams from 'node:url' for standalone parsing. Relative URLs need a base: new URL(relative, base). Use encodeURIComponent() for individual query values. Object.fromEntries(url.searchParams) gives a plain object (last value wins for duplicates).

Examples

const url = new URL('https://example.com:8080/path?q=hello#section');
console.log(url.protocol);  // 'https:'
console.log(url.hostname);  // 'example.com'
console.log(url.port);      // '8080'
console.log(url.pathname);  // '/path'
console.log(url.search);    // '?q=hello'
console.log(url.hash);      // '#section'

import { URL } from 'node:url';

const url = new URL('https://example.com/search?q=nodejs+url&page=2');
console.log(url.searchParams.get('q'));    // 'nodejs url'
console.log(url.searchParams.get('page')); // '2'

const url = new URL('https://example.com?q=node&page=1&tag=http&tag=url');

for (const [key, val] of url.searchParams) {
  console.log(`${key}: ${val}`);
}

url.searchParams.set('page', '2');
url.searchParams.delete('tag');
console.log(url.href);
// https://example.com/?q=node&page=2

import { URL } from 'node:url';

const resolved = new URL('../about', 'https://example.com/blog/post');
console.log(resolved.href);
// 'https://example.com/about'

const url = new URL('https://example.com:8080/path?q=hello#section');
console.log(url.protocol);  // 'https:'
console.log(url.hostname);  // 'example.com'
console.log(url.port);      // '8080'
console.log(url.pathname);  // '/path'
console.log(url.search);    // '?q=hello'
console.log(url.hash);      // '#section'

Python urllib.parse Module Reference

Python's urllib.parse module handles every aspect of URL manipulation without any third-party libraries.¹ It provides urlparse() and urlsplit() for decomposing URLs, parse_qs() and parse_qsl() for query strings, urlencode() for building query strings, urljoin() for resolving relative URLs, and quote() / unquote() for percent-encoding. All functions work with Python 3's str type and handle Unicode correctly. The module is part of the standard library, importable as from urllib.parse import urlparse, urljoin with no pip install required.

Core parsing API

urlparse(url) returns a ParseResult named tuple with six fields: scheme, netloc, path, params, query, and fragment.² The .hostname, .port, .username, and .password properties extract sub-fields from netloc automatically, so you do not need to parse the authority string yourself. Consequently, urlsplit(url) is nearly identical but omits the rarely used params field, returning a SplitResult with five fields instead of six. For most modern code, urlsplit is cleaner and recommended. Both functions accept url_encoded = False to skip automatic decoding of percent-encoded characters, giving you the raw encoded strings in the returned fields. This is useful when you need to inspect the original encoding of a URL that has already been processed.

Query string and parameter handling

parse_qs(query_string) returns a dict where values are always lists, even for single-value keys, preventing bugs when a key appears multiple times.³ parse_qsl(query_string) returns a list of (key, value) tuples, preserving order and allowing duplicate keys without collapsing. Building on this, urlencode(params) converts a dict or list of tuples back into a query string. Passing doseq=True handles dict values that are lists: urlencode({'tag': ['js', 'python']}, doseq=True) produces "tag=js&tag=python". Without doseq, the list is stringified as its Python repr, producing the unusable "tag=%5B%27js%27%2C+%27python%27%5D".

Choosing the right encoding function

quote(string, safe='') encodes special characters for URL safety, and unquote(string) reverses the encoding. The safe parameter controls which characters remain unencoded, with the default safe='/' preserving path separators. Use safe='' when encoding individual path segments so that slashes inside the segment are also percent-encoded. For query string keys and values, call quote_plus() instead, which converts spaces to plus signs and is the format expected by servers parsing application/x-www-form-urlencoded data.

Edge cases, encoding, and pitfalls

urljoin(base, url) resolves relative URLs following RFC 3986.⁴ urljoin('https://example.com/blog/', 'post') produces 'https://example.com/blog/post', but urljoin('https://example.com/blog', 'post') produces 'https://example.com/post' because the base path directory depends on whether the base ends with a slash. This trailing slash distinction is a common source of bugs when constructing base URLs dynamically. Consequently, quote() uses %20 for spaces (RFC 3986), while quote_plus() uses + (form data format).³ Use quote_plus only for query values in form submissions and use quote for path segments. Always call unquote_plus() to decode values encoded with quote_plus, since calling unquote() on a plus sign leaves it unchanged instead of converting it to a space.

The params field in urlparse and why urlsplit is preferred

urlparse returns a six-element ParseResult that includes a params field, which captures semicolon-delimited path parameters. This feature dates back to early RFCs and is almost never used in modern URLs. Building on this, urlsplit returns a five-element SplitResult without params, merging any semicolon content into the path.² For all modern HTTPS URLs, params is always empty, and urlsplit produces identical output with one fewer field to consider. The Python documentation recommends urlsplit for new code. If you are maintaining legacy code that uses urlparse, be aware that the params field exists but will be empty for any URL you encounter in practice. Switching to urlsplit is a drop-in replacement that simplifies your code.

urljoin behavior with absolute URLs and path edge cases

When the second argument to urljoin is an absolute URL (a string that starts with a scheme like https://), the function returns it completely unchanged and discards the base URL. This matches how browsers resolve absolute links in HTML documents. However, when the second argument is a relative URL that starts with a single slash (such as /about), urljoin replaces the entire path of the base with the new one while preserving the scheme and netloc. When the argument starts with no slash (such as about or ../about), urljoin resolves it relative to the base path, removing the last segment if the base does not end with a slash. These resolution rules follow RFC 3986 Section 5.2 and are consistent across Python versions.

Because the trailing slash changes the result so dramatically, normalize your base URLs once at startup rather than relying on each call site to remember the rule. A helper function that appends a slash when one is missing, then delegates to urljoin, makes every join predictable. This removes the most common cause of broken links in generated navigation and asset paths.

Encoding query parameters with urlencode and doseq

urlencode converts a dict or list of tuples into a percent-encoded query string. The critical parameter is doseq: when False (the default), dict values that are lists are stringified as their Python repr.⁵ When True, each element of a list value becomes a separate key=value pair. Building on this, urlencode({'tag': ['js', 'python']}, doseq=True) produces 'tag=js&tag=python', while urlencode({'tag': ['js', 'python']}) produces 'tag=%5B%27js%27%2C+%27python%27%5D' (the stringified list). Always pass doseq=True when your dict values may contain lists. For APIs that require sorted query parameters (for signature generation), use a list of tuples instead of a dict and sort it before passing to urlencode. This preserves the order you specify.

Decoding percent-encoded strings with unquote versus unquote_plus

unquote(string) decodes percent-encoded bytes back to their original characters, leaving plus signs unchanged. unquote_plus(string) does the same decoding but also converts plus signs to spaces, matching the form-encoding convention used by HTML form submissions. Calling unquote() on a query value that was encoded with quote_plus() leaves the plus signs intact, producing output like "hello+world" instead of "hello world". Always pair the encoding and decoding functions: use quote() with unquote() for path segments, and quote_plus() with unquote_plus() for query values. For data that may contain mixed encoding, inspect the input first to determine which function was used, since applying the wrong decoder silently produces incorrect output without raising an exception.

Parsing and building URLs in async Python code

urllib.parse functions are synchronous and CPU-bound, which means they do not block the event loop in async Python code. Building on this, for high-throughput async applications (such as an aiohttp server processing thousands of URLs per second), urlparse runs fast enough that offloading to a thread pool is unnecessary.⁵ The pure Python implementation in CPython processes a typical URL in a fraction of a microsecond, and the module uses functools.lru_cache to speed up repeated calls to urlsplit() and quote(). For applications that need to parse URLs from untrusted sources, wrap urlparse in a try/except to catch any unexpected errors, though urlparse itself rarely raises exceptions. The main concern with untrusted input is not parsing failure but semantic validation: a successfully parsed URL may still point to an unexpected host or contain malicious query parameters. Always validate the parsed components after parsing.

Notes

from urllib.parse import urlparse, parse_qs, urlencode, urljoin, quote, unquote. urlparse() → ParseResult with .scheme, .netloc, .path, .query, .fragment, .hostname, .port. parse_qs() → dict of lists (always). parse_qsl() → list of tuples (order-preserving). urlencode(d, doseq=True) → query string from dict of lists. urljoin(base, rel) → absolute URL. quote(s, safe="") → RFC 3986. quote_plus(s) → form data (+).

Examples

from urllib.parse import urlparse, parse_qs

url = 'https://user:[email protected]:8080/path?q=hello&tag=python&tag=url#top'

r = urlparse(url)
print(r.scheme)     # https
print(r.hostname)   # example.com
print(r.port)       # 8080
print(r.username)   # user
print(r.path)       # /path
print(r.query)      # q=hello&tag=python&tag=url
print(r.fragment)   # top

params = parse_qs(r.query)
print(params['tag'])  # ['python', 'url']

from urllib.parse import urlencode

# Simple dict
params = {'q': 'python url', 'page': 2, 'sort': 'date'}
print(urlencode(params))
# q=python+url&page=2&sort=date

# Multi-value keys with doseq
params2 = {'tag': ['python', 'web', 'url'], 'page': 1}
print(urlencode(params2, doseq=True))
# tag=python&tag=web&tag=url&page=1

from urllib.parse import urljoin

base = 'https://example.com/blog/'
print(urljoin(base, 'post/1'))          # https://example.com/blog/post/1
print(urljoin(base, '../about'))         # https://example.com/about
print(urljoin(base, '/home'))            # https://example.com/home
print(urljoin(base, 'https://other.com/')) # https://other.com/

from urllib.parse import quote, unquote, quote_plus, unquote_plus

# Path segments: use quote
path_seg = quote('my folder/file name.txt', safe='/')
print(path_seg)  # my%20folder/file%20name.txt

# Query values: use quote_plus
query_val = quote_plus('hello world & goodbye')
print(query_val)   # hello+world+%26+goodbye

print(unquote('%C3%A9'))      # é
print(unquote_plus('a+b'))   # a b

from urllib.parse import urlparse, parse_qs

url = 'https://user:[email protected]:8080/path?q=hello&tag=python&tag=url#top'

r = urlparse(url)
print(r.scheme)     # https
print(r.hostname)   # example.com
print(r.port)       # 8080
print(r.username)   # user
print(r.path)       # /path
print(r.query)      # q=hello&tag=python&tag=url
print(r.fragment)   # top

params = parse_qs(r.query)
print(params['tag'])  # ['python', 'url']