Token Counter Reference

What Is an AI Token?

When people first encounter AI billing, the word "token" appears everywhere without explanation. A token is not a word, not a character, and not a sentence; it is a variable-length text segment that a specific model's vocabulary treats as a single unit. Understanding tokens is foundational to understanding both the cost and the capacity limits of every major language model API.

What is a token?

Tokens versus words and characters

Most common English words are one token each in modern large vocabularies, which is why the mapping feels intuitive at first². Consequently, "the" is one token, "running" is one token, and "tokenization" is often one single token in newer 100K+ vocabulary models³. Older or smaller vocabularies split uncommon words more aggressively: "tokenization" might become "token" plus "ization", producing two tokens where a human reader sees one.

Characters and tokens are fundamentally distinct units that beginners often conflate. No tokenizer counts individual characters independently; the letter "a" in the middle of "cat" is part of the whole-word token, not a separate unit the model processes on its own. Numbers, URLs, and code with special characters almost always produce far more tokens per character than plain prose, which is why a 200-character code snippet can easily consume more tokens than a 200-character sentence.

Seeing why one word can become multiple tokens

Try a technical phrase like tokenization_qwen3_5.py or a URL with query parameters. Those strings often split into many tokens because punctuation, underscores, and rare word pieces are not treated as ordinary English words by the BPE algorithm. The string tokenization_qwen3_5.py might split into five or more tokens such as token, ization, _, qwen, 3, _, 5, ., py, even though it is a single logical unit to a human reader; this fragmentation is exactly why code-heavy prompts consume disproportionately more tokens than prose of equivalent visual length.

Why tokens are the billing and capacity unit

Language models process sequences of token IDs in a fixed-size matrix called the context window, and every operation the model performs is counted in tokens rather than words or characters at the hardware level. The cost of running the model scales directly with the number of tokens processed, which is why API providers bill per token rather than per word or per request: tokens are the actual unit of computation the GPU executes.

How per-token billing connects to context window capacity

Furthermore, the context window measures capacity in tokens because that is the unit the model actually handles internally: no meaningful word limit exists at the technical level, only a token limit that varies by model architecture. Both the cost you pay and the capacity you have available are denominated in tokens from the model's perspective. This shared unit means that every token you save through prompt compression directly increases the amount of context you can fit within the same request, which is why efficient prompt writing is not just a cost optimization but also a capacity optimization that lets you include more retrieved documents, longer conversation histories, or more detailed system instructions within the same context window.

Tokens in practice for developers

Developers most often encounter tokens in three scenarios: billing calculations, context window management, and rate limit monitoring. For billing, the token count of the input and output combined determines the cost of each API call. For context management, the accumulated tokens in a conversation thread or RAG assembly must stay below the model's limit.

For rate limits, providers cap on tokens per minute rather than requests per minute, so a single very long prompt can consume a large fraction of the rate limit budget. Building a clear mental model of token counts (not word counts) is the foundational skill for working with AI APIs cost-efficiently.

Using token counts for three API decisions

Check tokens before sending a request, before choosing a model, and before setting a context or rate-limit guardrail. Before sending, the count tells you whether the prompt fits within the context window and what the input cost will be; before choosing a model, the count reveals which models can accommodate the prompt and at what cost per token; before setting guardrails, the count determines where to place soft limits that trigger summarization or pruning before the hard context ceiling is reached.

Make the count check a named step in your API call wrapper rather than an afterthought, because a guardrail that lives only in documentation is the first thing dropped under deadline pressure. Surfacing the input cost and fit check at the moment the request is built turns three separate decisions into one visible gate that every call passes through.

Counting tokens before every API call

Counting tokens programmatically before each API call is the most reliable way to prevent context overflow and billing surprises. Both Anthropic and OpenAI provide tokenizer libraries that run in Node.js and Python, giving you the same count the API applies to your request. For Claude models, install @anthropic-ai/tokenizer and call countTokens(text); for GPT models using o200k_base, install tiktoken and use encoding.encode(text).length. Both libraries produce exact counts rather than estimates, so your pre-flight check is as accurate as the billing system.

Add a validation step before every API call that counts the full assembled prompt (system message plus conversation history plus new user message) and returns an error if the count exceeds 80% of the target model's context window. This single function prevents context overflow errors from reaching the API, eliminates billing spikes from runaway context accumulation, and gives you a clear signal when conversation history needs pruning. Build the check once and reuse it across every model endpoint you call.

Why token counts vary across AI providers

Token counts vary across providers because every model team builds their own tokenizer on their own training data. The same 200-word paragraph produces 267 tokens on Claude (Anthropic tokenizer), approximately 266 tokens on GPT-5.5 (o200k_base), and a different count again on models using cl100k_base or character-division approximations². These small per-sentence differences compound across long prompts and high-volume workloads.

Always count tokens against the specific model you plan to deploy with, not against a generic estimate or a different model's tokenizer. The ~ symbol in this tool marks models where an exact browser tokenizer is unavailable and an approximation is used instead. For those models, add a 15% buffer to your context window check and cost projection to account for the approximation margin.

Try in the tool

What Is a Context Window?

Context window is the term developers encounter most often when a prompt or conversation thread fails. The term refers to the maximum number of tokens, input and output combined, that a language model can hold in its active working memory during a single inference call. Understanding context windows explains why long conversations run out of room, why RAG pipelines have a retrieval budget, and why some tasks require larger models than others.

What is a context window?

Input versus output within the context window

The context window budget is shared between input and output tokens, which means every token you send reduces the space available for the model's reply. If a model has a 200,000-token context window and you send 180,000 tokens of input, only 20,000 tokens remain for the model's response, which may not be enough for a detailed answer. Conversely, a very short input leaves most of the budget available for a long, detailed output.

Yet most providers charge for output separately and at a higher rate than input, so a short prompt with a long output is not necessarily more cost-efficient than a medium prompt with a medium output. Balancing input and output length depends on both the context budget and the pricing structure of your chosen model.

Reserving response room before counting input

A safe prompt budget subtracts the expected output length first. If you expect a 10,000-token response, treat the input budget as the context window minus that response reserve, not as the full window. On a 200,000-token model, reserving 10,000 tokens for the response means the input ceiling is effectively 190,000 tokens; on a 1M-token model, the same reserve leaves 990,000 tokens for input, which is why the same prompt can fit on a larger-context model but fail on a smaller one even though the prompt text is identical.

How context windows affect different use cases

For conversational applications, each new turn adds tokens to the accumulated context, and the growth is linear and relentless: a 50-turn conversation with average 500 tokens per turn accumulates 25,000 tokens, and a 500-turn conversation reaches 250,000 tokens. With a 200,000-token window, the conversation has room for 400 such turns before overflow, but a more verbose session with 1,000 tokens per turn hits the ceiling in just 200 turns.

Sizing the retrieval budget in RAG pipelines

For RAG pipelines, the number of retrievable document chunks is directly bounded by the context window minus the space reserved for system prompt, user query, and expected response length. For one-shot document analysis, the entire document must fit in the context along with instructions, which is why very long documents require models with large context windows. A practical calculation for a RAG pipeline on Claude Sonnet 4.6 with a 1,000-token system prompt, a 200-token query, and a 3,000-token response reserve leaves 995,800 tokens for retrieved chunks; at 300 tokens per chunk, that budget accommodates over 3,300 chunks, though most implementations retrieve fewer than 20 per query to maintain retrieval precision.

Context window sizes across current models

Context windows have expanded dramatically over recent years, from 4,096 tokens in early GPT models to 1 million or more in current frontier models, a 250x increase that has fundamentally changed what kinds of tasks a single API call can handle. Among the 37 models this tool covers, Claude Haiku 4.5 supports 200K tokens¹, while GPT-5.5 leads the field at 1.1M tokens². A large group of models including Claude Opus 4.8, Claude Sonnet 4.6, Gemini 3.5, DeepSeek V4 Pro, MiniMax M2.7, and Qwen 3.6 all offer 1M or more tokens³, while Kimi K2.6 and Grok 4 sit below 300K⁴. Consequently, tasks that require holding more than 200K tokens of context must use one of the larger-window models.

Choosing a model by context budget

If your assembled prompt exceeds a smaller model's window, move to a larger-window model before trying to force the request through. The decision tree is straightforward: measure your typical assembled prompt length, subtract it from each model's context window, and select the cheapest model where the result leaves at least 20% headroom for the response; this approach avoids both overflow errors and overpaying for unnecessary context capacity.

Recompute the headroom whenever the prompt changes shape, because a context budget set once for a short system prompt stops holding once you add retrieved documents or a long conversation history. Keeping the subtraction live in the request path means the cheapest qualifying model is chosen automatically as the assembled length grows or shrinks.

Context overflow patterns and how to prevent each

Context overflow follows predictable patterns across different application types, and knowing these patterns lets you build defenses before the first failure. For chat applications, conversation history growth is the primary cause: every turn adds tokens, and an application that never prunes history will eventually overflow any model. For RAG pipelines, retrieving too many chunks at once or using chunks sized too large for the prompt budget pushes the assembled prompt past the model's limit. For document analysis tools, loading a full document without checking its token count first is the most common single-request overflow cause.

A context budget is a token ceiling that your code enforces before building each API request. Set the budget at 75% of the model's context window. When the assembled prompt exceeds that ceiling, trim the oldest conversation turns, reduce the number of retrieved chunks, or split the document and process each segment separately. A context budget enforced in code is more reliable than manual monitoring because it catches overflow before it reaches the API, preventing both explicit errors and silent truncation.

Choosing a model when context capacity is the binding constraint

For tasks that require processing large amounts of context in a single call, model selection is determined by context window capacity before any other factor. Among the models in this tool, GPT-5.5 leads at 1.1M tokens, followed by Claude Sonnet 4.6, Claude Opus 4.8, Gemini 3.1 Pro, DeepSeek V4 Pro, MiniMax M2.7, and Qwen 3.5 Plus at 1M tokens. Claude Haiku 4.5 supports 200K, Kimi K2.6 supports 262K, and Grok 4 supports 256K.

The practical difference between 256K and 1M becomes significant for tasks like full-codebase analysis (which may exceed 300K tokens for large projects), legal document review (some contracts exceed 100K tokens), and long-form research synthesis across many retrieved documents. Check your prompt token count in this tool before choosing a model, and confirm the selected model's context window leaves at least 20% headroom for the response at your expected input length.

Try in the tool

What Is a Tokenizer?

Before a language model sees any text, a tokenizer transforms the raw string into a sequence of integer IDs. This preprocessing step is invisible to most users but determines every token count, every context window usage figure, and every billing calculation associated with a model. Different models use different tokenizers, and those differences produce different token counts for identical text, which is why a token counter must specify which model it is counting for.

What is a tokenizer?

How byte-pair encoding builds a vocabulary

Most modern LLM tokenizers use byte-pair encoding (BPE), a subword tokenization algorithm that has become the de facto standard across the industry for good reason. The algorithm starts with individual bytes or characters as the base units, then iteratively merges the most frequently co-occurring pairs into new tokens until the vocabulary reaches a target size commonly ranging from 50,000 to 200,000 entries depending on the model and its training data.

The result is a vocabulary where common English words are single tokens, common word fragments like suffixes and prefixes are standalone tokens, and rare words get split into multiple smaller tokens the model can recombine at inference time. Consequently, text that closely resembles the training corpus tokenizes very efficiently, while text that differs significantly from that corpus (rare technical terms, code with unusual syntax, URLs with random strings) tokenizes much less efficiently and consumes disproportionately more tokens per character.

Seeing BPE merges in ordinary text

A common word like running may stay one token, while a rare technical name can split into smaller pieces that the model has never seen combined in its training data. That is why dense code, identifiers with unconventional naming patterns, and unusual punctuation sequences often consume far more token budget than the same number of plain English words. The BPE algorithm builds its vocabulary by iteratively merging the most frequent character pairs from the training corpus, which means common substrings like ing, tion, and the become single tokens while rare combinations get split into multiple pieces.

Why different models use different tokenizers

Each model team trains their tokenizer on their own pre-training corpus, which means the vocabulary reflects the specific data distribution the model was optimized for during its training phase. A tokenizer trained primarily on English web text will tokenize English prose more efficiently than a tokenizer trained on multilingual or code-heavy data, where the vocabulary must accommodate a much wider range of character patterns.

Furthermore, vocabulary size directly affects encoding efficiency in a way that impacts your per-request costs: a 200,000-token vocabulary like GPT-5.5's o200k_base encodes far more common sequences as single tokens than a 50,000-token vocabulary, reducing average tokens per word and making each API request measurably cheaper at the same per-million-token published rate.

Yet a larger vocabulary also means each token is drawn from a much wider set of options, which increases the model's parameter count, memory footprint, and training cost in ways that the model team must balance against the encoding efficiency gains. These competing trade-offs explain why OpenAI, Anthropic, and Google each built their own tokenizers rather than sharing a single standard, and why the right vocabulary size depends on the target use case.

Why tokenizer choice matters across providers

A tokenizer is effectively part of the model contract and cannot be swapped without changing the model's behavior, so the count from one provider is not a billing guarantee for another. The same 500-word prompt can produce 650 tokens on GPT-5.5 but 700 on an older cl100k vocabulary, and that 50-token difference changes both the per-request cost and whether the prompt fits within a smaller model's context window.

For production applications that switch between providers or A/B test models, counting tokens against the wrong vocabulary leads to cost projections that are off by 5 to 15 percent and context overflow errors that only appear after deployment. Always route your pre-flight count through the tokenizer that matches the model you plan to call, not a generic estimate or a different provider's vocabulary.

Tokenizer types used by each model on this tool

The 37 models this tool covers use four distinct tokenization approaches, and understanding which category a model falls into helps you assess how reliable the browser-side count will be for your specific prompts. GPT-5.5 uses o200k_base as its vocabulary¹, enabling exact counts that match the API billing precisely. Claude Opus 4.8, Sonnet 4.6, and Haiku 4.5 use Anthropic's published tokenizer library², which also enables exact counts with zero approximation error.

Models that rely on character-division approximations

Qwen 3.6 uses a vocabulary derived from cl100k_base with proprietary custom merges³, producing counts that are close but not guaranteed to match the actual API billing. DeepSeek V4 Pro trained its own tokenizer from scratch on a multilingual corpus⁴ and does not use cl100k_base. Gemini 3.5, Kimi K2.6, and MiniMax M2.7 publish tokenizer files (SentencePiece or BPE vocabularies) that can be loaded by third-party browser libraries, though none offer an official first-party browser npm package.

Grok 4 is the only model in this group that has no publicly available tokenizer, so this tool uses a character-length approximation for Grok 4 specifically. The character-division method divides total character count by 3.8 to produce an estimate that is typically within 10 to 15 percent of the actual API token count for English prose, but the error margin grows significantly for non-Latin scripts, dense code, or content heavy with special characters and emoji. For these content types, the approximation can undercount by 20 to 30 percent, which means a prompt that appears to fit comfortably within the context window might actually overflow when the API tokenizes it with the model's real vocabulary.

Using the correct tokenizer when you work with multiple providers

When your application calls multiple AI providers, using the correct tokenizer for each model matters for both context window management and cost projection. A token count from the Anthropic tokenizer is exact for Claude models but inaccurate for GPT-5.5, because the two vocabularies differ in their subword merge decisions. Using the wrong tokenizer in your pre-flight validation may allow a prompt through that the actual model later rejects for exceeding its context window at inference time.

In a Node.js application that calls multiple providers, maintain a map from model ID to the correct tokenizer instance for each one, initializing each encoder once at startup and reusing it across requests. For Claude models, use Anthropic's published tokenizer library. For GPT-5.5 and newer OpenAI models, use get_encoding('o200k_base') from tiktoken. For models without an official browser tokenizer package (Gemini, Grok, Kimi, MiniMax), apply the character-division estimate and add a 20% safety buffer when comparing against the context window limit, since the approximation error is significantly larger for these models than for those with exact browser tokenizers. Route each assembled prompt through its model-specific tokenizer before building the final API request.

How vocabulary size affects token count in practice

Across the vocabulary sizes used by the models in this tool, a larger vocabulary generally produces lower token counts for the same text. GPT-5.5's o200k_base vocabulary has 200,000 entries, encoding more common word sequences as single tokens than a 100,000-entry vocabulary. In practice, a 1,000-word English paragraph tokenizes to roughly 1,330 tokens on o200k_base but closer to 1,380 tokens on cl100k_base⁵, a 4% difference that compounds across very long prompts.

For most workload decisions, vocabulary efficiency is only one factor; per-token price and model quality matter more. Where vocabulary size creates a noticeable difference is in character-heavy content such as code, JSON, and URLs. These content types tokenize less efficiently than prose in all vocabularies, but the gap between smaller and larger vocabularies widens for dense symbol content. For code-heavy prompts, measure actual token counts in this tool rather than estimating from line count or word count.

Try in the tool

Input vs Output Tokens

Every major AI API bills input and output tokens at different rates, and output almost always costs more. For most developers, this distinction only matters when a pipeline starts generating unexpectedly high bills. Understanding the difference between input and output tokens, and how the ratio between them shapes real costs, is foundational to designing cost-efficient AI applications.

What is input vs output tokens?

Why output tokens cost more than input tokens

Processing input tokens is a parallel operation where the model attends across all input positions simultaneously in a single forward pass through the transformer layers. Generating output tokens is an inherently sequential process: the model produces each output token by running a complete forward pass, then appends the newly generated token to the growing context before starting the next pass from scratch.

This sequential generation pattern is significantly more computationally expensive per token than the parallel input processing that happens in a single batch, because each output token requires a complete forward pass through all the transformer layers and the model cannot parallelize this operation across future tokens it has not yet generated. Consequently, most providers charge 3 to 6 times more for output tokens than input tokens¹, reflecting the substantial real hardware cost difference between one-pass parallel computation and token-by-token sequential generation that compounds with every single token the model produces during a response and can push the per-request bill dramatically higher for long-form generation tasks.

Claude Opus 4.8 charges 5x more for output than input ($5 input, $25 output)², while GPT-5.5 charges 6x more ($5 input, $30 output)³, making it the most output-expensive model on this tool and the one where controlling response length matters most for keeping costs predictable across high-volume production workloads that generate long-form content.

Why input cost is visible before the request

You can count input tokens before calling the API because the request text is already assembled, giving you a precise pre-flight number. Output cost requires a sample run or a response-length assumption, so the INPUT $ column is a deterministic preflight check while output cost remains a projection until you have measured real responses. This asymmetry means that input cost is the one component you can control precisely before the request fires, whereas output cost depends on how the model chooses to respond, which varies by task complexity, prompt phrasing, and the model's own generation behavior.

For budgeting purposes, treat input cost as a fixed per-request floor and output cost as a variable range that you estimate from sample runs, then monitor both figures continuously in production as your prompt patterns and traffic mix evolve over time. Logging actual output token counts from the first few hundred production requests lets you refine the projection quickly and detect drift before it becomes a billing surprise.

How the input-to-output ratio affects total cost

For workloads that send long prompts but receive short structured responses such as classification labels, routing decisions, or yes/no answers, input cost dominates the bill because the output is a tiny fraction of the total token count. Conversely, for workloads that send short prompts but receive long generated responses like code files, detailed reports, or translations, output cost dominates the total because the generation is much longer than the prompt that triggered it.

Building on this fundamental asymmetry, a request with 1,000 input tokens and 5,000 output tokens on Claude Sonnet at $3/$15 per million costs only $0.003 in input but $0.075 in output, meaning the output is 25 times more expensive than the input for that specific call. Designing prompts that explicitly elicit concise, structured outputs is often far more cost-effective than trying to reduce input length, because the output side of the ledger typically holds the bulk of the spending when the generation-to-input ratio exceeds 2 to 1.

Controlling output token costs in practice

The single most effective lever for reducing output costs is explicit response format constraints that tell the model exactly what structure to produce and what information to leave out of the result entirely. Instructions like "respond in under 100 words", "return only the answer with no explanation", or "output only valid JSON" reduce output length directly and measurably at every request. Yet quality trade-offs matter: cutting output too short for complex reasoning tasks can degrade the model's ability to work through multi-step problems, so calibrate the constraint to each task type.

For tasks that require long reasoning chains (chain-of-thought), some providers offer a separate "reasoning" token type at a different rate, which lets you budget for the thinking steps independently from the final answer tokens and can significantly change the total cost projection for complex analytical workloads that require multi-step deduction.

For most production applications, measuring average output length from a representative sample run that covers the full range of input types in your workload and multiplying by the output rate gives a reliable estimate of per-request output costs before scaling to full production volume, and this empirical approach consistently outperforms theoretical estimates based on expected response length.

Estimating output costs before scaling

Run a small sample of representative prompts, record average output tokens, and multiply by the model's output rate before committing to a daily volume. A sample of 30 to 50 requests is usually enough to establish a reliable mean and reveal the variance in response lengths across different input types.

Calculate both the average and the 90th percentile of output tokens from your sample, then multiply each by the model's per-million output price to establish a typical cost and a conservative worst-case cost per request. The gap between these two figures tells you how much budget buffer you need: a workload with tight variance between the mean and the 90th percentile is predictable, while a workload with a wide gap requires a larger contingency fund to avoid billing surprises at scale.

Designing prompts to control output token costs

For applications that must generate detailed responses, output length is the primary cost lever. Specifying an explicit length constraint in your prompt reduces output token count directly: "respond in under 200 words" or "return only the JSON object without explanation" reduces the average response from 500 tokens to roughly 150 tokens on Claude Sonnet. At $15/M output, that 350-token reduction saves $0.00525 per request. At 100,000 daily requests, that single instruction change saves $525 per day in output cost alone.

Combining response_format with max_tokens for structured outputs

For structured output tasks, combine the response_format parameter (or an explicit JSON schema instruction) with a max_tokens limit set slightly above your expected response length. The format constraint eliminates prose explanations the model would otherwise add. The token ceiling provides a hard billing cap per request. Together, they reduce average output length more reliably than a word-count instruction alone, while giving you a predictable worst-case cost per API call.

Sampling output length before you commit to a model

Because output length is unknown before the API returns a response, exact total cost requires a sample run. Send 20 to 30 representative prompts through the API, collect the output token count from the usage field in each response, and calculate the average and the 90th percentile. Multiply each by the model's output price to estimate typical and worst-case output cost per request.

For production planning, use the 90th percentile as your planning figure rather than the mean. Tail-heavy output distributions appear in open-ended generation tasks where some responses are much longer than average. A cost model built on the 90th percentile rarely produces billing surprises; one built on the mean frequently does. Add the resulting output cost estimate to your input cost from the token counter to produce a complete per-request cost model before committing to a model selection.

Try in the tool