Prompt Token Counter: Conversions

Convert Words to Tokens

How to convert Words to Tokens

Estimating tokens from a word count provides a fast proxy for API cost calculations when you know the word length of your content but have not yet counted tokens directly. Multiply your word count by 1.33 to get an approximate token count for standard English prose. This estimate assumes clean paragraph text and may undercount for content with dense code, URLs, or non-Latin characters.

Common Words to Tokens conversions

Why the word-to-token ratio differs by model

The 1.33 tokens-per-word ratio applies to models with large vocabularies (100K tokens or more) such as GPT-5.5 (o200k_base) and the Claude family.¹ Older models using smaller vocabularies, like cl100k_base at 100K tokens, tokenize uncommon words less efficiently, which can push the ratio toward 1.5 tokens per word for text with technical jargon or rare vocabulary.²

Furthermore, non-English languages vary considerably: Chinese and Japanese text uses fewer tokens per character than English because many characters map to single tokens in multilingual vocabularies, while languages like Turkish or Finnish with complex morphology may tokenize at rates above 2 tokens per word.³ Calibrate the 1.33 baseline against real token counts when your prompts mix languages or use non-Latin scripts.

Calibrating the ratio for your own documents

Count a representative sample from your corpus and divide tokens by words to get your observed ratio. Use that ratio for early planning, then switch to the model-specific counter when the prompt is close to a cost or context limit. Sampling at least 20 to 30 documents from each content category in your dataset ensures that the observed ratio reflects the actual diversity of your corpus rather than the accidental simplicity of a handful of short or unusually clean documents. Once you have a calibrated ratio per content type, you can project token counts for new documents by multiplying their word count by the matching ratio, which gives you a reliable planning figure before you ever paste the text into a tokenizer.

How code versus prose tokenizes differently

Plain English prose at 1.33 tokens per word is the most efficient content type for most AI tokenizers, since common words map cleanly to single tokens and the BPE vocabulary was optimized for exactly this kind of natural language input. Code tokenizes significantly less efficiently for two structural reasons: first, variable names and function identifiers often split at camelCase or underscore boundaries into fragments the model treats as separate units; second, operators, brackets, and punctuation marks are typically individual tokens that each consume their own slot in the sequence, inflating the count well beyond what a word-length estimate would suggest.⁴

Calculating realistic token counts for code-heavy content

A 100-word Python function with type annotations and docstrings may produce 200 to 250 tokens rather than the 133 that an equivalent-length prose paragraph would generate. Markdown with inline code, headers with special characters, and text containing many URLs falls between prose and raw code in token density. For mixed documents, plan for 1.5 to 1.8 tokens per word rather than 1.33 to budget conservatively. The key variable is symbol density: a prose paragraph with one or two inline code references tokenizes at close to the 1.33 baseline, while a paragraph that is mostly code with a few connecting words can push toward 2.5 tokens per word or higher, which means a 200-word section of mixed content might produce 400 to 500 tokens instead of the 266 that the prose-only estimate predicts.

Practical budgeting with word-to-token estimates

For system prompt planning, multiply your draft system prompt word count by 1.33 to get a token estimate, then multiply by your expected request volume and the model's per-million price to project the daily cost of that single text field, which often turns out to be one of the largest line items in a high-volume pipeline. For content pipelines, measure the word count of a representative sample from each content category in your dataset, apply the appropriate ratio for that category rather than the generic 1.33, and project costs at scale across your expected request volume.

Add a 20% buffer when the content includes any non-prose elements such as code, URLs, tables, or special characters, since these push the token-per-word ratio above the 1.33 baseline and can surprise you at billing time. When precision matters, verify estimates before committing to a production pipeline design: paste actual content samples into the token counter to get exact or near-exact counts against your target model, then use those calibrated counts for your final cost projection rather than relying on the theoretical ratio alone.

When the estimate is reliable versus when to count exactly

When your content is standard English prose without embedded code, URLs, or non-Latin scripts, the 1.33 tokens-per-word estimate lands within 10% of the actual API count. That precision is sufficient for most planning decisions: sizing system prompts, estimating pipeline costs, and checking whether a document fits a context window before pasting it.

For billing-critical projections, paste actual content into the token counter to confirm the exact count against your target model. The difference between a 10% underestimate and the true count is small per request but compounds at scale: a 10% gap on 10 million input tokens per day represents 1 million additional tokens, worth $3 at Claude Sonnet pricing or $1 at Claude Haiku pricing. Run the exact count for any workload that consistently exceeds 1 million tokens per day.

Applying the word count ratio to system prompt optimization

Applying the 1.33 ratio to your system prompt word count gives you a fast read on the per-request cost floor before any user content is included. A 750-word system prompt carries roughly 998 tokens on every single call. At Claude Sonnet pricing of $3 per million input tokens, that prompt costs $0.003 per request.⁵ At 500,000 daily requests, the system prompt alone costs $1,500 per day from a single text field.

System prompts grow silently as teams add instructions without measuring their token impact. Add a word count check to your prompt review process: at every significant edit, count words, multiply by 1.33, and compare to the previous estimate. Growth beyond 20% since the last review is a trigger to audit which additions are essential versus redundant. Compressing a 1,000-token system prompt to 700 tokens saves 300 tokens per request; at 1 million daily requests on Sonnet pricing, that is $900 per day from one editing pass.

Token density across common content formats

Token density varies predictably across content formats, and knowing the typical range for your type helps calibrate the 1.33 estimate before applying it to a large dataset. Standard English prose lands at 1.0–1.2 tokens per word. In contrast, technical documentation with inline code averages 1.3–1.6 tokens per word; source code with type annotations and operators averages 1.5–2.0 tokens per word for Python and TypeScript, while JSON with short keys and nested structures often tokenizes above 2.0 tokens per word.⁴

For datasets with mixed content formats, sample a few representative items from each category, count them in the token counter, and calculate the observed tokens-per-word ratio for each type. Weighted by the proportion of each format in your dataset, these ratios produce a more accurate multiplier than the generic 1.33 for your specific content mix.

Applying token density by content format

Use the observed ratio for each format instead of one generic multiplier when your dataset mixes prose, code, JSON, and documentation. For a pipeline that processes 60 percent prose, 25 percent code, and 15 percent JSON, the weighted ratio would be the sum of each format's observed ratio multiplied by its proportion, producing a single blended multiplier that reflects your actual content mix rather than the generic 1.33 baseline. Recalibrate these ratios whenever you onboard a new document type or when the composition of your dataset shifts significantly, because a multiplier that was accurate for a prose-heavy corpus will undercount tokens after you add a large code repository to the mix.

Store the per-format ratios alongside the dataset definition so a future run reuses the calibrated multiplier instead of falling back to the generic 1.33, because an accurate baseline is only useful if it survives beyond the run that produced it. Recalibrating on a small sample whenever the content mix shifts keeps the estimate honest without re-measuring the entire corpus each time.

Try in the tool

Convert Characters to Tokens

How to convert Characters to Tokens

A character count is often the fastest signal you have before tokenizing. Divide your character count by 3.8 and round up to get an approximate token count. This estimate assumes standard English prose or mixed text and may undercount for dense symbol content.

Common Characters to Tokens conversions

Why the character-to-token ratio differs by model

For standard English text, the 3.8 characters-per-token ratio is a general approximation derived from average token density across large modern vocabularies, though the exact value shifts meaningfully depending on vocabulary size, training data composition, and the specific tokenizer implementation.¹ Shorter words like "the" and "is" average well under 3 characters per token, while longer technical terms may average 6 or more characters per token. Furthermore, the ratio varies significantly by content type: dense JSON with short keys and many brackets averages closer to 2-3 characters per token, code with operators and punctuation averages 3-4, and clean prose averages 4-5 characters per token.²

Non-Latin scripts often have fewer characters per token in multilingual models; Japanese kanji or Chinese characters may each map to a single token despite being individual Unicode characters in the raw string.³ Consequently, the character-division estimate tends to overcount significantly for languages like Chinese and Japanese and undercount for symbol-heavy code, which is why calibration against real API token counts is essential when your content mixes writing systems or relies heavily on non-Latin characters.

Choosing the right divisor for raw strings

Use 3.8 for ordinary prose, 2.5 to 3 for code-heavy strings, and 4 to 5 for clean long-form writing with generous margins. When a string is close to a context limit, always verify it with the model-specific counter instead of relying on the divisor alone, since the character-division estimate cannot account for model-specific vocabulary merges that may tokenize your particular content more or less efficiently than the generic ratio predicts.

How code versus prose tokenizes differently

For plain English prose, the 3.8 characters-per-token estimate is reasonably accurate and will serve you well for estimating API costs on natural language content. Yet code consistently produces far more tokens per character because the BPE vocabulary was not optimized for programming language syntax. A Python function body with type annotations, docstrings, and operators might average only 2.5 characters per token, meaning a 380-character code snippet produces roughly 152 tokens rather than the 100 the formula predicts.⁴

Markdown formatting adds less overhead than raw source code, since headers, bold markers, and links introduce special characters that produce additional tokens but at a significantly lower density than the operators and brackets found in programming languages. For production applications with mixed content that combines prose, code blocks, and structured markup, count actual tokens on representative samples rather than relying on the formula alone, because the blended token density can vary widely depending on the proportion of each content type and the specific programming languages involved.

Checking code before accepting the character estimate

For code-heavy inputs, always run the token counter before assuming the character estimate fits your context budget, since the formula will systematically undercount for any non-trivial code. Code consistently defies the 3.8 divisor because operators, brackets, and variable names fragment into many small tokens, which means a 1,000-character code snippet can produce 350 to 400 tokens instead of the 263 that the formula predicts. Paste a representative code sample into the counter and compare the actual token count against the character-divide estimate to measure the divergence for your specific codebase before setting context window limits or cost projections.

Practical budgeting with character-to-token estimates

Character counts are trivially easy to measure programmatically before content is processed, since every programming language exposes string length as a built-in property that requires no external library or dependencies. Applying the divide-by-3.8 formula allows a fast pre-filter that rejects inputs longer than the context window limit multiplied by 3.8 characters before you even run the tokenizer, catching obviously oversized inputs at negligible computational cost and saving both API calls and billing surprises downstream.

This pre-filter catches the majority of overflow risks cheaply, but it is not a substitute for exact token counting when the content falls near the limit where the approximation margin matters. For billing projections, apply the formula to the average character length of each content category in your dataset, project the token count per category, then multiply by the request volume and per-token price for each. Add 25% as a planning buffer when any of the content includes code or symbol-heavy fields, since these types consistently produce more tokens per character than the 3.8 baseline and the buffer absorbs that variance.

A two-stage validation pipeline using character count

Character count works as a cheap first gate before running the tokenizer, since most programming languages expose string length as a basic property that requires no external library or API call. The check is straightforward: if the character count divided by 3.8 exceeds the context window limit, reject or trim the input before tokenizing, catching obviously oversized inputs at negligible computational cost while avoiding unnecessary tokenizer overhead for the majority of requests that are clearly within bounds.

Implementing the two-stage gate

A two-stage validation pipeline uses character count as gate one and the exact tokenizer as gate two. Gate one rejects immediately if the character count divided by 3.8 exceeds the limit multiplied by 1.25, since the actual token count almost certainly overflows. Gate two runs the exact tokenizer only if the estimate falls within 75 to 125 percent of the limit, confirming the precise count before deciding whether to proceed. Inputs clearly below 75 percent pass without tokenizer overhead, which means the expensive tokenizer only runs on the small fraction of inputs near the boundary where the character estimate is unreliable, reducing both latency and infrastructure cost for high-volume pipelines.

Keep gate one deliberately conservative so the expensive tokenizer only runs on the narrow band where its precision matters, because spending tokenizer cycles on inputs that are obviously fine or obviously too large wastes the infrastructure you were trying to save. Logging the reject and pass rates at each gate shows whether the 75 to 125 percent band is correctly sized for your real traffic.

Calibrating the divisor for your specific content type

For content types where the 3.8 divisor is consistently inaccurate, calibrate it against real token counts from your corpus. Sample 100 items from each content category, measure the total character count and the total token count, then divide: total characters divided by total tokens gives the empirical characters-per-token ratio for that type. A Python codebase might yield 2.5 characters per token; a legal brief in English might yield 4.5 characters per token.

With a calibrated divisor per content type, your pre-filter accuracy improves significantly. For a pipeline processing code documents, using 2.5 as the divisor instead of 3.8 prevents the pre-filter from accepting code-heavy documents that will later fail the tokenizer gate. Record your calibration factors in your pipeline configuration and re-calibrate quarterly as your content mix evolves.

When the character-to-token estimate fails most often

When your content contains a high proportion of emoji, special Unicode characters, or right-to-left scripts, the character-division formula breaks down most severely. A single emoji is one character in most string length calculations, but it tokenizes as 1 to 3 tokens depending on the model's Unicode block handling. A document that is 20% emoji by character count may have actual token counts 40 to 60 percent above the formula's prediction.⁵

For these content types, treat the characters-to-tokens estimate as a rough order-of-magnitude check rather than a reliable budget figure. Identify content categories in your dataset where divergence is known to be high, and always run the exact tokenizer before sending those inputs. The token counter on this page handles Unicode correctly for models with published tokenizers (GPT-5.5 and Claude); for those models, use the exact count rather than the character estimate on Unicode-heavy content.

Try in the tool

Convert Pages to Tokens

How to convert Pages to Tokens

Page counts are useful for an early document-size check before extracting or pasting text. Multiply your page count by 665 (500 words per page times 1.33 tokens per word) to get an approximate token count for a standard document. This estimate assumes a typical English business or academic document of around 500 words per page.

Common Pages to Tokens conversions

Why the page-to-token ratio differs by model

A standard page estimate starts with 500 words per page, a reasonable average for business documents, academic papers, and reports with normal margins and 11-12pt font that most people encounter in day-to-day work.¹ Multiply by 1.33 tokens per word and a one-page document produces roughly 665 tokens, which serves as a useful baseline for initial capacity planning across the models this tool covers.² Yet presentation slides may have only 50 words per page (67 tokens) while dense legal briefs may pack 700 words per page (931 tokens), so the actual range is far wider than the baseline suggests.

Furthermore, scanned documents converted via OCR may introduce character noise, misrecognized glyphs, and formatting artifacts that increase the token count above what clean digital text would produce, adding invisible overhead that the page-based estimate cannot anticipate.³ Use 665 tokens per page as a rough starting point and then calibrate against actual documents from your own corpus, measuring the real tokens-per-page ratio for each document type you plan to process through your pipeline.

Calibrating page estimates for your document type

Count a few representative pages from your own corpus, divide the token count by the page count, and use that empirical ratio for early pipeline planning rather than the generic 665 baseline. The calibration matters most for tables, code-heavy pages, slide decks, and OCR output because those formats deviate sharply from the 500-word prose assumption. For each document type in your pipeline, measure at least five to ten pages spread across different sections of the source material, because a single page from the introduction of a report may be mostly prose while a page from the appendix could be entirely code or dense tabular data that tokenizes at two or three times the rate.

How code versus prose tokenizes differently

A page of prose and a page of code have dramatically different token densities, which is why the 665-token baseline is so unreliable for technical content that relies on anything other than plain English paragraphs. A page of clean Python code might contain 100 to 150 lines averaging 15 words each (1,500 to 2,250 words per page), but code tokenizes at 1.5 to 2 tokens per word rather than 1.33, compounding the already higher word count into a token count that can exceed 4,000 for a single page.

Estimating tokens for slide decks and mixed documents

A code-dense page may therefore produce 2,000 to 4,000 tokens, three to six times more than the 665-token estimate for an equivalent page of prose. Mixed documents such as technical specifications with code examples fall somewhere in between, and for these you should add 50 to 100 percent to the page-based estimate to avoid context overflow.⁴ A practical approach is to classify each document in your intake pipeline by its dominant format, apply the matching multiplier, and then verify the estimate against an extracted text sample before committing the full document to a model request, since the variance between document types is too large to ignore.⁵

Practical budgeting with page-to-token estimates

For document intake pipelines, page count is often the most accessible metric available before extracting text, since PDF metadata and document management systems routinely expose it without requiring any content parsing. A 50-page policy document is approximately 33,250 tokens at the standard 665 tokens per page, which fits comfortably in any of the 1M-context models and leaves ample room for system instructions and the model's response.

Knowing the approximate token count derived from the page count lets you make a fast feasibility decision before loading the full document into memory or sending it to the API. For documents that exceed the context window at their estimated token count, plan for chunking, summarization, or multi-pass analysis before any processing begins, rather than discovering the overflow at runtime when the API rejects the request.

What page-based estimates miss in document pipelines

Estimating from page count gives you a feasibility check, not a guarantee. A 100-page PDF estimated at 66,500 tokens may contain 80,000 tokens if it uses dense type, heavy tables, or complex formatting. Conversely, a 100-page slide deck with mostly images and sparse text may contain fewer than 5,000 tokens. Page count is the weakest proxy for token count because page layout varies so dramatically across document types.

For PDFs you plan to process through an AI pipeline, extract the text first using a library like pdf-parse (Node.js) or pdfminer.six (Python), then count the extracted text tokens in this tool. Running text extraction on even a small document sample pays back immediately: you learn whether the 665-token-per-page estimate is accurate for your corpus before designing your context budget around an assumption that may be wrong.

Planning multi-pass analysis when documents exceed the context window

When a document's estimated token count exceeds the target model's context window, you have three standard approaches: hierarchical summarization, chunked querying, and selective retrieval. Hierarchical summarization compresses the full document in sections, then summarizes the summaries, producing a condensed version that fits for final analysis. Chunked querying splits the document into overlapping sections and runs the same query against each chunk separately, then aggregates results.

Choosing the right multi-pass strategy

Selective retrieval (RAG) is the most cost-efficient approach when your query needs only a fraction of the document's content. Rather than processing the full document, you retrieve the top N most relevant chunks and process only those. For tasks that require whole-document reasoning, such as cross-referencing clauses in a contract, chunked querying or hierarchical summarization is necessary. Use the page-to-token estimate early in planning to identify which approach your corpus requires before building the pipeline.⁶ The choice between RAG and hierarchical summarization has significant implications for both cost and output quality, so identifying the right approach during the planning phase prevents expensive rearchitecture after the pipeline is already in production.

Validate the choice against a representative sample of your real queries before committing engineering time, because the right multi-pass strategy for a contract review differs sharply from the right one for a short FAQ lookup. Running the page-to-token estimate on that sample during planning surfaces the dominant query pattern and points to the cheapest approach that still meets quality.

Applying the pages-to-tokens ratio to context budget planning

For document intake workflows, the page-to-token ratio is most useful when combined with context budget math. Take your target model's context window, subtract the tokens reserved for system instructions and the model's response (typically 3,000 to 8,000 tokens), and divide by 665. The result is the maximum number of standard prose pages you can include per request without overflow.

For a Claude Sonnet workload with a 1M-token context window, reserving 8,000 tokens for overhead leaves 992,000 tokens for documents: enough for approximately 1,492 standard pages. For a Claude Haiku workload with a 200K context window, the same overhead leaves 192,000 tokens: approximately 289 pages.⁷ Knowing this ceiling before building your pipeline tells you whether chunking logic is required for your typical document size.

Try in the tool