What NPU TOPS Doesn't Tell You: How Memory Bandwidth Bounds Local LLM Performance

You open a laptop spec sheet and see 48 TOPS next to 45 TOPS. Higher must be better, right? That assumption feels natural until you run the same Llama 3 8B model in INT4 quantisation on both devices and watch them produce nearly identical single-digit token counts. The TOPS race in the Copilot+ PC market is real, but it is also largely irrelevant to the task most buyers actually intend: running a local LLM.

Memory bandwidth, not compute throughput, sets the ceiling for decode speed, and that number sits quietly in the RAM spec where most shoppers never look. After reading this guide, you can take any laptop or NPU device, derive its rough local-LLM decode speed from two numbers the OEM already publishes, and use CapyToolkit’s free browser-based token counter to verify whether a chosen model fits the memory envelope before you spend a dime. You will understand why a 32 GB LPDDR5X-8533 laptop often outperforms a 16 GB machine with a marginally higher TOPS rating, and you will have a concrete workflow for avoiding a costly hardware mismatch.

What “40 TOPS” Actually Means on a Copilot+ Spec Sheet

TOPS stands for trillions of operations per second.¹ It is a peak compute throughput figure that measures how many floating-point or integer operations a chip can theoretically perform in one second. Marketing teams love TOPS because it produces a large, impressive number that sounds definitive. The problem is that peak throughput describes the best possible scenario under ideal conditions, not the sustained performance you see during an actual workload. A chip can hit its headline TOPS figure when large matrix operations are running uninterrupted, but that state rarely describes moment-to-moment inference.

Copilot+ PC certification sets a 40 TOPS minimum for the NPU component, 16 GB of RAM, and a 256 GB SSD, which means every qualifying laptop on the market must meet these floors before shipping to consumers.¹ These floors exist to guarantee that the device can handle Windows AI features like Studio Effects, Live Captions, and background image generation at a basic level. They do not guarantee that the same device will run a quantised 7B language model at usable speeds. The three NPU architectures dominating the Copilot+ market in 2026, Qualcomm Snapdragon X Elite and X Plus, Intel Core Ultra series, and AMD Ryzen AI 300 series, each publish TOPS ratings, but those ratings are peak figures that do not directly map to on-device LLM decode throughput. Understanding what TOPS actually measures is the prerequisite for reading the next spec correctly: memory bandwidth.

NPU TOPS and Prefill: What the Metric Actually Signals

Evaluating NPU performance requires splitting LLM inference into two distinct phases: prompt prefill and token decode. During prefill, the model ingests the full prompt and builds the key-value cache, a batch matrix operation that runs quickly on compute-heavy hardware. During decode, the model generates one output token at a time and each step requires streaming the full weight set back through memory. TOPS correlates with the first phase and barely influences the second.

TOPS is a prefill throughput figure

TOPS measures how quickly the chip processes the prompt input, not how fast it generates output. When you send a long prompt to a model, the NPU must read every token, apply attention mechanisms across the full context window, and compute the initial hidden states. That prefill step is compute-heavy and benefits from high TOPS throughput. A faster NPU will process a 100K-token context window more quickly than a slower one, all else being equal.

Step where TOPS helps most

The prefill advantage matters most for workloads that encode long context windows. Some models in 2026 advertise 1M-token context windows, and RAG pipelines routinely feed thousands of retrieved documents into a single prompt. Encoding that volume of text quickly is where peak NPU throughput earns its keep. Long-context summarisation, legal document review, and multi-document analysis all belong to this family of tasks.

Why prefill speed and decode speed are essentially decoupled

Once the model begins generating output tokens, the bottleneck shifts entirely. Each new token requires reading the full set of model weights from memory, updating the key-value cache, and running a comparatively small matrix multiply. The weight read dominates the timing, and memory bandwidth governs that read speed, not how many trillions of operations per second the NPU can theoretically dispatch. Spec sheets rarely distinguish prefill from decode throughput because doing so would expose the gap between the headline number and the workload most buyers actually care about.

Memory Bandwidth and the Decode Bottleneck

Why decode is memory-bandwidth bound

Generating one output token requires reading the full model weights from memory. The NPU performs this read at the maximum rate the memory bus allows. LLM decoding is memory-bandwidth-bound rather than compute-bound; researchers have confirmed this in a lookahead decoding study that demonstrates the bottleneck explicitly.² If your device has 102 GB/s of LPDDR5X bandwidth and your 7B Q4 model occupies 4 GB, the theoretical ceiling is 25.5 tokens per second. No amount of additional compute throughput breaks that ceiling because the weight read is already consuming the full bus bandwidth. Every model running on a shared-memory architecture, from a Snapdragon X Elite to a Jetson Orin Nano, follows this same constraint.

The formula is simple: tokens per second approximates memory bandwidth in GB/s divided by model size in GB.² Running the arithmetic for a 13B Q4 model at roughly 7.5 GB on the same 102 GB/s bus gives approximately 13.6 tokens per second. That halving when model size doubles is not a coincidence; it is the mathematical consequence of bandwidth-bound execution.

Bandwidth versus TOPS impact

NPU Chip	TOPS	DRAM Bandwidth (GB/s)	7B Q4 Estimated tok/s	13B Q4 Estimated tok/s
Snapdragon X Elite (32 GB SKU)	453	136 (LPDDR5X-8533, 128-bit)	34.0	18.1
Intel Core Ultra 7 155H	114	120 (LPDDR5X-7467, 128-bit)	30.0	16.0
AMD Ryzen AI 9 HX 370	505	128 (LPDDR5X-8000, dual-channel)	32.0	17.1

The bandwidth-over-model-size formula produces these theoretical ceilings. Real-world tok/s will sit below these due to KV-cache validation, memory controller contention, and OS scheduling. The point of the table is not to predict exact performance but to show that decode throughput tracks bandwidth closely while TOPS varies by 39 points without lining up with generation speed. The AMD chip posts the highest TOPS but the second-highest bandwidth, so its decode throughput falls between the Intel and Snapdragon entries. The Panther Lake NPU benchmark shows how shared memory bandwidth constrains decode throughput regardless of TOPS rating (Panther Lake NPU vs RTX 3060 for local LLM performance).

Three worked throughput examples

Running a 7B Q4 model across a 64 GB/s bus yields roughly 16 tokens per second, adequate for short conversational prompts with low latency. Doubling the bus speed to 128 GB/s scales that same 7B Q4 model to roughly 32 tokens per second without changing the NPU compute engine. Stepping up to a larger 13B Q4 model on that 128 GB/s bus cuts throughput back down to 17 tokens per second because the larger footprint requires more read cycles per token.

If a laptop reviewer benchmarks local LLM decode speed and reports 9.4 tokens per second on a device rated at 60 TOPS, that result is not a bug in the NPU. It is the correct behaviour for a memory-bandwidth-bound workload. Decode is memory-bandwidth bound, and TOPS is a prefill story.

Shared DRAM means the NPU does not own the bus

The headline LPDDR5X number is a ceiling shared across the NPU, GPU, and CPU. When you run inference, the GPU may simultaneously refresh the display, the CPU may handle OS threads, and the NPU may read model weights. The effective bandwidth available to the model is often lower than the spec sheet maximum. A laptop with a wider memory bus or higher clocked RAM will sustain more decode throughput under real mixed workloads, which is another reason why the GB/s figure matters more than TOPS.

Because shared memory bandwidth imposes a hard ceiling on decode speed, reducing the physical footprint of model weights through quantisation becomes the primary lever for squeezing more tokens out of that bus. Smaller weight files mean fewer read cycles per token, and that is where quantisation level decisions directly translate into hardware performance.

Quantisation, Model Size, and What “8B” Actually Takes in RAM

A 7B parameter model in INT4 quantisation occupies roughly 4 GB. The same model in INT8 occupies about 7.5 GB, and in FP16 it reaches roughly 14 GB. The rule of thumb is parameters multiplied by bytes per parameter, though real files are slightly larger due to metadata, the KV cache structure, and tokenizer weights.⁶ For a 13B model, INT4 yields roughly 7.5 GB, INT8 yields roughly 15 GB, and FP16 yields roughly 26 GB.

Those numbers tell only part of the story. A 16K-token context window on an 8B INT4 model with Grouped-Query Attention consumes roughly 1 GB of KV-cache memory. That KV cache grows linearly with context length and sits in the same RAM pool as the model weights and the OS.⁷ A 128K-context window on the same model inflates that KV-cache footprint to approximately 8 GB. For FP16 inference, those figures double. That memory is unavailable to other processes and slightly reduces the bandwidth margin because some bus cycles go to cache reads instead of weight reads.

INT4 quantisation carries a one to three percent perplexity penalty compared to FP16 on instruction-following and reasoning benchmarks.⁸ For most query tasks, summarisation, and code generation, that penalty is invisible to the end user. INT4 is the default choice for on-device deployment because it cuts memory usage by three-quarters versus FP16 while preserving acceptable quality.

Precision matters when output format correctness is non-negotiable. Code generation, structured output, and math-heavy tasks all benefit from the extra precision INT8 provides. Token-for-token correctness becomes visible when the model is generating valid JSON schemas, complex regex, or multi-step arithmetic where a single quantisation artifact breaks execution. If your primary workload is reasoning or coding, INT8 may justify the 50 percent memory overhead compared to INT4.

The hardware consequence is straightforward. The same laptop that handles a 7B Q4 model comfortably may struggle with 7B Q8 because the larger model file consumes more of the available bandwidth. When you combine an INT8 weight file with a long KV-cache context, the effective bus for weight reads shrinks. A laptop with 32 GB of fast LPDDR5X-8533 RAM therefore wins not just from total capacity but from sustained bandwidth under load.

Using the Token Counter to Size Your AI PC Before You Buy

CapyToolkit’s Prompt Token Counter, which estimates token counts for multiple AI models in your browser, gives you per-model token counts and context window estimates before you commit to hardware. The workflow starts with a representative prompt. Paste your longest expected input into the counter, note the token count, and estimate how long your output will be. A reasonable rule of thumb is two to five times the input length for conversational use cases and up to ten times for generation tasks. Multiply the total tokens by an estimated cost per million tokens to understand the API bill if you run the same model in the cloud, then use that number to gauge whether local hardware makes financial sense.

The token counter exposes context window limits for dozens of models. You can compare how a 1M-context model such as Claude Opus 4.8 stacks up against a shorter-context alternative like Gemini 3.1 Pro. Running a 1M-context model on-device demands more RAM bandwidth than a 128K-context model regardless of TOPS rating, because the KV-cache allocation grows with context length. Before you shop for a laptop, paste your longest planned prompt into the Claude Opus 4.8 token counter or the GPT-5.5 token counter to see whether your use case even needs that much context. For comparison, the Claude Sonnet 4.6 token counter shows how a 1M-window model behaves at a lower price tier.

Rough inference time in seconds equals output tokens multiplied by model size in GB, divided by bandwidth in GB/s. Add prefill overhead of zero to two times the input token count divided by effective TOPS for a complete estimate. The token counter tells you the input and output token counts; the bandwidth formula from the previous section tells you the decode floor. Together they give you a credible wait time before you commit to hardware. If your task demands a 200-token output from a 13B Q4 model on a 102 GB/s bus, you are looking at roughly 15 seconds of decode time plus whatever the prefill step requires. That is slow enough to notice in a conversation, so you either shorten the expected output, move to a faster model, or buy a higher-bandwidth machine.

Not all models exist in INT4, and not all quantised versions are equal. The Gemini 3.1 Pro token counter and the Gemini 3.1 Flash-Lite token counter show how a smaller, lighter model at 1M context trades capability for speed. Flash-Lite generates tokens faster, uses less KV-cache, and runs comfortably on bandwidth-constrained hardware where Pro would choke. Running both counter pages side by side reveals the per-model cost and token difference before you choose.

Run the same prompt against every model in the counter that interests you. Note which models exceed your chosen maximum response-time threshold. Eliminate the ones that do not fit from your shortlist before you shop for hardware. This step prevents the common mistake of buying a high-TOPS laptop that still cannot decode your preferred model at an acceptable speed.

Copilot+ Certifications and What the Fine Print Leaves Out

Copilot+ PC requirements set the floor at 40 TOPS, 16 GB RAM, and a 256 GB SSD. That floor is designed to support Windows AI features, not local LLM inference. A minimum-spec Copilot+ device with 16 GB of LPDDR5X-6400 memory can deliver a worse on-device LLM experience than a laptop with a lower TOPS rating but 32 GB of LPDDR5X-8533 memory. The bandwidth ceiling dominates, and the higher-bandwidth SKU wins even if it posts a marginally lower peak TOPS on paper.

Memory bandwidth sits in the laptop specification sheet under Memory or RAM type. Look for the LPDDR5X-XXXX label, where the XXXX number represents the data rate in megatransfers per second. The bandwidth formula is straightforward: multiply the bus width in bytes by the data rate in MT/s. A 64-bit bus running at LPDDR5X-6400 delivers eight bytes times 6400 MT/s for roughly 51.2 GB/s. A 128-bit bus running at LPDDR5X-8533 delivers sixteen bytes times 8533 MT/s for roughly 136.6 GB/s. Always verify the bus width because two laptops with the same LPDDR5X frequency can have different architectures.

A 7B Q4 model with a reasonable KV-cache fits inside 16 GB of RAM. A 13B Q4 model pushes that boundary and requires more. Because system components share memory bandwidth, adding extra RAM capacity does not improve decode speed if the bus remains narrow. It is the GB/s figure that determines decode throughput, not the total capacity. A 32 GB SKU with a wide, fast bus is categorically better for local LLMs than a 16 GB SKU with a narrower bus, even if both claim Copilot+ compliance.

Store only the model weight files you actually use. Quantised 7B and 13B models occupy four to sixteen gigabytes each, and model loading time is I/O-bound. NVMe Gen4 SSDs load the weights in under a second for files that size. SSD speed does not affect inference throughput once the model is resident in RAM, so do not pay for Gen5 just because a review mentions faster storage.

Hardware Shortlist: What to Actually Buy for Local LLMs Today

With these spec-sheet realities in mind, you can evaluate real-world laptop tiers by memory bandwidth rather than marketing claims.

For budgets between 799 and 999 dollars, entry-level laptops typically ship with 40 TOPS NPUs and 16 GB of LPDDR5X-6400 memory. These devices handle 7B Q4 experiments, short-response tasks, and summarisation workloads. On a 128-bit LPDDR5X-6400 design, a 7B Q4 model reaches roughly 25.6 tokens per second in theory, which is usable for conversational work with noticeable latency on longer generation tasks.

In the 1,299 to 1,799 dollar range, recommended configurations feature Snapdragon X Elite or AMD Ryzen AI 9 processors with 32 GB of LPDDR5X-8533 memory. These devices handle 13B Q4 comfortably, run 7B Q8 with KV-cache headroom, and switch between multiple models without stalling. On the higher-bandwidth bus, a 7B Q4 decode ceiling reaches 34 tokens per second, a jump you feel immediately in conversational latency.

Serious users should look at 2,000 to 3,500 dollar options with discrete GPU backing or NPU-plus-GPU combinations. An RTX 4050 Laptop GPU with 192 GB/s of GDDR6 memory comfortably outperforms even the fastest 136.6 GB/s LPDDR5X system bus for bandwidth-bound workloads.⁹ These machines run 34B and larger models, sustain longer generation windows, and handle batch tasks without hiccups. Benchmark your intended model with the token counter before you buy anything in this tier, because the discrete GPU option makes sense only if your workload actually requires it.

The three-question pre-buy checklist:

1. Name your target model and quantisation level. Look up that model’s file size in gigabytes so you know the weight footprint.
2. Divide the model’s file size in GB by the device DRAM bandwidth in GB/s. That ratio gives the tokens-per-second ceiling for decode on that hardware.
3. Compare the ceiling against your tolerance for response latency. If the figure sits below 10 tokens per second for your target model, move up a tier.

Sources

1. 1.

   Tom’s Hardware, “Copilot+ PCs: All we know about the AI-ready laptops and exclusive Windows features,” tomshardware.com, May 2024. https://www.tomshardware.com/software/windows/microsoft-copilot-pcs-all-we-know

2. 2.

   kipply, “Transformer Inference Arithmetic,” kipp.ly, March 2022. https://kipp.ly/p/transformer-inference-arithmetic

3. 3.

   TechPowerUp, “Qualcomm Snapdragon Elite X SoC for Laptop Leaks: 12 Cores, LPDDR5X Memory, and WiFi7,” techpowerup.com, October 2023. https://www.techpowerup.com/314986/qualcomm-snapdragon-elite-x-soc-for-laptop-leaks-12-cores-lpddr5x-memory-and-wifi7

4. 4.

   TechPowerUp, “Intel Core Ultra 7 155H Specs,” techpowerup.com, accessed July 2026. https://www.techpowerup.com/cpu-specs/core-ultra-7-155h.c3307

5. 5.

   Tom’s Hardware, “AMD unwraps Ryzen AI 300 series Strix Point processors,” tomshardware.com, June 2024. https://www.tomshardware.com/pc-components/cpus/amd-unwraps-ryzen-ai-300-series-strix-point-processors-50-tops-of-ai-performance-zen-5c-density-cores-come-to-ryzen-9-for-the-first-time

6. 6.

   Muhammad Imran Zaman, “Here is how we can calculate the size of any LLM model,” huggingface.co, October 2024. https://huggingface.co/posts/ImranzamanML/127269471333935

7. 7.

   João Marques, “KV cache memory calculator: how much does your LLM actually use?,” dev.to, April 2026. https://dev.to/jagmarques/kv-cache-memory-calculator-how-much-does-your-llm-actually-use-85n

8. 8.

   Elias Frantar et al., “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” ICLR 2023, arXiv:2210.17323, 2022. https://arxiv.org/abs/2210.17323

9. 9.

   Notebookcheck, “NVIDIA GeForce RTX 4050 Laptop GPU - Benchmarks and Specs,” notebookcheck.net, accessed July 2026. https://www.notebookcheck.net/NVIDIA-GeForce-RTX-4050-Laptop-GPU-Benchmarks-and-Specs.675695.0.html