Can we run it on our own machine?" is now one of the first questions teams ask about an open-weight model. Data that cannot leave the building, a predictable monthly bill, or simply the wish to stop depending on a provider: the reasons vary, but the next question is always the same. Which hardware, and how much of it?Vendor pages and forum threads rarely answer it directly. Yet most of the answer is arithmetic. You need three numbers that are published officially: the model's parameter count, your hardware's memory, and your hardware's memory bandwidth. This article walks through the method step by step, shows where the simple rules break, and applies it to twelve current open models.Step 1: the weightsA model's weights must sit in GPU (or unified) memory. Their size is simply:weights (GB) = parameters (billions) × bytes per parameterThe bytes per parameter depend on the precision you load the model in:16-bit (BF16/FP16): 2 bytes per parameter. Reference quality, as published.8-bit (FP8/INT8): 1 byte per parameter. Common production compromise.4-bit (INT4/FP4): 0.5 bytes per parameter. Fits large models on smaller hardware.So Mistral Small 3.2 (24.0 billion parameters, per its official model card) needs 48 GB of weights in 16-bit, 24 GB in 8-bit and 12 GB in 4-bit. Lower precision costs some quality, and how much depends on the model and the task. Measure it on your own evaluation set before committing.Step 2: the KV cache, the part people forgetWeights are not the only thing in memory. For every token in the context, each attention layer stores a key and a value vector: the KV cache. Its size per token is:KV per token (bytes) = 2 × layers × KV heads × head dimension × bytes per valueTake Mistral Small 3.2. Its official config.json lists 40 layers, 8 key-value heads and a head dimension of 128. In 16-bit, that is 2 × 40 × 8 × 128 × 2 = 163,840 bytes, or 160 KiB per token.At a 32k-token context: about 5.4 GB.At a 128k-token context: about 21.5 GB, almost as much as the 8-bit weights themselves.Architecture matters a lot here. Qwen3.8 27B mixes layer types: according to its config, only 16 of its 64 layers use full attention (with 4 KV heads of dimension 256); the other 48 use linear attention, which keeps a fixed-size state instead of a growing cache. Its full-attention cache is 2 × 16 × 4 × 256 × 2 = 64 KiB per token, about 2.1 GB at 32k tokens, less than half of Mistral Small's.The cache also multiplies with concurrent requests: ten users with 32k-token conversations need ten caches.A common shortcut is to add 20% to the weights for the cache and runtime overhead. It is a reasonable floor for short contexts and a single user. It is badly wrong for long contexts or many concurrent users. In those cases, compute the cache explicitly.Step 3: speed is set by memory bandwidthWhen a model writes its answer, it generates one token at a time, and for each token it must read the weights it uses from memory. At small batch sizes this step is limited by memory bandwidth, not by compute. That gives a simple ceiling:max tokens/s (one stream) ≈ memory bandwidth (GB/s) ÷ bytes read per token (GB)For a dense model, the bytes read per token are the whole weights. For a mixture-of-experts model, only the active parameters are read for each token, which is why these models are fast for their size.Two examples with official figures:Qwen3.8 27B in 8-bit on an NVIDIA RTX PRO 6000 Blackwell (1,792 GB/s, per NVIDIA): 1,792 ÷ 27.8 ≈ 64 tokens/s.gpt-oss-120b in 4-bit (117 billion parameters, 5.1 billion active, per OpenAI's model card) reads about 2.55 GB per token. On an NVIDIA DGX Spark (273 GB/s): ≈ 107 tokens/s. On an Apple Mac Studio M5 Max (614 GB/s): ≈ 241 tokens/s. Its 4-bit weights (58.5 GB) fit in either machine's 128 GB of unified memory.Treat these as ceilings, not forecasts. Real throughput is lower: kernels do not reach peak bandwidth, the KV cache must be read too, and splitting a model across several GPUs adds communication that does not scale linearly. Batching many requests raises total throughput well above the single-stream figure. Use the ceiling to rule hardware out, then benchmark what is left.Step 4: from monthly volume to required throughputTeams usually think in requests per month, not tokens per second. The conversion is direct:required tokens/s = requests per month × output tokens per request ÷ seconds in a monthFor 200,000 requests a month, each producing about 225 words (≈ 300 tokens at 0.75 words per token), that is 60 million tokens over 2,592,000 seconds: about 23 tokens/s on average. Peaks will be higher than the average, and for interactive use you also want each individual answer to stream fast enough to read comfortably. Size for your peak, not your mean.The numbers for 12 open modelsApplying steps 1 and 2 (weights + 20%, short context) to twelve current open-weight models, with parameter counts from each publisher's official model card:Qwen3.5 9B (9.65B total, 9.65B active): 23 GB in 16-bit, 12 GB in 8-bit, 6 GB in 4-bitGemma 4 26B A4B (25.8B total, 4B active): 62 GB in 16-bit, 31 GB in 8-bit, 15 GB in 4-bitQwen3.6 35B-A3B (36B total, 3B active): 86 GB in 16-bit, 43 GB in 8-bit, 22 GB in 4-bitgpt-oss-20b (21B total, 3.6B active): 50 GB in 16-bit, 25 GB in 8-bit, 13 GB in 4-bitMistral Small 3.2 (24B total, 24B active): 58 GB in 16-bit, 29 GB in 8-bit, 14 GB in 4-bitQwen3.8 27B (27.8B total, 27.8B active): 67 GB in 16-bit, 33 GB in 8-bit, 17 GB in 4-bitGemma 4 31B (31.3B total, 31.3B active): 75 GB in 16-bit, 38 GB in 8-bit, 19 GB in 4-bitgpt-oss-120b (117B total, 5.1B active): 281 GB in 16-bit, 140 GB in 8-bit, 70 GB in 4-bitMistral Medium 3.5 (127.7B total, 127.7B active): 306 GB in 16-bit, 153 GB in 8-bit, 77 GB in 4-bitLlama 4 Maverick (401.6B total, 17B active): 964 GB in 16-bit, 482 GB in 8-bit, 241 GB in 4-bitQwen3.5 397B (403.4B total, 17B active): 968 GB in 16-bit, 484 GB in 8-bit, 242 GB in 4-bitNemotron 3 Ultra (560.5B total, 55B active): 1,345 GB in 16-bit, 673 GB in 8-bit, 336 GB in 4-bitAnd the hardware, from the manufacturers' spec sheets:NVIDIA GeForce RTX 5090: 32 GB, 1,792 GB/s memory bandwidthNVIDIA DGX Spark: 128 GB unified, 273 GB/s memory bandwidthApple Mac Studio M5 Max: up to 128 GB unified, 614 GB/s memory bandwidthApple Mac Studio M5 Ultra: up to 512 GB unified, 1.2 TB/s memory bandwidthNVIDIA RTX PRO 6000 Blackwell: 96 GB, 1,792 GB/s memory bandwidthNVIDIA L40S: 48 GB, 864 GB/s memory bandwidthNVIDIA H100 SXM: 80 GB, 3.35 TB/s memory bandwidthNVIDIA H200: 141 GB, 4.8 TB/s memory bandwidthNVIDIA DGX B200 (8 GPUs): 1,440 GB total, 64 TB/s total memory bandwidthWhat the arithmetic tells youA few patterns stand out once the numbers are side by side.Small dense models are now a single-GPU problem. Every model up to about 30 billion parameters fits on one 32 GB card in 8-bit or lower, with single-stream ceilings well above reading speed (Mistral Small 3.2 in 8-bit on an RTX 5090: 1,792 ÷ 24 ≈ 75 tokens/s).Unified-memory machines trade speed for capacity. A DGX Spark holds the same 8-bit Mistral Small 3.2, but at 273 GB/s its ceiling is about 11 tokens/s: fine for batch jobs, slow for chat. The arithmetic makes that visible before you buy.Mixture-of-experts changes the trade-off. gpt-oss-120b needs as much memory as a dense 117B model, but reads only 5.1 billion parameters per token. That is why a 128 GB unified-memory box can serve it at interactive speed in 4-bit, something no dense model of that size could do on the same machine.The largest open models remain server-class. Llama 4 Maverick, Qwen3.5 397B and Nemotron 3 Ultra need hundreds of gigabytes even in 4-bit: a 512 GB unified-memory workstation, or several data-center GPUs.The calculation in a few lines of PythonBYTES = {16: 2, 8: 1, 4: 0.5}def weights_gb(params_b, bits): return params_b * BYTES[bits]def kv_cache_gb(layers, kv_heads, head_dim, context_tokens, users=1, bytes_per_value=2): per_token = 2 * layers * kv_heads * head_dim * bytes_per_value return per_token * context_tokens * users / 1e9def max_tokens_per_s(bandwidth_gbs, active_params_b, bits): return bandwidth_gbs / (active_params_b * BYTES[bits])def required_tokens_per_s(requests_per_month, output_tokens): return requests_per_month * output_tokens / (30 * 24 * 3600)# Mistral Small 3.2, 8-bit, 32k context, one user, on an RTX 5090need = weights_gb(24.0, 8) + kv_cache_gb(40, 8, 128, 32_768)print(f"{need:.1f} GB needed") # 29.4 GB, fits in 32 GBprint(f"{max_tokens_per_s(1792, 24.0, 8):.0f} tok/s ceiling") # 75print(f"{required_tokens_per_s(200_000, 300):.0f} tok/s needed") # 23ConclusionSizing hardware for a local LLM comes down to three checks:Memory: weights (parameters × bytes per parameter) plus the KV cache, computed explicitly when contexts are long or users many.Speed ceiling: memory bandwidth divided by the bytes read per token, using active parameters for mixture-of-experts models.Demand: your monthly volume converted to tokens per second, sized for the peak.All three use figures that publishers and manufacturers print on their official pages. The arithmetic will not replace a benchmark on your own workload, but it tells you which machines are worth benchmarking, and which are ruled out before you spend anything.If you want to run these checks without the spreadsheet, I built a free tool that applies this method to the models and hardware above: Choose your GPU server for local AI.SourcesModel parameter counts and configurations: official model cards and config.json files published by Alibaba (Qwen), Google (Gemma), Mistral AI, OpenAI (gpt-oss), Meta (Llama) and NVIDIA (Nemotron).NVIDIA spec sheets: GeForce RTX 50 series comparison, RTX PRO 6000 Blackwell, L40S, H100, H200, DGX B200, DGX Spark.Apple: Mac Studio technical specifications.