Gemma 4 comes in 5 sizes, all under the Apache 2.0 license. This page leads with the 31B model.
Sizes
| Size | Parameters | Max tokens | Q4_K_M | FP16 | Smallest GPU with 10 percent free |
|---|---|---|---|---|---|
| E2BParameters 5.12B · Max tokens 131,072 · FP16 10.60 | 5.12B | 131,072 | 3.74 | 10.60 | GeForce RTX 3070 8GB |
| E4BParameters 8.00B · Max tokens 131,072 · FP16 16.04 | 8.00B | 131,072 | 5.33 | 16.04 | GeForce RTX 3070 8GB |
| 12BParameters 11.96B · Max tokens 262,144 · FP16 23.65 | 11.96B | 262,144 | 7.64 | 23.65 | GeForce RTX 3080 10GB |
| 26B-A4BParameters 25.81B · Max tokens 262,144 · FP16 49.34 | 25.81B | 262,144 | 14.79 | 49.34 | GeForce RTX 3090 24GB |
| 31BParameters 31.27B · Max tokens 262,144 · FP16 60.34 | 31.27B | 262,144 | 18.48 | 60.34 | GeForce RTX 3090 24GB |
E2B
Which GPUs fit
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+1.40 GB | +1.40 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+13.40 GB | +13.40 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+5.40 GB | +5.40 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+1.40 GB | +1.40 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+5.40 GB | +5.40 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+5.40 GB | +5.40 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+13.40 GB | +13.40 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+5.40 GB | +5.40 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+1.40 GB | +1.40 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+5.40 GB | +5.40 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+5.40 GB | +5.40 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+21.40 GB | +21.40 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+13.40 GB | +13.40 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+5.40 GB | +5.40 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+69.40 GB | +69.40 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+69.40 GB | +69.40 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+25.40 GB | +25.40 GB |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 2.60 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 0.60 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 2.60 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+5.88 GB | +5.88 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+1.88 GB | +1.88 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+3.88 GB | +3.88 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+17.88 GB | +17.88 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+1.88 GB | +1.88 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.88 GB | +9.88 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+5.88 GB | +5.88 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+9.88 GB | +9.88 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+9.88 GB | +9.88 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+17.88 GB | +17.88 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.88 GB | +9.88 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+5.88 GB | +5.88 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.88 GB | +9.88 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+9.88 GB | +9.88 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+25.88 GB | +25.88 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+17.88 GB | +17.88 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+9.88 GB | +9.88 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+73.88 GB | +73.88 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+73.88 GB | +73.88 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+29.88 GB | +29.88 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+7.03 GB | +7.03 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+3.03 GB | +3.03 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+5.03 GB | +5.03 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+19.03 GB | +19.03 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+3.03 GB | +3.03 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.03 GB | +11.03 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+7.03 GB | +7.03 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+11.03 GB | +11.03 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+11.03 GB | +11.03 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+19.03 GB | +19.03 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.03 GB | +11.03 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+7.03 GB | +7.03 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.03 GB | +11.03 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+11.03 GB | +11.03 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+27.03 GB | +27.03 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+19.03 GB | +19.03 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+11.03 GB | +11.03 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+75.03 GB | +75.03 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+75.03 GB | +75.03 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+31.03 GB | +31.03 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+7.67 GB | +7.67 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+3.67 GB | +3.67 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+5.67 GB | +5.67 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+19.67 GB | +19.67 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+3.67 GB | +3.67 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.67 GB | +11.67 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+7.67 GB | +7.67 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+11.67 GB | +11.67 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+11.67 GB | +11.67 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+19.67 GB | +19.67 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.67 GB | +11.67 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+7.67 GB | +7.67 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.67 GB | +11.67 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+11.67 GB | +11.67 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+27.67 GB | +27.67 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+19.67 GB | +19.67 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+11.67 GB | +11.67 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+75.67 GB | +75.67 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+75.67 GB | +75.67 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+31.67 GB | +31.67 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+8.26 GB | +8.26 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+4.26 GB | +4.26 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+6.26 GB | +6.26 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+20.26 GB | +20.26 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+4.26 GB | +4.26 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.26 GB | +12.26 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+8.26 GB | +8.26 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+12.26 GB | +12.26 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+12.26 GB | +12.26 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+20.26 GB | +20.26 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.26 GB | +12.26 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+8.26 GB | +8.26 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.26 GB | +12.26 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+12.26 GB | +12.26 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+28.26 GB | +28.26 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+20.26 GB | +20.26 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+12.26 GB | +12.26 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+76.26 GB | +76.26 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+76.26 GB | +76.26 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+32.26 GB | +32.26 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+8.90 GB | +8.90 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+4.90 GB | +4.90 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+6.90 GB | +6.90 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+20.90 GB | +20.90 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+4.90 GB | +4.90 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.90 GB | +12.90 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+8.90 GB | +8.90 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+12.90 GB | +12.90 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+12.90 GB | +12.90 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+20.90 GB | +20.90 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.90 GB | +12.90 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+8.90 GB | +8.90 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.90 GB | +12.90 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+12.90 GB | +12.90 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+28.90 GB | +28.90 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+20.90 GB | +20.90 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+12.90 GB | +12.90 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+76.90 GB | +76.90 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+76.90 GB | +76.90 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+32.90 GB | +32.90 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+9.38 GB | +9.38 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+5.38 GB | +5.38 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+7.38 GB | +7.38 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+21.38 GB | +21.38 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+5.38 GB | +5.38 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+13.38 GB | +13.38 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+9.38 GB | +9.38 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+13.38 GB | +13.38 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+13.38 GB | +13.38 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+21.38 GB | +21.38 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+13.38 GB | +13.38 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+9.38 GB | +9.38 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+13.38 GB | +13.38 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+13.38 GB | +13.38 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+29.38 GB | +29.38 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+21.38 GB | +21.38 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+13.38 GB | +13.38 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+77.38 GB | +77.38 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+77.38 GB | +77.38 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+33.38 GB | +33.38 GB |
VRAM by quantization
Totals at 8,192 tokens. Q4_K_M is the usual balance of size and quality. FP16 weights are the parameter count times 2 bytes. The FP16 column in the table is the full estimate: weights, cache, and 1 GB.
| Quantization | Bits | Weights | Cache | Overhead | Total |
|---|---|---|---|---|---|
| FP16/BF16Bits 16 · Cache 0.05 · Overhead 1.00 | 16 | 9.54 | 0.05 | 1.00 | 10.60 |
| Q8_0Bits 8.5 · Cache 0.05 · Overhead 1.00 | 8.5 | 5.07 | 0.05 | 1.00 | 6.12 |
| Q6_KBits 6.5625 · Cache 0.05 · Overhead 1.00 | 6.5625 | 3.91 | 0.05 | 1.00 | 4.97 |
| Q5_K_MBits 5.5 · Cache 0.05 · Overhead 1.00 | 5.5 | 3.28 | 0.05 | 1.00 | 4.33 |
| Q4_K_MBits 4.5 · Cache 0.05 · Overhead 1.00 | 4.5 | 2.68 | 0.05 | 1.00 | 3.74 |
| Q3_K_MBits 3.4375 · Cache 0.05 · Overhead 1.00 | 3.4375 | 2.05 | 0.05 | 1.00 | 3.10 |
| Q2_KBits 2.625 · Cache 0.05 · Overhead 1.00 | 2.625 | 1.57 | 0.05 | 1.00 | 2.62 |
Context length
Q4_K_M total as the context changes. Cache is the part that grows.
| Context tokens | Cache GB | Q4_K_M total | FP16 total |
|---|---|---|---|
| 2048FP16 total 10.56 | 0.02 | 3.70 | 10.56 |
| 8192FP16 total 10.60 | 0.05 | 3.74 | 10.60 |
| 32768FP16 total 10.74 | 0.19 | 3.88 | 10.74 |
| 131072FP16 total 11.30 | 0.76 | 4.44 | 11.30 |
E4B
Which GPUs fit
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+7.96 GB | +7.96 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+7.96 GB | +7.96 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+15.96 GB | +15.96 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+7.96 GB | +7.96 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+63.96 GB | +63.96 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+63.96 GB | +63.96 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+19.96 GB | +19.96 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 4.04 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 8.04 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 6.04 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 8.04 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 0.04 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 4.04 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 0.04 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 0.04 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 0.04 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 4.04 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 0.04 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 0.04 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 0.04 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+2.94 GB | +2.94 GB |
| GeForce RTX 3080 10GB10 GB · Tight fit | 10 GB | Tight fit+0.94 GB | +0.94 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+14.94 GB | +14.94 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.94 GB | +6.94 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+2.94 GB | +2.94 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+6.94 GB | +6.94 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+6.94 GB | +6.94 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+14.94 GB | +14.94 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.94 GB | +6.94 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+2.94 GB | +2.94 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.94 GB | +6.94 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+6.94 GB | +6.94 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+22.94 GB | +22.94 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+14.94 GB | +14.94 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+6.94 GB | +6.94 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+70.94 GB | +70.94 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+70.94 GB | +70.94 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+26.94 GB | +26.94 GB |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 1.06 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 1.06 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+4.75 GB | +4.75 GB |
| GeForce RTX 3070 8GB8 GB · Tight fit | 8 GB | Tight fit+0.75 GB | +0.75 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+2.75 GB | +2.75 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+16.75 GB | +16.75 GB |
| GeForce RTX 4060 8GB8 GB · Tight fit | 8 GB | Tight fit+0.75 GB | +0.75 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+8.75 GB | +8.75 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+4.75 GB | +4.75 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+8.75 GB | +8.75 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+8.75 GB | +8.75 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+16.75 GB | +16.75 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+8.75 GB | +8.75 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+4.75 GB | +4.75 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+8.75 GB | +8.75 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+8.75 GB | +8.75 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+24.75 GB | +24.75 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+16.75 GB | +16.75 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+8.75 GB | +8.75 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+72.75 GB | +72.75 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+72.75 GB | +72.75 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+28.75 GB | +28.75 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+5.74 GB | +5.74 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+1.74 GB | +1.74 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+3.74 GB | +3.74 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+17.74 GB | +17.74 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+1.74 GB | +1.74 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.74 GB | +9.74 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+5.74 GB | +5.74 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+9.74 GB | +9.74 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+9.74 GB | +9.74 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+17.74 GB | +17.74 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.74 GB | +9.74 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+5.74 GB | +5.74 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.74 GB | +9.74 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+9.74 GB | +9.74 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+25.74 GB | +25.74 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+17.74 GB | +17.74 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+9.74 GB | +9.74 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+73.74 GB | +73.74 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+73.74 GB | +73.74 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+29.74 GB | +29.74 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+6.67 GB | +6.67 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+2.67 GB | +2.67 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+4.67 GB | +4.67 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+18.67 GB | +18.67 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+2.67 GB | +2.67 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+10.67 GB | +10.67 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+6.67 GB | +6.67 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+10.67 GB | +10.67 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+10.67 GB | +10.67 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+18.67 GB | +18.67 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+10.67 GB | +10.67 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+6.67 GB | +6.67 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+10.67 GB | +10.67 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+10.67 GB | +10.67 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+26.67 GB | +26.67 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+18.67 GB | +18.67 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+10.67 GB | +10.67 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+74.67 GB | +74.67 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+74.67 GB | +74.67 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+30.67 GB | +30.67 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+7.66 GB | +7.66 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+3.66 GB | +3.66 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+5.66 GB | +5.66 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+19.66 GB | +19.66 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+3.66 GB | +3.66 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.66 GB | +11.66 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+7.66 GB | +7.66 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+11.66 GB | +11.66 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+11.66 GB | +11.66 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+19.66 GB | +19.66 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.66 GB | +11.66 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+7.66 GB | +7.66 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+11.66 GB | +11.66 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+11.66 GB | +11.66 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+27.66 GB | +27.66 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+19.66 GB | +19.66 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+11.66 GB | +11.66 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+75.66 GB | +75.66 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+75.66 GB | +75.66 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+31.66 GB | +31.66 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+8.41 GB | +8.41 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+4.41 GB | +4.41 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+6.41 GB | +6.41 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+20.41 GB | +20.41 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+4.41 GB | +4.41 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.41 GB | +12.41 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+8.41 GB | +8.41 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+12.41 GB | +12.41 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+12.41 GB | +12.41 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+20.41 GB | +20.41 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.41 GB | +12.41 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+8.41 GB | +8.41 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+12.41 GB | +12.41 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+12.41 GB | +12.41 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+28.41 GB | +28.41 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+20.41 GB | +20.41 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+12.41 GB | +12.41 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+76.41 GB | +76.41 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+76.41 GB | +76.41 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+32.41 GB | +32.41 GB |
VRAM by quantization
Totals at 8,192 tokens. Q4_K_M is the usual balance of size and quality. FP16 weights are the parameter count times 2 bytes. The FP16 column in the table is the full estimate: weights, cache, and 1 GB.
| Quantization | Bits | Weights | Cache | Overhead | Total |
|---|---|---|---|---|---|
| FP16/BF16Bits 16 · Cache 0.14 · Overhead 1.00 | 16 | 14.89 | 0.14 | 1.00 | 16.04 |
| Q8_0Bits 8.5 · Cache 0.14 · Overhead 1.00 | 8.5 | 7.91 | 0.14 | 1.00 | 9.06 |
| Q6_KBits 6.5625 · Cache 0.14 · Overhead 1.00 | 6.5625 | 6.11 | 0.14 | 1.00 | 7.25 |
| Q5_K_MBits 5.5 · Cache 0.14 · Overhead 1.00 | 5.5 | 5.12 | 0.14 | 1.00 | 6.26 |
| Q4_K_MBits 4.5 · Cache 0.14 · Overhead 1.00 | 4.5 | 4.19 | 0.14 | 1.00 | 5.33 |
| Q3_K_MBits 3.4375 · Cache 0.14 · Overhead 1.00 | 3.4375 | 3.20 | 0.14 | 1.00 | 4.34 |
| Q2_KBits 2.625 · Cache 0.14 · Overhead 1.00 | 2.625 | 2.44 | 0.14 | 1.00 | 3.59 |
Context length
Q4_K_M total as the context changes. Cache is the part that grows.
| Context tokens | Cache GB | Q4_K_M total | FP16 total |
|---|---|---|---|
| 2048FP16 total 15.94 | 0.05 | 5.24 | 15.94 |
| 8192FP16 total 16.04 | 0.14 | 5.33 | 16.04 |
| 32768FP16 total 16.41 | 0.52 | 5.71 | 16.41 |
| 131072FP16 total 17.91 | 2.02 | 7.21 | 17.91 |
12B
Which GPUs fit
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · Tight fit | 24 GB | Tight fit+0.35 GB | +0.35 GB |
| GeForce RTX 4090 24GB24 GB · Tight fit | 24 GB | Tight fit+0.35 GB | +0.35 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+8.35 GB | +8.35 GB |
| Radeon RX 7900 XTX 24GB24 GB · Tight fit | 24 GB | Tight fit+0.35 GB | +0.35 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+56.35 GB | +56.35 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+56.35 GB | +56.35 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+12.35 GB | +12.35 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 11.65 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 15.65 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 13.65 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 15.65 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 7.65 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 11.65 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 7.65 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 7.65 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 7.65 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 11.65 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 7.65 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 7.65 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 7.65 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+10.79 GB | +10.79 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+2.79 GB | +2.79 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+2.79 GB | +2.79 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+2.79 GB | +2.79 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+10.79 GB | +10.79 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+2.79 GB | +2.79 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+2.79 GB | +2.79 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+2.79 GB | +2.79 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+18.79 GB | +18.79 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+10.79 GB | +10.79 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+2.79 GB | +2.79 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+66.79 GB | +66.79 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+66.79 GB | +66.79 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+22.79 GB | +22.79 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 1.21 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 5.21 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 3.21 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 5.21 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 1.21 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 1.21 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+1.49 GB | +1.49 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+13.49 GB | +13.49 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+5.49 GB | +5.49 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+1.49 GB | +1.49 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+5.49 GB | +5.49 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+5.49 GB | +5.49 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+13.49 GB | +13.49 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+5.49 GB | +5.49 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+1.49 GB | +1.49 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+5.49 GB | +5.49 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+5.49 GB | +5.49 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+21.49 GB | +21.49 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+13.49 GB | +13.49 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+5.49 GB | +5.49 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+69.49 GB | +69.49 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+69.49 GB | +69.49 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+25.49 GB | +25.49 GB |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 2.51 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 0.51 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 2.51 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+2.97 GB | +2.97 GB |
| GeForce RTX 3080 10GB10 GB · Tight fit | 10 GB | Tight fit+0.97 GB | +0.97 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+14.97 GB | +14.97 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+2.97 GB | +2.97 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+14.97 GB | +14.97 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+2.97 GB | +2.97 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+22.97 GB | +22.97 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+14.97 GB | +14.97 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+6.97 GB | +6.97 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+70.97 GB | +70.97 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+70.97 GB | +70.97 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+26.97 GB | +26.97 GB |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 1.03 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 1.03 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+4.36 GB | +4.36 GB |
| GeForce RTX 3070 8GB8 GB · Tight fit | 8 GB | Tight fit+0.36 GB | +0.36 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+2.36 GB | +2.36 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+16.36 GB | +16.36 GB |
| GeForce RTX 4060 8GB8 GB · Tight fit | 8 GB | Tight fit+0.36 GB | +0.36 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+8.36 GB | +8.36 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+4.36 GB | +4.36 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+8.36 GB | +8.36 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+8.36 GB | +8.36 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+16.36 GB | +16.36 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+8.36 GB | +8.36 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+4.36 GB | +4.36 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+8.36 GB | +8.36 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+8.36 GB | +8.36 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+24.36 GB | +24.36 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+16.36 GB | +16.36 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+8.36 GB | +8.36 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+72.36 GB | +72.36 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+72.36 GB | +72.36 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+28.36 GB | +28.36 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+5.84 GB | +5.84 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+1.84 GB | +1.84 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+3.84 GB | +3.84 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+17.84 GB | +17.84 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+1.84 GB | +1.84 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.84 GB | +9.84 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+5.84 GB | +5.84 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+9.84 GB | +9.84 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+9.84 GB | +9.84 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+17.84 GB | +17.84 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.84 GB | +9.84 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+5.84 GB | +5.84 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+9.84 GB | +9.84 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+9.84 GB | +9.84 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+25.84 GB | +25.84 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+17.84 GB | +17.84 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+9.84 GB | +9.84 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+73.84 GB | +73.84 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+73.84 GB | +73.84 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+29.84 GB | +29.84 GB |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 3070 8GB8 GB · No CPU offload | 8 GB | Fits+2.97 GB | +2.97 GB |
| GeForce RTX 3080 10GB10 GB · No CPU offload | 10 GB | Fits+4.97 GB | +4.97 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+18.97 GB | +18.97 GB |
| GeForce RTX 4060 8GB8 GB · No CPU offload | 8 GB | Fits+2.97 GB | +2.97 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+10.97 GB | +10.97 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+10.97 GB | +10.97 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+10.97 GB | +10.97 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+18.97 GB | +18.97 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+10.97 GB | +10.97 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+6.97 GB | +6.97 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+10.97 GB | +10.97 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+10.97 GB | +10.97 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+26.97 GB | +26.97 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+18.97 GB | +18.97 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+10.97 GB | +10.97 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+74.97 GB | +74.97 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+74.97 GB | +74.97 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+30.97 GB | +30.97 GB |
VRAM by quantization
Totals at 8,192 tokens. Q4_K_M is the usual balance of size and quality. FP16 weights are the parameter count times 2 bytes. The FP16 column in the table is the full estimate: weights, cache, and 1 GB.
| Quantization | Bits | Weights | Cache | Overhead | Total |
|---|---|---|---|---|---|
| FP16/BF16Bits 16 · Cache 0.38 · Overhead 1.00 | 16 | 22.28 | 0.38 | 1.00 | 23.65 |
| Q8_0Bits 8.5 · Cache 0.38 · Overhead 1.00 | 8.5 | 11.83 | 0.38 | 1.00 | 13.21 |
| Q6_KBits 6.5625 · Cache 0.38 · Overhead 1.00 | 6.5625 | 9.14 | 0.38 | 1.00 | 10.51 |
| Q5_K_MBits 5.5 · Cache 0.38 · Overhead 1.00 | 5.5 | 7.66 | 0.38 | 1.00 | 9.03 |
| Q4_K_MBits 4.5 · Cache 0.38 · Overhead 1.00 | 4.5 | 6.27 | 0.38 | 1.00 | 7.64 |
| Q3_K_MBits 3.4375 · Cache 0.38 · Overhead 1.00 | 3.4375 | 4.79 | 0.38 | 1.00 | 6.16 |
| Q2_KBits 2.625 · Cache 0.38 · Overhead 1.00 | 2.625 | 3.65 | 0.38 | 1.00 | 5.03 |
Context length
Q4_K_M total as the context changes. Cache is the part that grows.
| Context tokens | Cache GB | Q4_K_M total | FP16 total |
|---|---|---|---|
| 2048FP16 total 23.60 | 0.33 | 7.59 | 23.60 |
| 8192FP16 total 23.65 | 0.38 | 7.64 | 23.65 |
| 32768FP16 total 23.84 | 0.56 | 7.83 | 23.84 |
| 131072FP16 total 24.59 | 1.31 | 8.58 | 24.59 |
| 262144FP16 total 25.59 | 2.31 | 9.58 | 25.59 |
26B-A4B
Which GPUs fit
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+30.66 GB | +30.66 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+30.66 GB | +30.66 GB |
| GeForce RTX 3060 12GB12 GB · Won't fit | 12 GB | 37.34 GB short | Won't fit |
| GeForce RTX 3070 8GB8 GB · Won't fit | 8 GB | 41.34 GB short | Won't fit |
| GeForce RTX 3080 10GB10 GB · Won't fit | 10 GB | 39.34 GB short | Won't fit |
| GeForce RTX 3090 24GB24 GB · CPU offload | 24 GB | 25.34 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · Won't fit | 8 GB | 41.34 GB short | Won't fit |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 33.34 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · Won't fit | 12 GB | 37.34 GB short | Won't fit |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 33.34 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 33.34 GB short | CPU offload |
| GeForce RTX 4090 24GB24 GB · CPU offload | 24 GB | 25.34 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 33.34 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · Won't fit | 12 GB | 37.34 GB short | Won't fit |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 33.34 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 33.34 GB short | CPU offload |
| GeForce RTX 5090 32GB32 GB · CPU offload | 32 GB | 17.34 GB short | CPU offload |
| Radeon RX 7900 XTX 24GB24 GB · CPU offload | 24 GB | 25.34 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 33.34 GB short | CPU offload |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · CPU offload | 48 GB | 13.34 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+5.19 GB | +5.19 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+53.19 GB | +53.19 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+53.19 GB | +53.19 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+9.19 GB | +9.19 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 14.81 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 18.81 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 16.81 GB short | CPU offload |
| GeForce RTX 3090 24GB24 GB · CPU offload | 24 GB | 2.81 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 18.81 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 10.81 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 14.81 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 10.81 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 10.81 GB short | CPU offload |
| GeForce RTX 4090 24GB24 GB · CPU offload | 24 GB | 2.81 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 10.81 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 14.81 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 10.81 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 10.81 GB short | CPU offload |
| Radeon RX 7900 XTX 24GB24 GB · CPU offload | 24 GB | 2.81 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 10.81 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+3.01 GB | +3.01 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+3.01 GB | +3.01 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+11.01 GB | +11.01 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+3.01 GB | +3.01 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+59.01 GB | +59.01 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+59.01 GB | +59.01 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+15.01 GB | +15.01 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 8.99 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 12.99 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 10.99 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 12.99 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 4.99 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 8.99 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 4.99 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 4.99 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 4.99 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 8.99 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 4.99 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 4.99 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 4.99 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+6.20 GB | +6.20 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+6.20 GB | +6.20 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+14.20 GB | +14.20 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+6.20 GB | +6.20 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+62.20 GB | +62.20 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+62.20 GB | +62.20 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+18.20 GB | +18.20 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 5.80 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 9.80 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 7.80 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 9.80 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 1.80 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 5.80 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 1.80 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 1.80 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 1.80 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 5.80 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 1.80 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 1.80 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 1.80 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+9.21 GB | +9.21 GB |
| GeForce RTX 4060 Ti 16GB16 GB · Tight fit | 16 GB | Tight fit+1.21 GB | +1.21 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · Tight fit | 16 GB | Tight fit+1.21 GB | +1.21 GB |
| GeForce RTX 4080 16GB16 GB · Tight fit | 16 GB | Tight fit+1.21 GB | +1.21 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+9.21 GB | +9.21 GB |
| GeForce RTX 5060 Ti 16GB16 GB · Tight fit | 16 GB | Tight fit+1.21 GB | +1.21 GB |
| GeForce RTX 5070 Ti 16GB16 GB · Tight fit | 16 GB | Tight fit+1.21 GB | +1.21 GB |
| GeForce RTX 5080 16GB16 GB · Tight fit | 16 GB | Tight fit+1.21 GB | +1.21 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+17.21 GB | +17.21 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+9.21 GB | +9.21 GB |
| Radeon RX 9070 XT 16GB16 GB · Tight fit | 16 GB | Tight fit+1.21 GB | +1.21 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+65.21 GB | +65.21 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+65.21 GB | +65.21 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+21.21 GB | +21.21 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 2.79 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 6.79 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 4.79 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 6.79 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 2.79 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 2.79 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · Tight fit | 12 GB | Tight fit+0.40 GB | +0.40 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+12.40 GB | +12.40 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+4.40 GB | +4.40 GB |
| GeForce RTX 4070 12GB12 GB · Tight fit | 12 GB | Tight fit+0.40 GB | +0.40 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+4.40 GB | +4.40 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+4.40 GB | +4.40 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+12.40 GB | +12.40 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+4.40 GB | +4.40 GB |
| GeForce RTX 5070 12GB12 GB · Tight fit | 12 GB | Tight fit+0.40 GB | +0.40 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+4.40 GB | +4.40 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+4.40 GB | +4.40 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+20.40 GB | +20.40 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+12.40 GB | +12.40 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+4.40 GB | +4.40 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+68.40 GB | +68.40 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+68.40 GB | +68.40 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+24.40 GB | +24.40 GB |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 3.60 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 1.60 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 3.60 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · No CPU offload | 12 GB | Fits+2.84 GB | +2.84 GB |
| GeForce RTX 3080 10GB10 GB · Tight fit | 10 GB | Tight fit+0.84 GB | +0.84 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+14.84 GB | +14.84 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.84 GB | +6.84 GB |
| GeForce RTX 4070 12GB12 GB · No CPU offload | 12 GB | Fits+2.84 GB | +2.84 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+6.84 GB | +6.84 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+6.84 GB | +6.84 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+14.84 GB | +14.84 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.84 GB | +6.84 GB |
| GeForce RTX 5070 12GB12 GB · No CPU offload | 12 GB | Fits+2.84 GB | +2.84 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+6.84 GB | +6.84 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+6.84 GB | +6.84 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+22.84 GB | +22.84 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+14.84 GB | +14.84 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+6.84 GB | +6.84 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+70.84 GB | +70.84 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+70.84 GB | +70.84 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+26.84 GB | +26.84 GB |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 1.16 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 1.16 GB short | CPU offload |
VRAM by quantization
Totals at 8,192 tokens. Q4_K_M is the usual balance of size and quality. FP16 weights are the parameter count times 2 bytes. The FP16 column in the table is the full estimate: weights, cache, and 1 GB.
| Quantization | Bits | Weights | Cache | Overhead | Total |
|---|---|---|---|---|---|
| FP16/BF16Bits 16 · Cache 0.27 · Overhead 1.00 | 16 | 48.07 | 0.27 | 1.00 | 49.34 |
| Q8_0Bits 8.5 · Cache 0.27 · Overhead 1.00 | 8.5 | 25.54 | 0.27 | 1.00 | 26.81 |
| Q6_KBits 6.5625 · Cache 0.27 · Overhead 1.00 | 6.5625 | 19.72 | 0.27 | 1.00 | 20.99 |
| Q5_K_MBits 5.5 · Cache 0.27 · Overhead 1.00 | 5.5 | 16.52 | 0.27 | 1.00 | 17.80 |
| Q4_K_MBits 4.5 · Cache 0.27 · Overhead 1.00 | 4.5 | 13.52 | 0.27 | 1.00 | 14.79 |
| Q3_K_MBits 3.4375 · Cache 0.27 · Overhead 1.00 | 3.4375 | 10.33 | 0.27 | 1.00 | 11.60 |
| Q2_KBits 2.625 · Cache 0.27 · Overhead 1.00 | 2.625 | 7.89 | 0.27 | 1.00 | 9.16 |
Context length
Q4_K_M total as the context changes. Cache is the part that grows.
| Context tokens | Cache GB | Q4_K_M total | FP16 total |
|---|---|---|---|
| 2048FP16 total 49.28 | 0.21 | 14.73 | 49.28 |
| 8192FP16 total 49.34 | 0.27 | 14.79 | 49.34 |
| 32768FP16 total 49.58 | 0.51 | 15.03 | 49.58 |
| 131072FP16 total 50.51 | 1.45 | 15.96 | 50.51 |
| 262144FP16 total 51.76 | 2.70 | 17.21 | 51.76 |
31B
Which GPUs fit
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+19.66 GB | +19.66 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+19.66 GB | +19.66 GB |
| GeForce RTX 3060 12GB12 GB · Won't fit | 12 GB | 48.34 GB short | Won't fit |
| GeForce RTX 3070 8GB8 GB · Won't fit | 8 GB | 52.34 GB short | Won't fit |
| GeForce RTX 3080 10GB10 GB · Won't fit | 10 GB | 50.34 GB short | Won't fit |
| GeForce RTX 3090 24GB24 GB · CPU offload | 24 GB | 36.34 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · Won't fit | 8 GB | 52.34 GB short | Won't fit |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 44.34 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · Won't fit | 12 GB | 48.34 GB short | Won't fit |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 44.34 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 44.34 GB short | CPU offload |
| GeForce RTX 4090 24GB24 GB · CPU offload | 24 GB | 36.34 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 44.34 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · Won't fit | 12 GB | 48.34 GB short | Won't fit |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 44.34 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 44.34 GB short | CPU offload |
| GeForce RTX 5090 32GB32 GB · CPU offload | 32 GB | 28.34 GB short | CPU offload |
| Radeon RX 7900 XTX 24GB24 GB · CPU offload | 24 GB | 36.34 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 44.34 GB short | CPU offload |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · CPU offload | 48 GB | 24.34 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+46.96 GB | +46.96 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+46.96 GB | +46.96 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · Tight fit | 48 GB | Tight fit+2.96 GB | +2.96 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 21.04 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 25.04 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 23.04 GB short | CPU offload |
| GeForce RTX 3090 24GB24 GB · CPU offload | 24 GB | 9.04 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 25.04 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 17.04 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 21.04 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 17.04 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 17.04 GB short | CPU offload |
| GeForce RTX 4090 24GB24 GB · CPU offload | 24 GB | 9.04 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 17.04 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 21.04 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 17.04 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 17.04 GB short | CPU offload |
| GeForce RTX 5090 32GB32 GB · CPU offload | 32 GB | 1.04 GB short | CPU offload |
| Radeon RX 7900 XTX 24GB24 GB · CPU offload | 24 GB | 9.04 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 17.04 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+6.01 GB | +6.01 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+54.01 GB | +54.01 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+54.01 GB | +54.01 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+10.01 GB | +10.01 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 13.99 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 17.99 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 15.99 GB short | CPU offload |
| GeForce RTX 3090 24GB24 GB · CPU offload | 24 GB | 1.99 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 17.99 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 9.99 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 13.99 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 9.99 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 9.99 GB short | CPU offload |
| GeForce RTX 4090 24GB24 GB · CPU offload | 24 GB | 1.99 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 9.99 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 13.99 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 9.99 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 9.99 GB short | CPU offload |
| Radeon RX 7900 XTX 24GB24 GB · CPU offload | 24 GB | 1.99 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 9.99 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · Tight fit | 24 GB | Tight fit+1.88 GB | +1.88 GB |
| GeForce RTX 4090 24GB24 GB · Tight fit | 24 GB | Tight fit+1.88 GB | +1.88 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+9.88 GB | +9.88 GB |
| Radeon RX 7900 XTX 24GB24 GB · Tight fit | 24 GB | Tight fit+1.88 GB | +1.88 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+57.88 GB | +57.88 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+57.88 GB | +57.88 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+13.88 GB | +13.88 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 10.12 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 14.12 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 12.12 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 14.12 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 6.12 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 10.12 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 6.12 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 6.12 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 6.12 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 10.12 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 6.12 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 6.12 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 6.12 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+5.52 GB | +5.52 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+5.52 GB | +5.52 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+13.52 GB | +13.52 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+5.52 GB | +5.52 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+61.52 GB | +61.52 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+61.52 GB | +61.52 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+17.52 GB | +17.52 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 6.48 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 10.48 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 8.48 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 10.48 GB short | CPU offload |
| GeForce RTX 4060 Ti 16GB16 GB · CPU offload | 16 GB | 2.48 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 6.48 GB short | CPU offload |
| GeForce RTX 4070 Ti Super 16GB16 GB · CPU offload | 16 GB | 2.48 GB short | CPU offload |
| GeForce RTX 4080 16GB16 GB · CPU offload | 16 GB | 2.48 GB short | CPU offload |
| GeForce RTX 5060 Ti 16GB16 GB · CPU offload | 16 GB | 2.48 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 6.48 GB short | CPU offload |
| GeForce RTX 5070 Ti 16GB16 GB · CPU offload | 16 GB | 2.48 GB short | CPU offload |
| GeForce RTX 5080 16GB16 GB · CPU offload | 16 GB | 2.48 GB short | CPU offload |
| Radeon RX 9070 XT 16GB16 GB · CPU offload | 16 GB | 2.48 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+9.39 GB | +9.39 GB |
| GeForce RTX 4060 Ti 16GB16 GB · Tight fit | 16 GB | Tight fit+1.39 GB | +1.39 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · Tight fit | 16 GB | Tight fit+1.39 GB | +1.39 GB |
| GeForce RTX 4080 16GB16 GB · Tight fit | 16 GB | Tight fit+1.39 GB | +1.39 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+9.39 GB | +9.39 GB |
| GeForce RTX 5060 Ti 16GB16 GB · Tight fit | 16 GB | Tight fit+1.39 GB | +1.39 GB |
| GeForce RTX 5070 Ti 16GB16 GB · Tight fit | 16 GB | Tight fit+1.39 GB | +1.39 GB |
| GeForce RTX 5080 16GB16 GB · Tight fit | 16 GB | Tight fit+1.39 GB | +1.39 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+17.39 GB | +17.39 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+9.39 GB | +9.39 GB |
| Radeon RX 9070 XT 16GB16 GB · Tight fit | 16 GB | Tight fit+1.39 GB | +1.39 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+65.39 GB | +65.39 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+65.39 GB | +65.39 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+21.39 GB | +21.39 GB |
| GeForce RTX 3060 12GB12 GB · CPU offload | 12 GB | 2.61 GB short | CPU offload |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 6.61 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 4.61 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 6.61 GB short | CPU offload |
| GeForce RTX 4070 12GB12 GB · CPU offload | 12 GB | 2.61 GB short | CPU offload |
| GeForce RTX 5070 12GB12 GB · CPU offload | 12 GB | 2.61 GB short | CPU offload |
| GPU | Memory | Status | Headroom |
|---|---|---|---|
| GeForce RTX 3060 12GB12 GB · Tight fit | 12 GB | Tight fit+0.35 GB | +0.35 GB |
| GeForce RTX 3090 24GB24 GB · No CPU offload | 24 GB | Fits+12.35 GB | +12.35 GB |
| GeForce RTX 4060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+4.35 GB | +4.35 GB |
| GeForce RTX 4070 12GB12 GB · Tight fit | 12 GB | Tight fit+0.35 GB | +0.35 GB |
| GeForce RTX 4070 Ti Super 16GB16 GB · No CPU offload | 16 GB | Fits+4.35 GB | +4.35 GB |
| GeForce RTX 4080 16GB16 GB · No CPU offload | 16 GB | Fits+4.35 GB | +4.35 GB |
| GeForce RTX 4090 24GB24 GB · No CPU offload | 24 GB | Fits+12.35 GB | +12.35 GB |
| GeForce RTX 5060 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+4.35 GB | +4.35 GB |
| GeForce RTX 5070 12GB12 GB · Tight fit | 12 GB | Tight fit+0.35 GB | +0.35 GB |
| GeForce RTX 5070 Ti 16GB16 GB · No CPU offload | 16 GB | Fits+4.35 GB | +4.35 GB |
| GeForce RTX 5080 16GB16 GB · No CPU offload | 16 GB | Fits+4.35 GB | +4.35 GB |
| GeForce RTX 5090 32GB32 GB · No CPU offload | 32 GB | Fits+20.35 GB | +20.35 GB |
| Radeon RX 7900 XTX 24GB24 GB · No CPU offload | 24 GB | Fits+12.35 GB | +12.35 GB |
| Radeon RX 9070 XT 16GB16 GB · No CPU offload | 16 GB | Fits+4.35 GB | +4.35 GB |
| NVIDIA A100 80GB80 GB · No CPU offload | 80 GB | Fits+68.35 GB | +68.35 GB |
| NVIDIA H100 SXM 80GB80 GB · No CPU offload | 80 GB | Fits+68.35 GB | +68.35 GB |
| Apple M5 Max 48GB unified48 GB · 36.00 GB usable · No CPU offload | 48 GB | Fits+24.35 GB | +24.35 GB |
| GeForce RTX 3070 8GB8 GB · CPU offload | 8 GB | 3.65 GB short | CPU offload |
| GeForce RTX 3080 10GB10 GB · CPU offload | 10 GB | 1.65 GB short | CPU offload |
| GeForce RTX 4060 8GB8 GB · CPU offload | 8 GB | 3.65 GB short | CPU offload |
VRAM by quantization
Totals at 8,192 tokens. Q4_K_M is the usual balance of size and quality. FP16 weights are the parameter count times 2 bytes. The FP16 column in the table is the full estimate: weights, cache, and 1 GB.
| Quantization | Bits | Weights | Cache | Overhead | Total |
|---|---|---|---|---|---|
| FP16/BF16Bits 16 · Cache 1.09 · Overhead 1.00 | 16 | 58.25 | 1.09 | 1.00 | 60.34 |
| Q8_0Bits 8.5 · Cache 1.09 · Overhead 1.00 | 8.5 | 30.95 | 1.09 | 1.00 | 33.04 |
| Q6_KBits 6.5625 · Cache 1.09 · Overhead 1.00 | 6.5625 | 23.89 | 1.09 | 1.00 | 25.99 |
| Q5_K_MBits 5.5 · Cache 1.09 · Overhead 1.00 | 5.5 | 20.02 | 1.09 | 1.00 | 22.12 |
| Q4_K_MBits 4.5 · Cache 1.09 · Overhead 1.00 | 4.5 | 16.38 | 1.09 | 1.00 | 18.48 |
| Q3_K_MBits 3.4375 · Cache 1.09 · Overhead 1.00 | 3.4375 | 12.51 | 1.09 | 1.00 | 14.61 |
| Q2_KBits 2.625 · Cache 1.09 · Overhead 1.00 | 2.625 | 9.56 | 1.09 | 1.00 | 11.65 |
Context length
Q4_K_M total as the context changes. Cache is the part that grows.
| Context tokens | Cache GB | Q4_K_M total | FP16 total |
|---|---|---|---|
| 2048FP16 total 60.11 | 0.86 | 18.24 | 60.11 |
| 8192FP16 total 60.34 | 1.09 | 18.48 | 60.34 |
| 32768FP16 total 61.28 | 2.03 | 19.41 | 61.28 |
| 131072FP16 total 65.03 | 5.78 | 23.16 | 65.03 |
| 262144FP16 total 70.03 | 10.78 | 28.16 | 70.03 |
Gemma 4 hardware notes
- Unified memory counts 75 percent of the listed pool as usable. Discrete cards use the full listed memory. The memory column still shows the listed figure.
- Last updated 2026-07-20.
- Weights, cache, and the 1 GB overhead are estimates. How this is calculated.
- Architecture numbers and the parameter count come from the public model record, checked 2026-10-10.
Gemma 4 VRAM FAQ
How much VRAM does Gemma 4 31B need at Q4_K_M?
18.48 GB at 8,192 tokens. Weights are 16.38 GB, the cache is 1.09 GB, and overhead is 1 GB.
Does a 24 GB GPU fit Gemma 4 31B at Q4_K_M?
GeForce RTX 3090 24GB is Fits, 5.52 GB left. The quant table uses 8,192 tokens, the smaller of 8,192 and the configured maximum of 262,144.
What context length is configured for Gemma 4 31B?
The configured context length is 262,144 tokens. The quant table uses 8,192 tokens, the smaller of 8,192 and the configured maximum of 262,144.
Which source supplies the parameter count for Gemma 4 31B?
31.27B parameters, from the Hugging Face model record for google/gemma-4-31B-it.
Related models
| Model | Q4_K_M GB |
|---|---|
| gpt-oss-20b | 12.15 |
| Llama 3 8B | 6.21 |
| Llama 3.1 8B | 6.21 |
| Llama 3.1 70B | 40.46 |