Model weights occupy the largest fixed block
The downloaded checkpoint contains the learned parameters plus supporting files such as tokenizer data and configuration. These weights must be copied into VRAM before the first forward pass. On a consumer GPU the transfer occurs over PCIe, so the initial load time is noticeable for models above a few billion parameters. Once resident, the weights remain in VRAM for the life of the session unless the runtime explicitly offloads layers. Keeping the full set on the GPU delivers the highest token rate because no further transfers occur during generation.
- Weights are static after loading and do not grow with context length.
- Partial layer offload moves selected transformer blocks to RAM while keeping others on the GPU.
- Integrated GPUs share a single memory pool, so the same allocation serves both weights and system tasks.
KV cache is the variable working memory
During generation each new token attends to all previous tokens. To avoid recomputing attention over the entire history, the model stores key and value vectors in the KV cache. The cache size scales directly with sequence length, number of layers, hidden dimension, and batch size. For a 7-billion-parameter model at 8k context the cache can easily reach several gigabytes. Longer contexts or larger batches increase the requirement further. Because the cache is created and updated on every decode step, it must sit in the fastest memory available. When it no longer fits in VRAM the runtime can move older layers to system RAM, but each transfer adds latency.
- Dynamic cache grows on demand and works with most generation settings.
- Static cache pre-allocates a fixed maximum size and enables torch.compile but wastes space on short sequences.
- Quantized cache lowers precision of the stored vectors to reduce footprint at a small accuracy cost.
Runtime buffers and headroom complete the picture
Inference runtimes allocate additional memory for activations, output logits, and temporary compute buffers. These allocations vary with model architecture, attention implementation, and batch settings. Flash Attention and similar kernels can reduce peak usage, yet the exact savings depend on the backend. After accounting for weights and KV cache, a practical allowance of 10-20 percent extra VRAM prevents crashes when these buffers expand. Systems that also drive a display or run other GPU workloads need the higher end of that range.
Offloading trades speed for capacity
When total demand exceeds VRAM, the runtime can keep some layers or the KV cache in system RAM. Only the active layer stays on the GPU during each forward pass. The next layer is prefetched asynchronously while the current layer computes. The approach works for both dynamic and static caches and is exposed through cache_implementation settings or explicit offloading flags. The cost is extra PCIe traffic that lowers tokens per second, especially on long contexts where the KV cache moves frequently. Offloading therefore serves as a deliberate compromise rather than a free upgrade.
- KV offload keeps weights on the GPU while moving only the cache to RAM.
- Layer offload spreads the model itself across both memory pools.
- Systems with unified memory still require budgeting because the single pool serves OS and application needs simultaneously.
Cache selection in practice
Hugging Face Transformers exposes several cache classes that balance memory and speed. The default DynamicCache grows as needed and supports offloading. StaticCache pre-allocates a fixed size and enables compilation but can waste attention slots on short sequences. QuantizedCache reduces precision to shrink the cache footprint. Offloading is available for both dynamic and static variants through dedicated implementation strings or constructor flags. Users who generate many sequences of similar length benefit most from static allocation, while variable-length workloads usually prefer the dynamic option with selective offloading.
- Enable offloading only after confirming that full GPU placement produces out-of-memory errors.
- Test the exact context length and batch size you plan to use before committing to a cache type.
- Monitor actual allocation in runtime logs rather than relying on theoretical formulas alone.
Putting the pieces together
A workable local AI setup begins with the model weights, adds the KV cache required by the longest context you expect, then includes runtime buffers and headroom. If the total exceeds available VRAM, decide whether layer offload, KV offload, or a smaller model is the better path. Offloading extends reach on memory-constrained hardware but introduces measurable slowdowns that vary with PCIe bandwidth and context length. Measuring the actual allocations on your hardware remains the only reliable way to set expectations before deployment.