VRAM
The high-bandwidth memory on the graphics card itself; an AI model's weights have to fit into it.
// EXPLANATION
When running a local language model, VRAM is the tightest limit: you need roughly parameter count × bytes per weight (for example 8 billion parameters need about 4-5 GB at 4-bit and about 16 GB at 16-bit). If the overflow spills into system memory, speed drops sharply.
// ALSO KNOWN AS
VRAMvideo memory
// RELATED TERMS
GPU (graphics processing unit)A processor with thousands of small cores that performs the same operation on lots of data in parallel, used for graphics and AI computation.
QuantizationA technique that lowers memory and speed cost by storing model weights with fewer bits (for example 8 or 4 instead of 16).
LLM (large language model)A neural network trained on very large amounts of text that generates text by probabilistically predicting the next piece (token) of a sequence.
DirectStorageThe Microsoft API that moves game assets from an NVMe SSD straight into graphics-card memory with minimal CPU involvement.
