Inside LLM Compute: Matrices, GPU & VRAM
Follow a token from its embedding through memory transfers, tensor-core math, attention, KV cache, and output generation. Includes interactive matrix and memory simulations. · interactive original ↗
01 · Foundations: what is a matrix?
A scalar is one number; a vector is a one-dimensional array; a matrix is a two-dimensional array; and a tensor generalizes these to arbitrary dimensions. LLMs represent token embeddings and activations as tensors.
Scalar → Vector → Matrix → Tensor
[3, 7, 2]
[[1, 2, 3], [4, 5, 6]]
Tensor shape: [batch, tokens, hidden]
A token embedding is a vector. Multiple token vectors form a matrix; batches of sequences form higher-rank tensors.
Matrix multiplication dimensions
B: (n × p)
C = A × B: (m × p)
The inner dimensions must match. Each output element is a dot product between a row of A and a column of B.
Core idea: text becomes token IDs, token IDs become numerical vectors, and learned weight matrices transform those vectors repeatedly.
02 · Interactive matrix multiplication
Choose an output cell in C. The highlighted row in A and column in B are multiplied element-by-element and summed. Then explore how a GPU moves and computes this data.
Pick a row of A (r1, r2) and a column of B (c1–c3), or a cell of C, to see that dot product. Type over a number in A or B to change it — C recomputes.
Multiply-and-accumulate operations form the output. Tensor Cores can process many fused matrix operations in parallel.
Why tiling and reuse matter
A naive implementation reloads the same values repeatedly. Tiled GEMM loads blocks of A and B into fast on-chip storage so many multiply-accumulate operations reuse them. Pipelines can overlap loading, computing, and writing.
Important correction: GPU computation and VRAM access are not strictly separate, alternating steps. Modern GPU kernels overlap data movement and arithmetic.
03 · GPU hardware and memory hierarchy
VRAM stores the bulk data; GPU compute units execute arithmetic. Between them are caches, shared memory, and registers. NVIDIA GPUs organize compute units into streaming multiprocessors (SMs).
| Component | Role | LLM example |
|---|---|---|
| VRAM / HBM | Bulk storage | Weights, KV cache, activations |
| L2 Cache | Shared on-chip caching | Reuse hot data; reduce HBM traffic |
| Shared Memory / L1 | Fast on-chip working set | Tiled GEMM operand reuse |
| Registers | Very fast per-thread storage | Accumulators and intermediate values |
| Tensor Cores | High-throughput matrix arithmetic | GEMM, Attention projections, FFN |
| CUDA Cores | General arithmetic / control | Elementwise math, portions of reductions |
04 · A Transformer does much more than MatMul
A decoder-only Transformer repeatedly applies normalization, Q/K/V projections, positional operations, causal attention, residual additions, and feed-forward networks. Final logits are converted to the next token.
Attention: Q, K and V
Attention(Q,K,V) = softmax(QKᵀ / √dₖ + mask) V
Q is the query representation; K provides matching features; V carries the information being mixed. The QKᵀ product creates attention scores, a causal mask blocks future positions, softmax normalizes scores, and multiplying by V aggregates information. Optimized kernels such as FlashAttention avoid materializing the full attention matrix in HBM.
Feed-forward network (FFN)
An FFN applies learned matrix projections and a nonlinear activation. Many modern LLMs use gated variants such as SwiGLU. The displayed formula is a simplified example with a column-vector convention.
Example: 4096 → 11008 → 4096
Other GPU operations in an LLM
| Operation | Role | Typical pressure |
|---|---|---|
| GEMM / MatMul | Matrix products | Compute or memory |
| RMSNorm / LayerNorm | Normalize activations | Memory / reduction |
| RoPE | Inject position information | Elementwise / memory |
| Softmax | Normalize attention scores | Reduction / memory |
| SiLU / GELU | Nonlinear activations | Compute / memory |
| Attention | Mix contextual information | Compute / KV I/O |
| Residual Add | Combine pathways | Memory bandwidth |
| Sampling | Choose next token | Reduction / sampling |
| Quantize / Dequantize | Compact / restore numeric formats | Memory / compute |
05 · What occupies VRAM?
Inference memory is more than model weights. The working set includes weights, cached keys/values, intermediate activations, runtime buffers, and allocation overhead.
Illustrative memory composition
Weight storage and precision
For a hypothetical 7-billion-parameter model, raw weight size is approximately parameter count × bytes per parameter.
| Precision | Bytes / parameter | 7B model weights |
|---|---|---|
| FP32 | 4 | 28 GB |
| FP16 / BF16 | 2 | 14 GB |
| INT8 | 1 | 7 GB |
| INT4 | 0.5 | 3.5 GB |
KV Cache: why context and concurrency consume memory
At each decoder layer, previously computed keys and values are cached. During decode, attention reads historical K/V and appends new K/V. This simple formula assumes dense, uniform caching across sequences.
06 · Prefill versus Decode
LLM generation has two performance regimes. Prefill processes many prompt tokens together. Decode then produces one token per sequence at a time, while using the growing KV cache.
① Prefill
Many prompt tokens are processed in parallel, often producing large matrix multiplications with good compute utilization. The KV cache is built. Time to first token (TTFT) includes prefill and additional serving overhead.
② Decode
For each active sequence, the model processes its latest token, reads past KV, and appends new KV. At low batch sizes this often behaves like matrix-vector work and can be limited by memory bandwidth.
| Dimension | Prefill | Decode |
|---|---|---|
| Input shape | Many prompt tokens | One new token per sequence |
| Typical GEMM | Large matrices | Smaller matrices / matrix-vector at low batch |
| KV Cache | Build | Read and append |
| Common limit | Compute + attention | Bandwidth / KV I/O at low batch |
| Metric | TTFT | TPOT / tokens per second |
Decode is not always memory-bound. Larger batches, long-context attention, and different model architectures can shift the bottleneck toward compute.
07 · Compute-bound vs memory-bound
Performance is constrained both by arithmetic throughput (FLOP/s) and the rate at which operands move (bytes/s). Arithmetic intensity measures FLOPs per byte transferred. Explore the simplified Roofline model below.
A bandwidth-only lower-bound example
If 14 GB of weights must be read from HBM at an ideal 1 TB/s, the transfer alone takes at least 14 ms. This is not a real token latency prediction: cache reuse, KV traffic, arithmetic, and scheduling also matter.
08 · Training vs Inference
Inference primarily runs the forward pass using fixed weights. Training additionally computes gradients through a backward pass and updates weights with an optimizer.
| Dimension | Inference | Training |
|---|---|---|
| Forward pass | Yes | Yes |
| Backward / gradients | No | Yes |
| Update weights | No | Yes |
| Optimizer states | Normally no | Usually yes |
| Activation storage | Forward working set | Saved or recomputed for backward |
Training also stores gradients, optimizer states (for example Adam moments), and backward-required activations. Precision, sharding, checkpointing, and optimizer choice change the memory footprint substantially.
09 · Key takeaways
| Question | Answer |
|---|---|
| Where are large weight matrices? | Primarily in VRAM/HBM. |
| Who performs matrix multiplication? | Usually GPU Tensor Cores accelerate the heavy GEMM kernels. |
| Does every intermediate go to VRAM? | No. Registers, caches and kernel fusion can avoid writes. |
| Why does long context use more memory? | KV cache generally grows with cached tokens and active sequences. |
| Why can a high-FLOPS GPU still be slow? | Memory bandwidth, utilization, kernel overhead, and KV traffic. |
| Why is training more memory-intensive? | Gradients, optimizer states, and backward activations. |
Think in two coupled systems: (1) the mathematical graph of Transformer operations, and (2) the physical path that moves tensors through GPU memory to execution units. Both determine latency and throughput.
What to learn next
Follow this sequence: VRAM/HBM → L2 → Shared Memory → Registers → Tensor Core MMA → Accumulator → Output; then study FlashAttention, PagedAttention, continuous batching, quantization, and GPU profiling.