GPU · VRAM · TRANSFORMER
Scalar → Vector → Matrix → Tensor
5
[3, 7, 2]
[[1, 2, 3], [4, 5, 6]]
Tensor shape: [batch, tokens, hidden]
[3, 7, 2]
[[1, 2, 3], [4, 5, 6]]
Tensor shape: [batch, tokens, hidden]
A: (m × n)
B: (n × p)
C = A × B: (m × p)
B: (n × p)
C = A × B: (m × p)
A (2×4)
×
B (4×3)
=
C (2×3)
GPU Device
VRAM / HBM
Model weights · Activations · KV cache · Workspace
↕
L2 Cache
Shared on-chip cache
↕
SM 1
Registers
Shared Memory / L1
Tensor Cores
CUDA Cores
SM 2 … N
Registers
Shared Memory / L1
Tensor Cores
CUDA Cores
Text → Token IDs → Embedding
↓
RMSNorm / LayerNorm
↓
Q = XWq · K = XWk · V = XWv
↓
RoPE + Attention
QKᵀ → Mask → Softmax → × V
QKᵀ → Mask → Softmax → × V
KV Cache (VRAM)
↓
Output Projection + Residual + Norm
↓
FFN / MLP
MatMul → Activation / Gating → MatMul
MatMul → Activation / Gating → MatMul
↓
Residual → Repeat Transformer Blocks × N
↓
Final Norm → Vocabulary Projection → Logits
↓
Sampling → Next Token
Q = X Wq ; K = X Wk ; V = X Wv
Attention(Q,K,V) = softmax(QKᵀ / √dₖ + mask) V
Attention(Q,K,V) = softmax(QKᵀ / √dₖ + mask) V
FFN(X) ≈ W₂ · activation(W₁X + b₁) + b₂
Example: 4096 → 11008 → 4096
Example: 4096 → 11008 → 4096
| Operation |
|---|
WeightsKV CacheActivationsWorkspace
| Precision | Bytes / parameter | 7B model weights |
|---|---|---|
| FP32 | 4 | 28 GB |
| FP16 / BF16 | 2 | 14 GB |
| INT8 | 1 | 7 GB |
| INT4 | 0.5 | 3.5 GB |
KV bytes = 2 × layers × cached_tokens × KV_heads × head_dim × bytes_per_element × sequences
—
① Prefill
Many prompt tokens
→
Build KV cache
② Decode
New token
→
Read + append KV
| Prefill | Decode |
|---|
Attainable FLOP/s ≤ min(Peak FLOP/s, Bandwidth × Arithmetic Intensity)
—
TFLOP/s
Arithmetic Intensity (FLOP/byte)
14 GB ÷ 1,000 GB/s = 0.014 s = 14 ms
Forward pass
Activations + Loss
Activations + Loss
→
Backward pass
Gradients
Gradients
→
Optimizer step
Update weights
Update weights
| Inference | Training |
|---|