LLM ENGINEERING · OFFLINE GUIDE
LLM Terminology: A Complete Beginner’s Guide
Build a solid understanding of layers, RNNs, FFNs, gradients, fine-tuning, LoRA, QLoRA, FLOPS, GPUs, and production inference.
EnglishTechnical GlossaryWorks OfflineInteractive LoRA Calculator01 · The LLM Landscape
Large Language Models (LLMs) bring together neural networks, mathematics, training, fine-tuning, GPU computing, and inference deployment.
LLM (Large Language Model)GPT · Llama · Qwen
↓
Neural Network ArchitectureTransformer · Layer · Attention · FFN
Mathematical FoundationsTensor · Matrix · Gradient · Loss
Model TrainingBackpropagation · Optimizer · Epoch
Model Fine-tuningFine-tuning · LoRA · QLoRA
Compute PerformanceFLOPS · GPU · VRAM · BF16
Model InferenceInference · KV Cache · Quantization
02 · Neural Network Fundamentals
| Term | Meaning | Purpose |
|---|---|---|
| Neural Network | Neural network | Core computational architecture of a model |
| Neuron | Neuron | Weighted computations and nonlinear transformations |
| Parameter | Parameter | Values learned by the model |
| Weight | Weight | Controls the influence of input signals |
| Bias | Bias | Shifts a neuron's output |
| Layer | Layer | A stage of computation |
| FNN | Feedforward neural network | Information flows forward through the network |
| RNN | Recurrent neural network | Processes sequences using previous hidden states |
| Transformer | Transformer architecture | Dominant architecture for modern LLMs |
| Attention | Attention mechanism | Models relationships among tokens |
| FFN | Feedforward network | Nonlinear transformation module in a Transformer |
What is a Layer?
Think of a layer as a processing stage. A 70-layer Transformer usually means 70 Transformer blocks—not 70 individual neurons.
Input: I enjoy learning AI
↓
Embedding LayerConverts tokens into vectors
↓
Transformer Layer 1Models relationships between tokens
↓
Transformer Layer 2 … NProgressively transforms contextual representations
↓
Output LayerProduces scores and a probability distribution for the next token
A Transformer block typically contains attention, an FFN, normalization, and residual connections.
Comparing FNN, FFN, RNN, and Transformer
| Architecture | How it works | Role |
|---|---|---|
| FNN | Data flows forward through the network | Basic network architecture |
| FFN | Multilayer feedforward transformation | Module inside a Transformer block |
| RNN | Passes the previous hidden state to the next step | Earlier sequence-modeling approach |
| LSTM | Improves RNNs using gates | Common in earlier NLP systems |
| Transformer | Uses attention to model token relationships | Mainstream LLM architecture |
FNN and FFN are sometimes used interchangeably. In Transformer discussions, FFN usually refers to the feedforward subnetwork inside a block.
03 · Mathematics and Model Training
| Term | Meaning | Explanation |
|---|---|---|
| Scalar | Scalar | A single numerical value |
| Vector | Vector | A one-dimensional collection of numbers |
| Matrix | Matrix | A two-dimensional array of numbers |
| Tensor | Tensor | Multidimensional numerical data |
| Matrix Multiplication | Matrix multiplication | A core operation in LLMs |
| Gradient | Gradient | How sensitive the loss is to parameter changes |
| Loss Function | Loss function | Measures prediction error |
| Backpropagation | Backpropagation | Computes gradients for model parameters |
| Gradient Descent | Gradient descent | Updates parameters using gradients |
| Learning Rate | Learning rate | Controls the size of parameter updates |
| Optimizer | Optimizer | Updates parameters, e.g., AdamW |
| Batch Size | Batch size | Number of training examples per batch |
| Epoch | Epoch | One full pass through the training dataset |
| Step | Step | Usually one optimizer update |
How Do Gradients Work?
Suppose the input is 1 + 1 =, and the correct answer is 2, but the model assigns 3 a higher probability. Training aims to reduce the loss.
Illustration: update weights along a local downhill direction to reduce loss.
Wnew = Wold − η · ∂L/∂W
W is the weight, η is the learning rate, L is the loss, and ∂L/∂W is the gradient.
A gradient does not update a model by itself. Backpropagation computes gradients; the optimizer uses them to update weights.
04 · Fine-tuning, LoRA, and QLoRA
| Technique | Meaning | Key difference |
|---|---|---|
| Pre-training | Pre-training | Trains a model from scratch on large datasets |
| Fine-tuning | Fine-tuning | Continues training on task-specific data |
| Full Fine-tuning | Full fine-tuning | Updates all or nearly all model parameters |
| PEFT | Parameter-efficient fine-tuning | Trains a small set of added or selected parameters |
| LoRA | Low-rank adaptation | Represents weight updates with small matrices |
| QLoRA | Quantized low-rank adaptation | Quantizes the base model and trains LoRA adapters |
| SFT | Supervised fine-tuning | Trains on inputs paired with target responses |
| DPO | Direct preference optimization | Improves outputs using preference data |
| RLHF | Reinforcement learning from human feedback | Uses human preference signals in reinforcement learning |
Why Does LoRA Save Resources?
Suppose a weight matrix is 4096 × 4096, containing 16,777,216 parameters. LoRA freezes the original matrix and learns two smaller matrices instead.
W′ = W + BA
B: 4096 × 16 ｜ A: 16 × 4096
B: 4096 × 16 ｜ A: 16 × 4096
With rank 16, trainable adapter parameters = 4096 × 16 + 16 × 4096 = 131,072。
Original matrix16.78M
Added LoRA parameters131K
In this single-matrix example, the LoRA adapter has only 0.78% of the original parameter count. The full-model ratio depends on which modules receive adapters.
LoRA vs QLoRA
| Comparison | LoRA | QLoRA |
|---|---|---|
| Base model weights | Usually BF16 / FP16 | Usually 4-bit |
| Trainable weights | LoRA adapter | LoRA adapter |
| VRAM | Higher | Lower |
| Training speed | Typically more straightforward | May incur dequantization overhead |
| Quality | Depends on data and settings | Can approach LoRA quality; not guaranteed |
| Best suited for | Sufficient GPU memory | Limited GPU memory |
QLoRA generally freezes the quantized base model and uses backpropagation to train LoRA adapters. It does not directly update all 4-bit base weights.
05 · GPUs, FLOPS, VRAM, and Precision
| Term | Meaning | Meaning |
|---|---|---|
| FLOP | Floating-point operation | One floating-point operation |
| FLOPS | Floating-point operations per second | Computational throughput |
| TFLOPS | TeraFLOPS | 10¹² FLOPS |
| PFLOPS | PetaFLOPS | 10¹⁵ FLOPS |
| VRAM | GPU video memory | Stores weights and intermediate data |
| CUDA | NVIDIA parallel computing platform | GPU software ecosystem |
| Tensor Core | Tensor Core | Accelerates matrix computations |
| FP32 | 32-bit floating point | Higher precision |
| FP16 | 16-bit floating point | Lower memory requirements |
| BF16 | Brain Floating Point | Commonly used in training |
| INT8 | 8-bit integer | Quantized representation |
| INT4 | 4-bit integer | Even lower memory usage |
| HBM | High-bandwidth memory | Used by high-end AI GPUs |
| Memory Bandwidth | Memory bandwidth | Data transfer rate |
Theoretical Weight Storage for a 70B Model
| Precision | Theoretical weight size |
|---|---|
| FP32 | 280 GB |
| FP16 / BF16 | 140 GB |
| INT8 | 70 GB |
| INT4 | 35 GB |
Sizes use decimal GB and include only model weights—not quantization metadata, KV cache, activations, optimizer states, or CUDA workspace. High FLOPS do not guarantee fast single-request inference: decoding can be limited by memory bandwidth.
06 · Inference Terminology
| Term | Meaning | Purpose |
|---|---|---|
| Inference | Inference | Generates results with a trained model |
| Token | Token | Basic unit of text processing |
| Tokenizer | Tokenizer | Converts text into token IDs |
| Embedding | Embedding | Converts tokens into numerical vectors |
| Context Window | Context window | Maximum supported token context |
| Prefill | Prefill | Processes input tokens and builds the KV cache |
| Decode | Decode | Generates tokens incrementally |
| KV Cache | Key/value cache | Avoids recomputing previous tokens' keys and values |
| TTFT | Time to first token | Time from request to first output token |
| TPS | Tokens per second | Generation speed or system throughput |
| Latency | Latency | Time between request and response |
| Throughput | Throughput | Work completed per unit time |
| Quantization | Quantization | Reduces numerical precision |
| Continuous Batching | Continuous batching | Dynamically batches inference requests |
| Speculative Decoding | Speculative decoding | Speeds up generation using a draft model or related methods |
07 · End-to-End Training and Inference
Training DataText / Code / Documents
↓
Tokenizer → EmbeddingConverts text to tokens, then vectors
↓
Transformer × N LayersLayer Norm · Residual · Matrix Multiplication
AttentionQ / K / V
FFNLinear + Activation
↓
Prediction → LossPredicts tokens and computes error
↓
TrainingBackward → Gradient → Optimizer → Weight Update
InferencePrefill → Decode → Token Output
Training repeatedly updates weights. Inference generates outputs with trained weights and does not perform ordinary weight updates.
08 · Interactive LoRA Rank Calculator
Adjust the rank to see how the number of trainable parameters changes for a single 4096 × 4096 matrix. All calculations run locally in your browser.
Trainable LoRA parameters0.131M
Percentage of original matrix0.78%
4096 × 16 + 16 × 4096 = 131,072
A higher rank generally increases adapter capacity, memory use, and compute cost, but does not guarantee better quality.
09 · Advanced LLM Terms
| Category | Term | Purpose |
|---|---|---|
| Architecture | MoE (Mixture of Experts) | Multiple expert subnetworks |
| Architecture | GQA (Grouped Query Attention) | Reduces KV cache overhead |
| Architecture | RoPE | Encodes token positions |
| Architecture | RMSNorm | Normalizes hidden states |
| Architecture | SwiGLU | Common FFN activation |
| Training | Gradient Accumulation | Accumulates gradients across microbatches |
| Training | Gradient Checkpointing | Trades extra compute for lower memory usage |
| Training | Mixed Precision | Uses multiple numerical precisions |
| Distributed training | DDP | Distributed data parallelism |
| Distributed training | FSDP / ZeRO | Shards model states |
| Inference | FlashAttention | Efficient attention computation |
| Inference | PagedAttention | Paged KV cache management |
| Inference | Tensor Parallelism | Splits tensor computation across GPUs |
| Inference | Pipeline Parallelism | Runs different model layers on different GPUs |
| Applications | RAG | Retrieves external information to support generation |
| Applications | Embedding Model | Converts content into vectors |
| Applications | Vector Database | Vector storage and search |
10 · Recommended Learning Roadmap
01
Mathematical Foundations
Vector → Matrix → Tensor → Matrix Multiplication
02
Neural network
Weight → Bias → Layer → FFN → Activation
03
Model Training
Forward → Loss → Gradient → Backpropagation → Optimizer
04
Transformer
Embedding → Q/K/V → Attention → FFN → Layer
05
Model Fine-tuning
Fine-tuning → PEFT → LoRA → QLoRA → DPO
06
GPUs and Performance
FLOPS → VRAM → HBM → Precision → Quantization
07
Production Inference
Prefill → Decode → KV Cache → Batching → Parallelism
Training（Training）Gradient · Optimizer · GPU Compute · Training Memory
Inference（Inference）Token Generation · KV Cache · Memory Bandwidth · Latency · Throughput