Glossary

TL;DR

Every term used across this track, in one place, grouped by topic. Use this as a reference while reading, not something to memorize up front.

Property	Value
Level	Reference
Reading time	~10 minutes, or look up as needed
Prerequisites	Designing a Training System
You will understand	Where to look up any term from this track

Core Mechanics

Term	Meaning
Tensor	A multi-dimensional array — the basic data structure of deep learning
`nn.Module`	PyTorch's base class for any layer or model
Forward pass	Computing a prediction from an input by running it through the network
Backpropagation	Computing the gradient of the loss with respect to every parameter, via the chain rule
Autodiff	The engine (`autograd`) that computes those gradients automatically
Chain rule	The calculus rule letting a derivative through many composed functions be computed as a product of local derivatives
Loss function	The single number a training run tries to minimize
Gradient descent	Repeatedly nudging parameters in the direction that reduces the loss

Optimization & Training

Term	Meaning
SGD	Stochastic Gradient Descent — the basic parameter-update rule
Momentum	Accumulating a running average of past gradients to smooth and accelerate updates
Adam	An optimizer combining momentum with per-parameter adaptive learning rates
AdamW	Adam with weight decay decoupled from the gradient-based update, rather than folded into it
Learning rate	How large a step the optimizer takes on each parameter update
Warmup	Gradually increasing the learning rate at the start of training, before reaching its target value
Cosine decay	A smooth, cosine-shaped learning rate schedule decreasing toward the end of training
Gradient clipping	Capping the magnitude of the gradient to prevent an unstable, oversized update
Batch size	How many examples are processed together in one forward/backward pass
Vanishing / exploding gradients	Gradient magnitude shrinking or growing exponentially with depth during backpropagation
Dead ReLU	A ReLU unit permanently stuck outputting zero, no longer receiving a learning signal
Checkpoint	A saved snapshot of model and optimizer state during training

Architecture

Term	Meaning
Attention	The mechanism letting each token weigh and incorporate every other token's representation
Query / Key / Value (Q/K/V)	The three learned projections attention uses to compute what to attend to and what to retrieve
Scaled dot-product attention	The specific attention formula: softmax(QKᵀ/√d)V
Multi-head attention	Running several attention computations in parallel, each learning a different kind of relationship
Positional encoding	Information added to token representations so the model can tell tokens' order apart
RoPE	Rotary Positional Embedding — a common modern positional encoding scheme
Residual (skip) connection	Adding a layer's input directly to its output, giving gradients an unobstructed path
Layer normalization	Rescaling activations across a layer's features to a stable distribution
Batch normalization	Rescaling activations using statistics computed across a training batch
Dropout	Randomly zeroing activations during training to prevent over-reliance on any single pathway

Generalization

Term	Meaning
Weight decay / L2 regularization	Penalizing large weights in the loss to encourage simpler solutions
Overfitting	Memorizing training-specific patterns instead of learning generalizable ones
Early stopping	Halting training once validation performance stops improving
Embedding	A learned vector representation positioning similar things near each other
Contrastive learning	Training an embedding space by pulling similar pairs together and pushing dissimilar pairs apart
InfoNCE	A common contrastive loss function used to train embedding models
Cosine similarity	The angle-based similarity measure between two vectors, commonly used on embeddings

Fine-Tuning & Compression

Term	Meaning
Transfer learning	Starting from a model already trained on general data and adapting it to a specific task
Fine-tuning	Continuing to train a pretrained model's weights on new, task-specific data
LoRA	Low-Rank Adaptation — learning a small low-rank weight update instead of updating the full matrix
Low-rank hypothesis	The empirical observation that most useful weight updates for a task live in a small-dimensional subspace
Rank (r)	LoRA's key hyperparameter — the dimensionality of the low-rank update
Alpha scaling	A LoRA scaling factor controlling how strongly the low-rank update affects the frozen weights
Quantization	Mapping continuous weight values onto a smaller discrete set to save memory
Scale / zero-point	The parameters of an affine quantization mapping between float and integer ranges
Symmetric / asymmetric quantization	Whether the quantization range is centered at zero or shifted to fit the data's actual range
Per-channel quantization	Using a separate scale/zero-point per channel instead of one for the whole tensor, for better accuracy
QAT (Quantization-Aware Training)	Training with quantization effects simulated during the forward pass, rather than applied only afterward
Straight-through estimator	A trick for backpropagating through quantization's non-differentiable rounding step
Knowledge distillation	Training a smaller student model to imitate a larger teacher's output distribution
Soft targets	A teacher's full probability distribution over outputs, used as a richer training signal than hard labels
Temperature	A scaling factor that softens a probability distribution, revealing more information about near-miss answers
KL divergence	A measure of how different two probability distributions are — the core term in a distillation loss

Alignment

Term	Meaning
Bradley-Terry model	A statistical model of pairwise preference probability, underlying reward modeling and DPO
Reward model	A model trained to score how good a response is, used in classic RLHF
RLHF	Reinforcement Learning from Human Feedback — the classic reward-model-plus-RL alignment recipe
PPO	Proximal Policy Optimization — the RL algorithm classic RLHF typically uses
DPO	Direct Preference Optimization — aligning a model to preference pairs without a separate reward model
Beta (β), in DPO	The parameter controlling how far the model is allowed to drift from its frozen reference copy

Distributed Training

Term	Meaning
DDP	Distributed Data Parallel — a full model copy per GPU, gradients synced via all-reduce
FSDP	Fully Sharded Data Parallel — parameters, gradients, and optimizer state sharded across GPUs
DTensor	A distributed tensor tracking its own sharding layout — the basis of FSDP2's implementation
All-reduce	A collective operation combining a tensor across all participants so every one ends up with the same result
All-gather	A collective operation reconstructing a full tensor from every participant's shard
Mixed precision	Computing in reduced precision (bf16/fp16) while keeping an FP32 master copy of the weights
Master weights	The full-precision weight copy preventing small updates from rounding away to nothing

You've covered the whole track. Project Ideas proposes building a small transformer entirely from scratch — the project that puts every mechanism from this glossary in your own hands. For a fast one-page refresher on any of it, revisit the Deep Learning Crash Course.

Glossary

TL;DR

Every term used across this track, in one place, grouped by topic. Use this as a reference while reading, not something to memorize up front.

Property	Value
Level	Reference
Reading time	~10 minutes, or look up as needed
Prerequisites	Designing a Training System
You will understand	Where to look up any term from this track

Core Mechanics

Term	Meaning
Tensor	A multi-dimensional array — the basic data structure of deep learning
`nn.Module`	PyTorch's base class for any layer or model
Forward pass	Computing a prediction from an input by running it through the network
Backpropagation	Computing the gradient of the loss with respect to every parameter, via the chain rule
Autodiff	The engine (`autograd`) that computes those gradients automatically
Chain rule	The calculus rule letting a derivative through many composed functions be computed as a product of local derivatives
Loss function	The single number a training run tries to minimize
Gradient descent	Repeatedly nudging parameters in the direction that reduces the loss

Optimization & Training

Term	Meaning
SGD	Stochastic Gradient Descent — the basic parameter-update rule
Momentum	Accumulating a running average of past gradients to smooth and accelerate updates
Adam	An optimizer combining momentum with per-parameter adaptive learning rates
AdamW	Adam with weight decay decoupled from the gradient-based update, rather than folded into it
Learning rate	How large a step the optimizer takes on each parameter update
Warmup	Gradually increasing the learning rate at the start of training, before reaching its target value
Cosine decay	A smooth, cosine-shaped learning rate schedule decreasing toward the end of training
Gradient clipping	Capping the magnitude of the gradient to prevent an unstable, oversized update
Batch size	How many examples are processed together in one forward/backward pass
Vanishing / exploding gradients	Gradient magnitude shrinking or growing exponentially with depth during backpropagation
Dead ReLU	A ReLU unit permanently stuck outputting zero, no longer receiving a learning signal
Checkpoint	A saved snapshot of model and optimizer state during training

Architecture

Term	Meaning
Attention	The mechanism letting each token weigh and incorporate every other token's representation
Query / Key / Value (Q/K/V)	The three learned projections attention uses to compute what to attend to and what to retrieve
Scaled dot-product attention	The specific attention formula: softmax(QKᵀ/√d)V
Multi-head attention	Running several attention computations in parallel, each learning a different kind of relationship
Positional encoding	Information added to token representations so the model can tell tokens' order apart
RoPE	Rotary Positional Embedding — a common modern positional encoding scheme
Residual (skip) connection	Adding a layer's input directly to its output, giving gradients an unobstructed path
Layer normalization	Rescaling activations across a layer's features to a stable distribution
Batch normalization	Rescaling activations using statistics computed across a training batch
Dropout	Randomly zeroing activations during training to prevent over-reliance on any single pathway

Generalization

Term	Meaning
Weight decay / L2 regularization	Penalizing large weights in the loss to encourage simpler solutions
Overfitting	Memorizing training-specific patterns instead of learning generalizable ones
Early stopping	Halting training once validation performance stops improving
Embedding	A learned vector representation positioning similar things near each other
Contrastive learning	Training an embedding space by pulling similar pairs together and pushing dissimilar pairs apart
InfoNCE	A common contrastive loss function used to train embedding models
Cosine similarity	The angle-based similarity measure between two vectors, commonly used on embeddings

Fine-Tuning & Compression

Term	Meaning
Transfer learning	Starting from a model already trained on general data and adapting it to a specific task
Fine-tuning	Continuing to train a pretrained model's weights on new, task-specific data
LoRA	Low-Rank Adaptation — learning a small low-rank weight update instead of updating the full matrix
Low-rank hypothesis	The empirical observation that most useful weight updates for a task live in a small-dimensional subspace
Rank (r)	LoRA's key hyperparameter — the dimensionality of the low-rank update
Alpha scaling	A LoRA scaling factor controlling how strongly the low-rank update affects the frozen weights
Quantization	Mapping continuous weight values onto a smaller discrete set to save memory
Scale / zero-point	The parameters of an affine quantization mapping between float and integer ranges
Symmetric / asymmetric quantization	Whether the quantization range is centered at zero or shifted to fit the data's actual range
Per-channel quantization	Using a separate scale/zero-point per channel instead of one for the whole tensor, for better accuracy
QAT (Quantization-Aware Training)	Training with quantization effects simulated during the forward pass, rather than applied only afterward
Straight-through estimator	A trick for backpropagating through quantization's non-differentiable rounding step
Knowledge distillation	Training a smaller student model to imitate a larger teacher's output distribution
Soft targets	A teacher's full probability distribution over outputs, used as a richer training signal than hard labels
Temperature	A scaling factor that softens a probability distribution, revealing more information about near-miss answers
KL divergence	A measure of how different two probability distributions are — the core term in a distillation loss

Alignment

Term	Meaning
Bradley-Terry model	A statistical model of pairwise preference probability, underlying reward modeling and DPO
Reward model	A model trained to score how good a response is, used in classic RLHF
RLHF	Reinforcement Learning from Human Feedback — the classic reward-model-plus-RL alignment recipe
PPO	Proximal Policy Optimization — the RL algorithm classic RLHF typically uses
DPO	Direct Preference Optimization — aligning a model to preference pairs without a separate reward model
Beta (β), in DPO	The parameter controlling how far the model is allowed to drift from its frozen reference copy

Distributed Training

Term	Meaning
DDP	Distributed Data Parallel — a full model copy per GPU, gradients synced via all-reduce
FSDP	Fully Sharded Data Parallel — parameters, gradients, and optimizer state sharded across GPUs
DTensor	A distributed tensor tracking its own sharding layout — the basis of FSDP2's implementation
All-reduce	A collective operation combining a tensor across all participants so every one ends up with the same result
All-gather	A collective operation reconstructing a full tensor from every participant's shard
Mixed precision	Computing in reduced precision (bf16/fp16) while keeping an FP32 master copy of the weights
Master weights	The full-precision weight copy preventing small updates from rounding away to nothing

Glossary

Glossary

Core Mechanics

Optimization & Training

Architecture

Generalization

Fine-Tuning & Compression

Alignment

Distributed Training

Next

On this page

Glossary

Glossary

Core Mechanics

Optimization & Training

Architecture

Generalization

Fine-Tuning & Compression

Alignment

Distributed Training

Next

On this page