Deep Learning Crash Course
All of deep learning on one page — neural networks, training loops, the transformer, fine-tuning, compression, alignment, and distributed training, from first principles
Deep Learning Crash Course
Start here
This single page covers all of deep learning at working depth, from first principles rather than library APIs. Read it start to finish in about 30 minutes and you will understand how a neural network actually learns, how a transformer is built, and how the techniques that make modern models small, aligned, and fast at inference actually work under the hood.
Every section ends with a Go deeper link to a full page on that topic. Read this first, then follow the links for whatever you need in detail.
| Property | Value |
|---|---|
| Level | Everyone — starts from zero, ends at production |
| Reading time | ~30 minutes |
| Prerequisites | None. Basic Python and high-school calculus help but aren't required — every idea is explained from the ground up. |
| You will understand | The mechanics beneath every modern model, well enough to build, train, and debug one |
1. What Deep Learning Actually Is
How this track differs from HuggingFace
The HuggingFace track teaches the library APIs that load and run pretrained models — pipeline(), Trainer, peft. This track teaches what those APIs are actually doing underneath — the math and mechanics of nn.Module, backpropagation, attention, and gradient descent, in raw PyTorch. Understanding both is what separates using a model from actually knowing how it works.
A neural network is a function with learnable parameters — numbers that start random and get adjusted, repeatedly, until the function produces useful outputs for a given input. "Deep" means the function is built from many stacked layers, each transforming its input a little, so the network as a whole can represent far more complex relationships than any single layer could alone.
| Piece | What it does |
|---|---|
| Parameters (weights) | The numbers that get learned — everything else is fixed |
| Forward pass | Run the input through the layers to get a prediction |
| Loss function | A single number measuring how wrong that prediction was |
| Backward pass (backpropagation) | Compute how much each parameter contributed to that wrongness |
| Optimizer | Nudge every parameter slightly in the direction that reduces the loss |
A neural network doesn't "understand" anything — it fits a function to data. Every capability that looks like understanding (translation, reasoning, code generation) emerges from a huge number of small, mechanical weight adjustments made this way, repeated billions of times over a huge amount of data. Keeping that mental model honest makes debugging far easier: when a model is wrong, ask what in its training data or loss function produced that behavior, not what it was "thinking."
Go deeper: What Is Deep Learning?.
2. Building a Network: nn.Module
PyTorch's nn.Module is the building block for every layer and every model.
import torch.nn as nn
class TinyClassifier(nn.Module):
def __init__(self, input_dim, hidden_dim, num_classes):
super().__init__()
self.fc1 = nn.Linear(input_dim, hidden_dim)
self.activation = nn.ReLU()
self.fc2 = nn.Linear(hidden_dim, num_classes)
def forward(self, x):
return self.fc2(self.activation(self.fc1(x)))| Piece | Role |
|---|---|
nn.Linear | A learnable linear transformation — the workhorse layer |
Activation function (ReLU, GELU, ...) | A nonlinearity between layers — without one, stacking layers is mathematically no different from one big linear layer |
forward() | Defines how data actually flows through the layers |
The nonlinearity is not optional decoration. Stack ten linear layers with no activation between them and the result collapses algebraically into a single linear layer — all that depth buys you nothing. The activation function is what lets depth actually add expressive power.
Go deeper: Neural Network Fundamentals.
3. How a Network Learns: The Training Loop
One training step
Forward pass
Run a batch of inputs through the model to get predictions
Compute the loss
Compare predictions to the true labels with a loss function
Backward pass
Autodiff computes the gradient of the loss with respect to every parameter
Optimizer step
Nudge every parameter against its gradient, scaled by the learning rate
for batch in dataloader:
optimizer.zero_grad()
predictions = model(batch.inputs)
loss = loss_fn(predictions, batch.labels)
loss.backward()
optimizer.step()loss.backward() is where PyTorch's autodiff engine does the real work — it automatically computes the gradient (the direction and size of the effect) of the loss with respect to every single parameter, using the chain rule, without you writing any calculus by hand.
The learning rate is the single most consequential hyperparameter. Too high and training diverges — the loss oscillates or explodes. Too low and training is needlessly slow, or gets stuck in a shallow local minimum. Most training problems that look mysterious trace back to this one number.
Go deeper: Training Loops & Optimization — optimizers (SGD, Adam, AdamW), learning rate schedules, gradient clipping.
4. The Transformer, From Scratch
Every modern language model — and most modern vision and audio models — is built from the same core mechanism: attention.
The idea in one sentence
Attention in one sentence
For each token, compute how much it should "attend to" every other token in the sequence, then take a weighted average of their representations — so every token's output can incorporate context from anywhere else in the input, not just its neighbors.
import torch, math
def scaled_dot_product_attention(Q, K, V):
scores = Q @ K.transpose(-2, -1) / math.sqrt(Q.shape[-1])
weights = torch.softmax(scores, dim=-1)
return weights @ V| Term | Role |
|---|---|
| Query (Q) | "What is this token looking for?" |
| Key (K) | "What does each token offer?" |
| Value (V) | "What does each token actually contribute, once attended to?" |
| Multi-head attention | Run several attention computations in parallel, each learning to focus on a different kind of relationship |
One transformer block
Attention has no inherent sense of order. Swap two tokens' positions and, without something extra, attention treats the sequence identically — it operates on a set, not a sequence. That's why every transformer adds explicit positional information (learned positions, sinusoidal encodings, or rotary embeddings like RoPE) — without it, "the dog bit the man" and "the man bit the dog" would look the same to the attention mechanism.
Go deeper: The Transformer Architecture — full multi-head attention, positional encodings, and building a complete block.
5. Keeping a Network From Memorizing: Regularization
A network with enough parameters can simply memorize its training data instead of learning generalizable patterns — a failure called overfitting. Several techniques fight this directly:
| Technique | What it does |
|---|---|
| Dropout | Randomly zero out a fraction of activations during training, forcing the network not to rely too heavily on any single pathway |
| Weight decay | Penalize large weights in the loss function, encouraging simpler solutions |
| Batch / layer normalization | Rescale activations to a stable distribution, which also smooths and speeds up training |
| Early stopping | Stop training once validation performance stops improving, before the model starts memorizing |
The gap between training loss and validation loss is the diagnostic. Both dropping together means the model is still learning generalizable patterns. Training loss still dropping while validation loss rises is the definitive sign of overfitting — and the point to stop, or add more regularization.
Go deeper: Regularization & Generalization.
6. Embeddings and Representation Learning
An embedding is a learned vector representation of something — a word, a sentence, an image — positioned so that similar things end up with similar vectors.
embed("king") - embed("man") + embed("woman") ≈ embed("queen")That arithmetic works because the embedding space was trained to place semantically related concepts near each other along meaningful directions — not because it was explicitly told the relationship. This is what makes embeddings useful for everything from search to RAG's retrieval step (see Embeddings Explained) to classification.
Go deeper: Embeddings & Representation Learning — contrastive learning, how embedding spaces are actually trained.
7. Adapting a Trained Network: Fine-Tuning and LoRA
Training from scratch is expensive. Transfer learning — starting from a model already trained on a large, general dataset and adapting it to a specific task — is almost always the better starting point.
The math behind LoRA
Full fine-tuning updates every weight matrix W. LoRA instead learns a low-rank update ΔW = BA, where B and A are much smaller matrices, and freezes the original W entirely:
h = Wx + ΔWx = Wx + BAx
W: d × d (frozen, e.g. 4096 × 4096 = ~16.8M parameters)
B: d × r (trained, e.g. 4096 × 8 = ~32K parameters)
A: r × d (trained, e.g. 8 × 4096 = ~32K parameters)With r (the rank) much smaller than d, B and A together have a tiny fraction of the parameters of W — yet because most useful weight updates for a specific task turn out to live in a low-dimensional subspace, this small number of trainable parameters is often enough to match full fine-tuning quality.
This is the same LoRA used practically in the HuggingFace fine-tuning page via the peft library — that page shows how to use it; this page shows why the math works.
Go deeper: Fine-Tuning & Transfer Learning.
8. Making a Network Smaller: Quantization From First Principles
A trained weight is normally stored as a 32-bit or 16-bit floating-point number. Quantization maps that continuous range onto a much smaller set of discrete values — commonly 8-bit or 4-bit integers — trading a little numerical precision for a large reduction in memory and, often, faster compute.
# A minimal linear (affine) quantization, the core idea behind most schemes
scale = (w_max - w_min) / 255
zero_point = round(-w_min / scale)
w_int8 = round(w / scale) + zero_point # quantize
w_reconstructed = (w_int8 - zero_point) * scale # dequantizeThe reconstruction is never perfect — that gap is the quantization error. Why it usually doesn't hurt much: trained weights aren't spread uniformly across their range; most cluster near zero, and a well-chosen scale/zero_point (or smarter, non-uniform schemes) puts most of the representable precision where the weights actually are.
Go deeper: Quantization From First Principles — the mechanics behind the practical formats covered in the HuggingFace and Small Language Models tracks.
9. Compressing a Network: Knowledge Distillation
Instead of shrinking a trained network's weights, distillation trains a smaller network from scratch to imitate a larger one's behavior.
# Distillation loss: match the teacher's soft probability distribution,
# not just its single hard predicted label
student_logits = student(x)
with torch.no_grad():
teacher_logits = teacher(x)
loss = kl_divergence(
softmax(student_logits / T), softmax(teacher_logits / T)
) * T**2The temperature T is the key trick. Dividing both models' outputs by T > 1 before the softmax "softens" the probability distribution — instead of the teacher saying "99.9% cat," it says something like "92% cat, 5% dog, 3% other," revealing information about which wrong answers it considers plausible. That extra signal, absent from hard labels alone, is what makes the student learn more than it could from the raw training labels.
Go deeper: Knowledge Distillation Mechanics — the math behind the practical distillation trainers used in the Small Language Models track.
10. Aligning a Network: The Math Behind DPO
Supervised fine-tuning teaches a model to imitate examples. Alignment teaches it to prefer better outputs among options it can already produce — using pairs of (chosen, rejected) responses.
DPO (Direct Preference Optimization) derives a loss directly from the Bradley-Terry model of pairwise preference, which states the probability that response A is preferred over B is a function of the difference in their underlying "quality" scores:
loss = -log σ( β · [ log π(chosen|x)/π_ref(chosen|x) − log π(rejected|x)/π_ref(rejected|x) ] )In plain terms: increase the model's relative probability of the chosen response over the rejected one, compared to a frozen reference copy of the model — and β controls how far the model is allowed to drift from that reference. This is the mathematical trick that lets DPO skip training a separate reward model entirely, which is what made it replace classic RLHF for most teams.
Go deeper: Alignment & DPO Math — the full derivation, and how it relates to the practical DPOTrainer in the HuggingFace track.
11. Training Across Many GPUs: The Mechanics
A model or batch too large for one GPU needs to be split across several. Two fundamentally different ways to split it:
What actually gets split
Data parallelism (DDP)
RecommendedEvery GPU holds a full copy of the model and processes a different slice of the batch. After the backward pass, gradients are synchronized (averaged) across all GPUs before the optimizer step, so every copy stays identical.
Model/parameter sharding (FSDP2)
The model's own parameters, gradients, and optimizer state are split across GPUs — no GPU ever holds the whole model. Each GPU gathers the full parameters it needs just before computing, then frees them again. PyTorch's current FSDP2 represents each sharded parameter as a DTensor, which is what makes it compose cleanly with techniques like LoRA that only train a subset of parameters.
DDP needs the model to fit on one GPU; FSDP exists for when it doesn't. Reaching for sharding before you've actually hit a memory wall adds real communication overhead for no benefit — DDP's only communication cost is syncing gradients once per step, while FSDP-style sharding pays a communication cost on every forward and backward pass to gather and release parameters.
Go deeper: Distributed Training Internals — gradient synchronization, mixed precision math, and FSDP2 in depth.
12. When Training Goes Wrong
| Symptom | Usual cause |
|---|---|
Loss is NaN | Learning rate too high, or an unstable operation (e.g. log(0)) — check gradient clipping and numerical stability first |
| Loss never decreases | Learning rate too low, a bug in the loss function, or gradients not flowing (a frozen layer that should be trainable) |
| Training loss drops, validation loss rises | Overfitting — add regularization, get more data, or stop earlier |
| Gradients shrink to near zero in early layers | Vanishing gradients — check activation functions and normalization, consider residual connections |
| Loss oscillates wildly | Learning rate too high, or batch size too small for the chosen learning rate |
| Model trains fine, then value explodes mid-run | Gradient explosion — add gradient clipping |
Go deeper: Evaluating & Debugging Training.
13. Vocabulary You Need
| Term | Meaning |
|---|---|
| Tensor | A multi-dimensional array — the basic data structure of deep learning |
nn.Module | PyTorch's base class for any layer or model |
| Forward pass | Computing a prediction from an input |
| Backpropagation | Computing the gradient of the loss with respect to every parameter, via the chain rule |
| Autodiff | The engine that computes those gradients automatically |
| Loss function | The single number training tries to minimize |
| Optimizer | The algorithm that updates parameters using their gradients (SGD, Adam, AdamW) |
| Learning rate | How large a step the optimizer takes on each update |
| Attention | The mechanism letting each token weigh and incorporate every other token's representation |
| LoRA | Low-Rank Adaptation — fine-tuning via a small learned update instead of the full weight matrix |
| Quantization | Mapping continuous weight values onto a smaller discrete set to save memory |
| Distillation | Training a smaller student model to imitate a larger teacher's output distribution |
| DPO | Direct Preference Optimization — aligning a model to preference pairs without a reward model |
| DDP | Distributed Data Parallel — every GPU holds a full model copy |
| FSDP | Fully Sharded Data Parallel — the model itself is split across GPUs |
| Overfitting | Memorizing training data instead of learning generalizable patterns |
Go deeper: Glossary.
14. The Twelve Rules
| # | Rule |
|---|---|
| 1 | A network fits a function to data — it doesn't "understand." Debug the data and the loss, not the model's "intent." |
| 2 | Nonlinearities are what make depth matter. Stacked linear layers without them collapse to one linear layer. |
| 3 | The learning rate is the highest-leverage hyperparameter. Most mysterious training failures trace back to it. |
| 4 | Attention has no inherent order. Positional information must always be added explicitly. |
| 5 | Watch the train/validation gap, not just the training loss. It's the actual overfitting signal. |
| 6 | Most useful fine-tunes live in a low-rank subspace. That's why LoRA works, not just that it's cheaper. |
| 7 | Quantization error is small because weights aren't uniform. Precision should be spent where the weights actually are. |
| 8 | Distillation's real signal is the soft label distribution, not just the teacher's top answer — that's what temperature exposes. |
| 9 | DPO replaces a reward model with a mathematical comparison to a frozen reference copy. That's the whole trick. |
| 10 | Don't shard until you've actually hit a memory wall. FSDP's communication cost isn't free. |
| 11 | NaN loss means numerical instability, not a mysterious bug. Check the learning rate and gradient clipping first. |
| 12 | The gap between train and validation loss is the most informative single chart in a training run. Look at it before anything else. |
15. Where To Go Next
The mechanics, in order
| Page | Covers |
|---|---|
| What Is Deep Learning? | Parameters, loss, gradient descent — the whole idea from zero |
| Neural Network Fundamentals | nn.Module, layers, activation functions |
| Training Loops & Optimization | Optimizers, learning rate schedules, gradient clipping |
| The Transformer Architecture | Attention, multi-head attention, positional encoding, building a block |
| Regularization & Generalization | Dropout, weight decay, normalization, overfitting |
| Embeddings & Representation Learning | How embedding spaces are actually trained |
Adapting and compressing a model
| Page | Covers |
|---|---|
| Fine-Tuning & Transfer Learning | Transfer learning, the LoRA derivation in full |
| Quantization From First Principles | The math behind every practical quantization format |
| Knowledge Distillation Mechanics | Soft targets, temperature, the distillation loss |
| Alignment & DPO Math | The Bradley-Terry model and the full DPO derivation |
Scaling and shipping it
| Page | Covers |
|---|---|
| Distributed Training Internals | DDP vs FSDP2 mechanics, mixed precision math |
| Evaluating & Debugging Training | Reading loss curves, diagnosing training bugs |
| Production & Operations | Checkpointing, reproducibility, experiment tracking |
| Failure Modes & Debugging | Symptom to cause, across every stage |
| Designing a Training System | A full worked design with real numbers |
| Glossary | Every term and abbreviation |
Then build something
Reading takes you only so far. Project Ideas lays out one from-scratch project that puts the whole mechanism — architecture, training loop, and evaluation — in your own hands.
Project Ideas
A starting brief for a production-shaped capstone — a monitored, cost-aware LLM gateway that ties the whole MLOps track together
What Is Deep Learning?
A neural network is a parameterized function fit to data by gradient descent — not something that understands. The idea from zero, with a worked example.