What Is Deep Learning?
A neural network is a parameterized function fit to data by gradient descent — not something that understands. The idea from zero, with a worked example.
What Is Deep Learning?
TL;DR
A neural network is a function with learnable parameters. Training means adjusting those parameters, repeatedly, so the function's output gets closer to what you want, measured by a loss function and improved via gradient descent. "Deep" means the function is built from many stacked layers — depth adds representational capacity that width alone doesn't.
| Property | Value |
|---|---|
| Level | Beginner — start here |
| Reading time | ~18 minutes |
| Prerequisites | None. Basic algebra is enough — every idea is built from the ground up. |
| You will understand | What a neural network actually is, mechanically, before any of the complexity of real architectures |
A Neural Network Is a Function With Knobs
Start as small as possible. A linear function:
y = w * x + bw and b are parameters. If you don't know the right values, you can learn them from examples — pairs of (x, y) you already know are correct — by adjusting w and b until the function's predictions get close to the real y values.
A neural network is the same idea, scaled up enormously: instead of one w and one b, there might be billions of parameters, arranged into layers, computing a far more complex function. But the core mechanism — adjust parameters to reduce error on known examples — is identical.
A network doesn't "understand" anything — it fits a function to data. Every capability that looks like understanding (translation, summarization, reasoning) emerges from a huge number of small, mechanical parameter adjustments, repeated over a huge amount of data. This isn't a philosophical point — it's a practical one: when a model behaves oddly, the productive question is "what in the training data or loss function produced this," not "what was it thinking."
The Four Pieces
| Piece | What it does | In the tiny example above |
|---|---|---|
| Parameters | The numbers that get learned | w and b |
| Forward pass | Compute a prediction from an input | Evaluate w * x + b for a given x |
| Loss function | A single number measuring how wrong the prediction was | e.g. (prediction - true_y)^2 |
| Gradient descent | Adjust every parameter in the direction that reduces the loss | Nudge w and b based on how much each affected the error |
The training loop, at a conceptual level
The mechanics of exactly how the gradient is computed (backpropagation) and how the update is applied (an optimizer) are covered in full in Training Loops & Optimization — this page is deliberately staying at the conceptual level first, so the shape of the idea is solid before the mechanics pile on.
A Worked Example: Fitting a Line by Hand
Suppose the true relationship is y = 2x, and you're given three examples: (1, 2), (2, 4), (3, 6). Start with a bad guess: w = 0.5, b = 0.
Step 0: w = 0.5, b = 0
x=1: prediction = 0.5, true = 2, error = -1.5
x=2: prediction = 1.0, true = 4, error = -3.0
x=3: prediction = 1.5, true = 6, error = -4.5
Loss (mean squared error) = mean((-1.5)^2, (-3.0)^2, (-4.5)^2) ≈ 11.5Gradient descent computes how much the loss would change if w moved slightly, and moves w in the direction that reduces the loss — here, that means increasing w, since every prediction was too low.
Step 1: w = 0.5 + (learning_rate * gradient) ≈ 1.4 (rough, illustrative)
x=1: prediction = 1.4, true = 2, error = -0.6
x=2: prediction = 2.8, true = 4, error = -1.2
x=3: prediction = 4.2, true = 6, error = -1.8
Loss ≈ 1.8 ← already much lower than 11.5Repeat this enough times and w converges toward 2.0, b toward 0 — the network has "learned" the relationship, purely by reducing a number, step after step. Every neural network you'll ever train does exactly this, just with vastly more parameters and a vastly more complex function standing in for w * x + b.
Loss Functions: Turning "Wrong" Into a Number
Different tasks need different ways of measuring "how wrong":
| Task | Common loss | What it measures |
|---|---|---|
| Regression (predict a number) | Mean Squared Error (MSE) | Average of (prediction - true)^2 — penalizes large errors disproportionately |
| Classification (predict a category) | Cross-entropy | How far the model's predicted probability distribution is from putting all its probability on the correct class |
import torch.nn.functional as F
mse = F.mse_loss(predictions, targets) # regression
ce = F.cross_entropy(class_logits, true_class_indices) # classificationThe loss function defines what "good" means for a network. Everything downstream — every training run, every optimizer step — is just machinery for reducing whatever number the loss function reports. Choosing the wrong loss for a task (e.g. MSE for a classification problem) will still technically train, but toward the wrong notion of correctness.
Why "Deep" Matters
A network with one layer can only represent relatively simple functions — a straight line, or a simple curve, depending on the activation used. Stacking layers lets the network build up more complex functions as compositions of simpler ones, the same way a complex sentence is built from simpler grammatical pieces.
Depth vs width
A wider single layer
More units in one layer increases capacity somewhat, but a single layer's functional form is fundamentally limited regardless of how wide it gets.
More layers (depth)
RecommendedEach additional layer can build on the representations the previous layer learned — early layers might learn simple patterns, later layers combine them into increasingly abstract ones. This compounding is why depth, not just width, is what unlocked the complex capabilities of modern networks.
There's a catch, covered in full in Neural Network Fundamentals: stacking layers only adds real capacity if there's a nonlinearity between them. Without one, any number of stacked linear layers collapses mathematically into a single linear layer.
Deep Learning vs Classical Machine Learning
What actually changed
Classical ML
A human engineer decides what features matter (e.g. "word count," "average sentence length") and hand-builds them from raw data. The model then learns to weigh those hand-chosen features.
Deep learning
RecommendedThe network learns the features themselves, directly from raw or minimally-processed data (pixels, tokens), as part of the same training process that learns to use them. This is what "representation learning" means, covered in depth in Embeddings & Representation Learning.
This is the single biggest shift deep learning represents: not a new kind of math, but removing the need for a human to decide in advance what matters about the input.
Tensors: The Basic Data Structure
Everything in deep learning — inputs, parameters, activations, gradients — is stored as a tensor: a multi-dimensional array, generalizing scalars (0-D), vectors (1-D), and matrices (2-D) to any number of dimensions.
import torch
scalar = torch.tensor(3.0) # shape: ()
vector = torch.tensor([1.0, 2.0, 3.0]) # shape: (3,)
matrix = torch.zeros(4, 3) # shape: (4, 3)
batch = torch.zeros(32, 4, 3) # shape: (32, 4, 3) — a batch of 32 matricesGPUs are fast at deep learning specifically because its core operation — matrix multiplication — is embarrassingly parallel. Every output element of a matrix multiplication can be computed independently of every other, which is exactly the kind of workload a GPU's thousands of small cores are built for, unlike the mostly-sequential work a CPU is optimized for.
Concept Checks
Check yourself
Why is 'the model doesn't understand, it fits a function' more than a philosophical point?
Because it changes how you debug. If a model produces a wrong or strange output, the productive question is what in the training data, loss function, or architecture produced that behavior mechanically — not what the model was "thinking." Treating outputs as the result of a fitting process, not a reasoning process, points debugging effort at the actual levers that can fix it.
In the worked linear-fitting example, why did every prediction being too low tell you which direction to move w?
Because the gradient of the loss with respect to w indicates which direction reduces the error — if every prediction undershoots the true value, increasing w moves every prediction closer to correct, which is exactly the direction that reduces the loss. Gradient descent formalizes this same reasoning for any function, however complex, not just a single-parameter line.
Why doesn't stacking more layers automatically increase a network's representational capacity?
Because without a nonlinearity between layers, composing linear transformations produces another linear transformation — mathematically no different from a single layer, regardless of how many were stacked. Depth only adds real capacity when a nonlinear activation function sits between the layers, which is covered in full on the next page.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Neural network | A function with learnable parameters, fit to data |
| Parameters | The numbers adjusted during training |
| Forward pass | Computing a prediction from an input |
| Loss function | A single number measuring how wrong a prediction was |
| Gradient descent | Adjusting parameters in the direction that reduces the loss |
| MSE / cross-entropy | The standard losses for regression and classification respectively |
| Depth | Stacked layers compounding simpler representations into more complex ones |
| Representation learning | Learning features directly from data, instead of hand-engineering them |
| Tensor | The multi-dimensional array underlying every value in deep learning |
Next
With the core idea in place, the next page covers how a real network is actually built in code: Neural Network Fundamentals.
Deep Learning Crash Course
All of deep learning on one page — neural networks, training loops, the transformer, fine-tuning, compression, alignment, and distributed training, from first principles
Neural Network Fundamentals
nn.Module, the layers that make up a network, and why a nonlinearity between them is what makes depth mean anything