Fine-Tuning & Transfer Learning
Why transfer learning works, how layer freezing works mechanically, and the full derivation of LoRA's low-rank update
Fine-Tuning & Transfer Learning
TL;DR
A model trained on a large, general task has already learned reusable features — adapting it to a narrower task needs far less data and compute than training from scratch, because most of the hard work is already done. LoRA takes this further: instead of updating every weight, it learns a small low-rank update on top of a frozen base, because the update a specific task actually needs tends to live in a much lower-dimensional space than the full weight matrix.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~19 minutes |
| Prerequisites | Embeddings & Representation Learning |
| You will understand | Why transfer learning works, how to freeze layers deliberately, and the full math behind LoRA |
Why Transfer Learning Works
A model trained on a large, general dataset — millions of images, billions of tokens of text — has learned features useful far beyond its original training objective. Early layers in a vision model tend to learn edge detectors and simple textures; early layers in a language model tend to learn syntax and common word patterns. These are useful for almost any downstream task in that domain, not just the one the model was originally trained on.
Transfer learning is reusing that already-learned general capability instead of re-deriving it from a small task-specific dataset. Training a model from scratch on 500 labeled examples usually fails outright — there isn't enough signal to learn good general features and the specific task at once. Starting from a model that already has the general features and only needs to adapt them is a completely different, far more tractable problem.
Freezing Layers, Mechanically
The simplest form of transfer learning: keep most of the pretrained model exactly as it is, and only let a small part of it update.
for name, param in model.named_parameters():
if "encoder.layer" in name and get_layer_index(name) < 8:
param.requires_grad = False # frozen — no gradient computed, no update
optimizer = torch.optim.AdamW(
filter(lambda p: p.requires_grad, model.parameters()), lr=2e-5
)Setting requires_grad = False tells PyTorch's autodiff engine not to bother computing a gradient for that parameter at all — it's excluded from backpropagation and from anything the optimizer touches. The typical pattern is freezing early layers (general features, worth preserving exactly as trained) while leaving later layers, or a new task-specific head, trainable (task-specific features, worth adapting).
Freezing too little defeats the point of transfer learning; freezing too much starves the model of the capacity it needs to adapt. There's no universal answer for how many layers to freeze — it depends on how similar the new task is to the original training objective. A very similar task can get away with freezing almost everything; a substantially different one needs more of the network free to adapt.
Full Fine-Tuning vs LoRA
Full fine-tuning leaves every parameter trainable. At any real scale, this means storing a full optimizer state (for Adam, roughly two extra numbers per parameter) for the entire model, even though the actual behavioral change needed for one task is usually much smaller than the model's total capacity.
The low-rank hypothesis
The key empirical claim LoRA is built on
Even though a model's full weight matrix W has high rank (it encodes everything the model knows), the update ΔW needed to adapt it to a specific downstream task tends to have a much lower "intrinsic rank" — the useful change can be well-approximated by a matrix with far fewer independent directions than W itself has.
If that's true, there's no need to parameterize ΔW as a full, dense d × d matrix at all — a low-rank factorization can capture nearly all of the useful update with a tiny fraction of the parameters.
The parameterization
ΔW = BA
W: d × d (frozen — the original pretrained weight matrix)
B: d × r (trained — initialized to all zeros)
A: r × d (trained — initialized randomly, e.g. small Gaussian noise)
r ≪ d (the rank — a hyperparameter, typically 4 to 64)class LoRALinear(nn.Module):
def __init__(self, base_linear, rank, alpha):
super().__init__()
self.base = base_linear
for p in self.base.parameters():
p.requires_grad = False
d_in, d_out = base_linear.in_features, base_linear.out_features
self.A = nn.Parameter(torch.randn(rank, d_in) * 0.01)
self.B = nn.Parameter(torch.zeros(d_out, rank))
self.scaling = alpha / rank
def forward(self, x):
return self.base(x) + (x @ self.A.T @ self.B.T) * self.scalingWhy B starts at zero
B is initialized to all zeros so that ΔW = BA is exactly zero at the start of training. This means the LoRA-augmented model produces identical output to the unmodified base model on step one — training starts from the known-good pretrained behavior and gradually introduces the adaptation, rather than starting from a random perturbation that could initially make things worse. A still needs random (not zero) initialization, or the product BA would stay stuck at zero forever — its gradient would have nothing to push against.
What the scaling factor does
The forward pass is h = Wx + (α/r)·BAx. The α/r scaling factor lets you control the magnitude of the LoRA update somewhat independently of the rank r you chose — without it, changing r would also silently change how strongly the adaptation affects the output, making rank and effective learning rate tangled together in a way that's awkward to tune. In practice, α is often set equal to r or to a small multiple of it.
The parameter savings, concretely
For a 4096 × 4096 weight matrix (a realistic attention projection size) with r = 8:
| Parameters | Relative to full | |
|---|---|---|
Full weight matrix W | 4096 × 4096 = 16,777,216 | 100% |
LoRA B + A, r=8 | (4096×8) + (8×4096) = 65,536 | ~0.39% |
This is the mechanism the practical peft library's LoraConfig implements — the r, lora_alpha, and target_modules arguments used in the HuggingFace fine-tuning page map directly onto r, α, and which weight matrices get a B/A pair attached, exactly as derived here.
Concept Checks
Check yourself
Why does transfer learning typically need far less data than training from scratch?
Because the pretrained model has already learned general, reusable features from a much larger dataset than the downstream task provides — early layers in particular tend to capture broadly useful patterns (edges and textures in vision, syntax and common structures in language). Fine-tuning only has to adapt or add task-specific behavior on top of that, a much smaller learning problem than deriving both general and specific features from a small dataset at once.
Why is B initialized to zero in LoRA, but A is initialized randomly rather than also to zero?
B is zeroed so that ΔW = BA is exactly zero at the start of training, meaning the model's initial behavior exactly matches the unmodified pretrained model rather than an arbitrary random perturbation. A still needs random initialization because if both were zero, the product BA would have no gradient signal to move it away from zero — training would never start moving the adapter at all.
What does the α/r scaling factor actually control, separately from the rank r itself?
It controls how strongly the LoRA update affects the model's output, independent of how many parameters (rank r) the adapter has. Without this separate scaling, increasing r would also silently increase the update's effective magnitude, entangling "how much capacity the adapter has" with "how strongly it's allowed to change the output" — two things you usually want to tune somewhat independently.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Transfer learning | Reusing a pretrained model's general features instead of relearning them from a small dataset |
| Layer freezing | requires_grad = False excludes a parameter from backpropagation and updates entirely |
| Low-rank hypothesis | A task-specific weight update tends to have much lower intrinsic rank than the full weight matrix |
| ΔW = BA | LoRA's low-rank factorization of the weight update |
| B initialized to zero | Ensures training starts from the exact pretrained behavior, not a random perturbation |
| α/r scaling | Controls update magnitude independently of rank |
| Parameter savings | A rank-8 LoRA adapter on a 4096×4096 matrix uses well under 1% of the full parameter count |
Next
Adaptation handled — the next page covers shrinking the resulting model for deployment, starting from the actual math of quantization: Quantization From First Principles.
Embeddings & Representation Learning
What an embedding actually is mechanically, why training makes embedding spaces semantically structured, and how contrastive learning builds them on purpose
Quantization From First Principles
The exact math behind mapping float weights to integers, worked numerically, plus symmetric vs asymmetric, per-tensor vs per-channel, and the straight-through estimator behind QAT