Fine-Tuning with PEFT
Full fine-tuning vs LoRA vs QLoRA — the mechanics, the memory math, and how to train an adapter without wrecking the base model
Fine-Tuning with PEFT
TL;DR
Full fine-tuning updates every weight in a model — expensive in memory and time, and usually more than the task needs. PEFT (Parameter-Efficient Fine-Tuning) freezes the base model and trains a small number of extra weights instead. LoRA is the default technique: it matches full fine-tuning quality on most tasks while training well under 1% of the parameters. QLoRA goes further, running LoRA on top of a 4-bit quantized frozen base so a model that needs tens of gigabytes in bfloat16 fits on a single consumer GPU.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | The Hub, Model Cards & Trust |
| You will understand | Why PEFT works, how LoRA and QLoRA differ, and how to configure and train an adapter without degrading the base model |
Why Not Just Fine-Tune Everything?
Full fine-tuning updates every parameter in the model, and that has a cost far beyond the weights themselves. For every trainable parameter, an optimizer like Adam keeps two more numbers (momentum and variance), and gradients need to be stored too. A model that needs 2 bytes/param just to sit in memory in bfloat16 needs roughly 4x that to actually train — weights, gradients, and two optimizer states, all at the same parameter count.
| Cost driver | Full fine-tuning | Why it matters |
|---|---|---|
| Optimizer state | One full copy per trainable parameter, twice over (Adam) | Dominates memory usage, not the weights themselves |
| Checkpoint size | A full copy of the model, per fine-tune | Every experiment costs a multi-gigabyte artifact |
| Catastrophic forgetting risk | Every weight can move, including ones the task didn't need to touch | General capability can degrade alongside the target skill |
| Hardware floor | Multiple GPUs for anything past a few billion parameters | Rules out a large share of teams and personal projects |
Most fine-tuning tasks don't need every weight to move. Teaching a model a narrower skill — a classification scheme, a response format, a domain vocabulary — is a much smaller adjustment than the full parameter space suggests. PEFT techniques are built on that observation: the update the task needs has far lower effective complexity than the model it's applied to.
Go deeper on the underlying models: Transformers & Pipelines.
LoRA: Training a Low-Rank Update Instead of the Model
LoRA (Low-Rank Adaptation) freezes every original weight matrix W and instead learns a small update to it, ΔW, decomposed as the product of two much smaller matrices:
W' = W + ΔW = W + B·A
W: frozen, shape (d_out, d_in)
A: trainable, shape (r, d_in)
B: trainable, shape (d_out, r)r — the rank — is small, typically 8 to 64, against a d_in/d_out that might be several thousand. The full update matrix ΔW would have d_out × d_in parameters; the low-rank factorization has only r × (d_in + d_out), often a few hundred times smaller. Because W never moves, there's no separate optimizer state to keep for it — only for A and B.
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 8,388,608 || all params: 27,xxx,xxx,xxx || trainable%: 0.03%The knobs that actually matter
| Parameter | What it controls | Practical guidance |
|---|---|---|
r (rank) | Capacity of the learned update | 8–16 for narrow tasks (format, style, a classification scheme); 32–64 for tasks needing more new "knowledge" |
lora_alpha | Scaling factor applied to the update (alpha / r is the effective multiplier) | A common default is alpha = 2 × r; raise it if the adapter's effect on outputs feels too weak |
target_modules | Which weight matrices get an adapter | At minimum the attention query/value projections (q_proj, v_proj); adding key/output projections and MLP layers increases capacity and memory |
lora_dropout | Regularization on the adapter path | A small value (0.05–0.1) helps on small datasets, where overfitting the adapter is the real risk |
More target_modules is not automatically better. Every module you add multiplies trainable parameters and gradient memory. Start with attention projections only; add MLP layers if evaluation shows the model still isn't picking up the target behavior, not as a default.
QLoRA: LoRA on a Quantized Base
QLoRA combines LoRA with a 4-bit quantized frozen base model (see Quantization & Optimization for the full picture of how that quantization works). The frozen weights sit in memory at roughly a quarter of their bfloat16 footprint; the trainable LoRA matrices stay in full precision.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="bfloat16",
)
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.8-27B", quantization_config=bnb_config, device_map="auto"
)
model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"]))A 27B-parameter model that needs roughly 54GB in bfloat16 fits in around 14GB at 4-bit — the difference between requiring a multi-GPU server and fitting on a single consumer GPU.
Why the quantization error doesn't compound during training. The frozen 4-bit weights never receive a gradient update — they stay exactly where quantization put them for the whole run. Only the full-precision A and B matrices train. This is different from quantizing a model after training it, where you're accepting quantization error on a model whose weights have already settled into their final, precision-sensitive positions.
LoftQ is an alternative initialization strategy for the LoRA matrices, designed to reduce the quality gap introduced by starting from a quantized base (replace_lora_weights_loftq — requires a safetensors checkpoint and 4-bit bitsandbytes quantization). It's worth reaching for when QLoRA quality noticeably trails a full-precision LoRA run on the same data.
Training the Adapter
Once wrapped with get_peft_model, the model is a normal PyTorch module and trains through the standard transformers.Trainer:
from transformers import TrainingArguments, Trainer
args = TrainingArguments(
output_dir="out",
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # effective batch size 16
gradient_checkpointing=True, # trade compute for activation memory
learning_rate=2e-4,
num_train_epochs=3,
bf16=True,
)
trainer = Trainer(model=model, args=args, train_dataset=train_ds)
trainer.train()| Setting | Typical value for LoRA | Why it differs from full fine-tuning |
|---|---|---|
| Learning rate | 1e-4 to 2e-4 | An order of magnitude higher than full fine-tuning's ~1e-5 to 2e-5 — the adapter has far fewer parameters to move, and moves them faster |
gradient_accumulation_steps | As needed to reach a usable effective batch size | Compensates for a small per_device_train_batch_size when memory is tight |
gradient_checkpointing | Usually on | Recomputes activations during the backward pass instead of storing them — trades compute time for a large memory saving |
| Epochs | Often 1–3 | More epochs on a narrow dataset increases the risk covered below |
For instruction-formatted data specifically, trl's SFTTrainer wraps this same loop with dataset formatting and packing conveniences built in — see Alignment with TRL for the full treatment; for a small, already-structured classification or extraction dataset, Trainer directly is often simpler.
Catastrophic forgetting is still possible with an adapter
LoRA trains far fewer parameters than full fine-tuning, but it can still overwrite a model's general ability if pushed too hard: too high a learning rate, too many epochs over a narrow or repetitive dataset, or a target_modules set that's larger than the task needs. Guard against it the same way you would for full fine-tuning — keep the learning rate and epoch count modest, and evaluate on a general-capability check alongside the target-task metric so a regression shows up before you ship the adapter.
Merging, or Keeping the Adapter Separate
merged_model = model.merge_and_unload() # bakes B·A into W, returns a plain modelShip the adapter, or merge it?
Keep the adapter separate
RecommendedThe adapter is tens of megabytes, versioned independently of the public base model. Swapping tasks means swapping a small file, not redownloading a multi-gigabyte checkpoint. This is the right default, and what most PEFT-based projects publish.
Merge with `merge_and_unload()`
Produces one self-contained checkpoint with no runtime dependency on peft or the separate base weights. Worth doing when you need a single artifact for a deployment target that doesn't know how to load adapters, or when serving through an engine (e.g. GGUF-based) that expects one merged file.
Concept Checks
Check yourself
Why does LoRA need no extra optimizer state for the frozen weight matrix W, while it does for A and B?
Optimizer state (Adam's momentum and variance) exists only for parameters that receive gradients and get updated. W never changes during LoRA training — it stays frozen — so there is nothing to track for it. A and B are the only parameters actually being learned, and they're a small fraction of W's size, which is exactly where LoRA's memory savings come from.
A LoRA fine-tune with r=64 across every attention and MLP layer is barely faster to train than full fine-tuning. What's the likely cause?
The target_modules set and rank are large enough that the trainable parameter count has climbed close to a meaningful fraction of the full model — PEFT's advantage comes from staying small, not from the technique itself. The fix is usually to prune target_modules back to attention projections and lower r, then re-check whether the task still needs the extra capacity.
Why doesn't QLoRA's 4-bit quantization error compound the way it would in a normally trained then post-hoc-quantized model?
Because the quantized weights are frozen for the entire training run — they never receive a gradient update, so there's no interaction between quantization noise and a moving target. Only the full-precision LoRA matrices train, adjusting around the fixed, already-quantized base rather than the base drifting under them.
Your fine-tune nails the target task but now answers unrelated general questions worse than the base model did. What happened, and what's the fix?
Catastrophic forgetting — the adapter moved the model further than the narrow task required, usually from too high a learning rate, too many epochs on repetitive data, or too large a target_modules set. The fix is to lower the learning rate or epoch count and re-evaluate on a general-capability check alongside the target metric, so the regression is caught before shipping rather than after.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| PEFT | Freeze the base model, train a small number of added parameters instead |
| LoRA | Learn a low-rank update B·A to a frozen weight matrix instead of updating it directly |
Rank (r) | The bottleneck dimension of the update — the main capacity/memory knob |
target_modules | Which weight matrices get an adapter — more modules means more capacity and more memory |
| QLoRA | LoRA on top of a 4-bit quantized frozen base — dramatically lower memory, small quality cost |
| LoftQ | An adapter initialization strategy that narrows the QLoRA-vs-full-precision-LoRA quality gap |
| Catastrophic forgetting | The model losing general ability by overfitting a narrow fine-tune |
| Merging | Baking the adapter into the base for a single self-contained checkpoint — at the cost of losing the small, swappable artifact |
Next
Fine-tuning made the model smaller to train by only touching a fraction of its weights. The next step is making it smaller to run: Quantization & Optimization.
Model Hub & Cards
Reading a model card like a practitioner, the full trust-signals table, revision pinning, and how Xet and safetensors work at the mechanism level
Quantization & Optimization
Why lowering numerical precision barely hurts quality, the format landscape (bnb, GGUF, AWQ/GPTQ, FP8), and matching the format to the serving engine