Project Ideas
A starting brief for a from-scratch capstone — building, training, and evaluating a small transformer language model with your own hands
Project Ideas
This is a brief, not a build guide
Like the HuggingFace and Small Language Models briefs, this page sets up one well-scoped project idea and leaves the building to you. Everything you need conceptually is in the Fundamentals track; this is where you'd point it.
The idea: build and train a small transformer language model, entirely from scratch
Why this, specifically: every page in this track explains a mechanism — attention, backpropagation, LoRA, quantization — in isolation. The single project that proves you actually understand how they fit together is building the thing they all describe: a working transformer, with your own nn.Module code, your own training loop, trained on real data, evaluated honestly. This is deliberately the opposite exercise from the HuggingFace track's projects, which start from a pretrained model — here, nothing is pretrained until you've trained it yourself.
The whole thing, in your own code
Why this shape, specifically
Design choices worth making deliberately
Pick a narrow, focused corpus, not 'the internet'
RecommendedA small model trained on a focused domain (a specific author's works, a programming language's documentation, a niche Wikipedia category) will produce noticeably coherent, on-topic output with a fraction of the data and compute a general-purpose corpus would need — and the point of this project is seeing the mechanism work, not building a competitive general model.
Write the training loop yourself, don't reach for `Trainer`
RecommendedThe value of this project is specifically in the parts a library would otherwise hide — watching the loss curve behave (or misbehave) in response to a learning rate change you made, debugging a NaN loss yourself, and internalizing what's actually happening every training step teaches something a library call can't.
Keep the model genuinely small
A few million to a few tens of millions of parameters is enough to see real language modeling behavior emerge, trains in a reasonable time on a single GPU (or even a strong CPU, slowly), and keeps every iteration cycle fast enough to actually debug.
Suggested stack
| Layer | Choice | Role |
|---|---|---|
| Framework | Raw PyTorch (nn.Module, torch.optim) | No training-loop abstraction — that's the point |
| Tokenizer | A simple BPE implementation, or tokenizers' trainable BPE | Understand what a tokenizer actually does to your corpus |
| Architecture | A small decoder-only transformer, from The Transformer Architecture | Attention, positional encoding, a handful of blocks |
| Training | Your own loop, AdamW, a learning rate schedule | The mechanics from Training Loops & Optimization |
| Evaluation | Held-out perplexity + qualitative generation | Both a number and a sanity check by reading actual output |
Stretch directions, if the base project goes well
- Add LoRA fine-tuning (Fine-Tuning & Transfer Learning) on top of your own trained model, implementing the low-rank update yourself rather than using
peft, to feel exactly what it freezes and what it trains. - Quantize your own trained model (Quantization From First Principles) with a hand-written int8 quantizer, and measure the real accuracy/perplexity cost on your own held-out set.
- Use your finished small model as the draft model for speculative decoding against a larger pretrained model, connecting directly to the Small Language Models track — a genuinely useful outcome from a from-scratch training exercise.
Before you start
Work through What Is Deep Learning? through The Transformer Architecture at minimum — the rest of the fundamentals track fills in as you hit each stage above. If a step above uses a term or technique you haven't seen yet, that's the signal for which fundamentals page to read next, not a sign you're missing something.