SLM Architectures
The architectural techniques — GQA, tied embeddings, sparse MoE, and long-context tricks — that let small models punch above their parameter count
SLM Architectures
TL;DR
Small models don't just have fewer parameters than large ones — they're built differently, with specific techniques that reclaim capability per parameter: Grouped Query Attention to shrink the memory-hungry KV cache, tied embeddings to recover capacity from an otherwise oversized embedding table, sparse Mixture-of-Experts to decouple stored size from per-token compute cost, and long-context tricks that make a usable context window affordable at small scale.
| Property | Value |
|---|---|
| Level | Beginner |
| Reading time | ~20 minutes |
| Prerequisites | What Is a Small Language Model? |
| You will understand | The specific architectural choices that make modern small models capable, and how to read a model card for them |
Why Architecture Matters More at Small Scale
At frontier scale, a model has parameters to spare — inefficiencies in how they're spent get absorbed by sheer size. A 1–4B model has no such slack. Every architectural choice that wastes capacity or memory is a choice the model pays for directly, in capability it doesn't have room to develop elsewhere. This is why small model families iterate on architecture as aggressively as they iterate on training data.
Four techniques show up repeatedly across the current generation of small models — SmolLM3, the Llama-3.2 1B/3B pair, the smaller Qwen3 variants — and each solves a specific budget problem.
Grouped Query Attention (GQA)
The problem it solves
During generation, a transformer keeps a KV cache — the key and value vectors for every token generated so far, so it doesn't have to recompute them on every step. For a model with many attention heads, that cache grows fast, and it's held in memory for the entire length of the conversation.
On a data-center GPU with generous memory, a large KV cache is an inconvenience. On a phone, or in a browser tab, it can be the difference between "runs" and "doesn't fit."
How it works
Standard multi-head attention gives every query head its own key and value projections. Grouped Query Attention shares key/value projections across groups of query heads instead of giving each one its own.
Attention variants, KV cache size
Multi-Head Attention (MHA)
Grouped Query Attention (GQA)
Multi-Query Attention (MQA)
GQA sits deliberately between the two extremes: most of MQA's memory savings, with much less of its quality cost. It's the choice most current small instruct models make — Llama-3.2 (1B and 3B), SmolLM3, and the smaller Qwen3 models all use GQA specifically because the KV cache matters disproportionately more when the whole model is meant to fit in a constrained memory budget.
KV cache size scales with context length, not just model size. A small model with a long supported context can still hit a memory wall from the KV cache alone, even though its weights are tiny. This is exactly why GQA matters more for small, long-context models than it does for a large model with a short context — and why SLM architecture pages and LLM architecture pages don't emphasize it equally.
Tied Embeddings
The problem it solves
A transformer has two embedding-related matrices: one that turns input tokens into vectors, and one that turns the model's final hidden state back into a probability over the vocabulary (the output/unembedding layer). For a large vocabulary (often 50,000–150,000+ tokens) and a wide hidden dimension, that's two large matrices.
In a frontier model with tens of billions of parameters, the embedding tables are a small slice of the total. In a 1B model, they can be a genuinely large fraction of everything the model has.
How it works
Weight tying makes the input embedding matrix and the output projection matrix literally the same weights, used in both directions. Instead of paying for two large matrices, you pay for one, and the parameters freed up go toward the rest of the network — attention and feed-forward layers, where they do more to improve reasoning and instruction-following than a duplicated embedding table would.
Tied embeddings are a small-model-specific trade, not a free win everywhere. At large scale, untied embeddings can slightly help quality because the model has spare capacity to afford them. At small scale, that spare capacity doesn't exist — reclaiming it via tying is usually the better trade.
Sparse Mixture-of-Experts (MoE): Decoupling Size from Cost
The core idea
A dense model uses every parameter on every token. A sparse MoE model instead has many parallel "expert" sub-networks, and a small router picks only a few of them to actually run for each token. The model can be enormous on disk while being cheap per token, because most of it sits idle for any given piece of input.
This is described with two different numbers, and mixing them up is the single most common mistake when sizing an MoE model:
| Number | What it means |
|---|---|
| Total parameters | Everything stored — every expert, whether or not it's used for a given token |
| Active parameters | What actually computes for a given token — the router's chosen experts plus the shared layers |
A real, worked example
Qwen/Qwen3-Coder-30B-A3B-Instruct names both numbers directly: 30B total parameters, A3B — roughly 3B active parameters per token. Its memory footprint on disk looks like a 30B dense model. Its per-token compute cost looks like a 3B dense model.
Deployment cost tracks active parameters, not total. If you're comparing an MoE model against a dense one for latency or compute budget, compare its active parameter count against the dense model's total — comparing the two "total" numbers will make the MoE model look far more expensive than it actually is to run. Storage/download size is the one place total parameters is still the right number to look at.
When MoE makes sense for a small deployment
Plenty of storage, tight per-token compute
RecommendedA large on-disk model with cheap active compute is a good fit when the model can be downloaded once (or bundled with an app) and storage isn't the bottleneck.
Storage or download size is the tight constraint
A dense model at the active parameter size will simply be smaller on disk — for a browser or a constrained mobile install size, that can matter more than the compute savings MoE offers.
Long Context at Small Scale
Attention cost, in principle, scales the same way regardless of model size. In practice, long context is a harder budget line for a small model than for a large one, for two compounding reasons: the KV cache (see GQA above) competes for the same tight memory budget as everything else, and a smaller model has less raw capacity to spend on attending reliably across a very long input.
Two techniques address this directly:
| Technique | Idea |
|---|---|
| RoPE scaling | Adjusts the rotary position encoding so a model trained at one context length can extrapolate to a longer one at inference time, with some quality trade-off |
| NoPE (no explicit positional encoding) | Removes explicit position encoding in favor of letting the model's own structure (e.g. causal masking) carry positional information — SmolLM3 takes this approach specifically to improve long-context behavior |
SmolLM3 pairs GQA (to keep the KV cache small) with a NoPE-style approach (to improve long-context reliability) — the two techniques target different bottlenecks of the same underlying problem: making a long context genuinely usable within a small model's tight memory and capacity budget, not just technically supported.
Reading a Model Card for This
When evaluating a small model for a memory- or latency-constrained deployment, four things on the model card matter more than the headline parameter count:
| Check | Why |
|---|---|
| Attention type (GQA/MQA/MHA) | Directly predicts KV cache size and memory behavior at your target context length |
| Total vs active parameters | If it's MoE, active parameters is what predicts actual compute cost |
| Tied vs untied embeddings | A hint at how much of the parameter budget went to the embedding table vs. the rest of the network |
| Stated context length and how it's achieved | RoPE-scaled and natively-trained-long context behave differently under real long-context load |
Concept Checks
Check yourself
Why does GQA matter more for a phone-deployed model than for a model served on a data-center GPU?
The KV cache grows with context length and competes for the same tight memory budget as everything else on a phone or in a browser tab. A data-center GPU has enough memory headroom that a larger KV cache is merely wasteful; a phone can run out of memory entirely. GQA shrinks that cache specifically, which matters far more when memory is the binding constraint.
Qwen3-Coder-30B-A3B-Instruct is described as '30B' with 'A3B.' Which number predicts its per-token latency, and which predicts its download size?
Active parameters (~3B) predicts per-token compute cost and therefore latency — that's what actually runs for any given token. Total parameters (30B) predicts download and on-disk storage size, since every expert has to be stored even though most sit idle for any single token. Confusing the two makes the model look either far cheaper or far more expensive to run than it actually is.
Why are tied embeddings a more attractive trade for a 1B model than for a 70B model?
The embedding table's size is roughly fixed by vocabulary size and hidden dimension, regardless of overall model size — so it's a much larger fraction of a 1B model's total budget than of a 70B model's. Tying frees up that duplicated capacity for the rest of the network, where a small model needs it far more urgently than a large model with parameters to spare does.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| GQA | Shares K/V projections across query-head groups to shrink the KV cache |
| KV cache | Memory held for every generated token's key/value vectors — grows with context length |
| Tied embeddings | Input and output embedding matrices share weights, recovering capacity for small models |
| Sparse MoE | Many experts stored, only a few active per token — decouples size from per-token cost |
| Total vs active parameters | Storage cost vs. actual compute cost — always check both on an MoE model |
| RoPE scaling | Extrapolates a trained context length to a longer one at inference, with a quality trade-off |
| NoPE | Omits explicit position encoding to improve long-context reliability — used by SmolLM3 |
Next
Architecture sets the ceiling; training data determines how much of it gets used. Training Data & Curricula.
What Is a Small Language Model?
Why "small" is a deliberate engineering constraint, not a weaker version of a big model, and how to decide when it's the right call
Training Data & Curricula
Why data quality matters more as models shrink, the multi-stage curriculum pattern, and why small models are trained past the compute-optimal point on purpose