Small Language Models Fundamentals Glossary Every term and abbreviation used across the Small Language Models track, expanded
TL;DR
Every term used across this track, in one place, grouped by topic. Use this as a reference while reading, not something to memorize up front.
Property Value Level Reference Reading time ~10 minutes, or look up as needed Prerequisites Designing an SLM-Based System You will understand Where to look up any term from this track
Term Meaning SLM Small Language Model — roughly sub-10B parameters, built to fit a hard deployment constraint rather than defined by an exact cutoff Active vs total parameters For sparse MoE models: what actually computes per token (active), vs everything stored on disk (total) GQA (Grouped Query Attention) Shares key/value projections across groups of query heads, shrinking the KV cache MQA (Multi-Query Attention) The more extreme version of GQA — all query heads share one set of KV projections Tied embeddings Input and output embedding matrices share weights, recovering capacity in small models Sparse MoE (Mixture-of-Experts) Many "expert" sub-networks exist; only a few activate per token, decoupling total size from per-token cost NoPE A positional-encoding approach that omits standard RoPE, used by some models (e.g. SmolLM3) to improve long-context behavior RoPE scaling Techniques for extending a model's usable context length beyond what it was originally trained on KV cache Stored key/value activations from previous tokens, reused during generation — a major memory cost for long contexts
Term Meaning Chinchilla-optimal The training-compute-minimizing ratio of data to parameters for a target training loss — not the inference-cost-minimizing ratio Overtraining (for inference) Training past the Chinchilla-optimal point to get a smaller, cheaper-to-serve final model, accepting less training-compute efficiency Synthetic data Training examples generated (or filtered) by another model, often a larger teacher Distillation Training a smaller student model to imitate a larger teacher model Teacher / student The larger source model and the smaller model being trained to imitate it Black-box distillation Training the student on the teacher's generated text as ordinary supervised data Logit-matching / white-box distillation Training the student to match the teacher's full output distribution, not just its top answer On-policy distillation The student generates its own rollouts, which the teacher then scores or corrects Off-policy distillation Training on static, pre-generated teacher text, regardless of what the student itself would generate GKD (Generalized Knowledge Distillation) TRL's on-policy distillation trainer MiniLLM A reverse-KL-divergence, RL-optimized on-policy distillation method Catastrophic forgetting Fine-tuning degrading a model's general ability — more pronounced in small models with less spare capacity IA3 A PEFT method that rescales activations with learned vectors instead of adding weight deltas TinyLoRA A LoRA variant scaling adapters down to as little as one trainable parameter BEFT (Bias-Efficient Fine-Tuning) Tuning only a bias term (often b_v) for parameter-minimal adaptation MiSS (Matrix Shard Sharing) A single-trainable-matrix PEFT method distinct from LoRA's decomposition
Term Meaning GGUF The quantized model file format used by llama.cpp and compatible runtimes Q8_0 / Q5_K_M / Q4_K_M Common GGUF quantization levels, in decreasing bit-width and increasing compression K-quants GGUF quantization types that use different bit-widths per tensor based on sensitivity AWQ / GPTQ Calibrated post-training quantization formats, 3-4 bit, oriented toward GPU serving (vLLM, TGI) bitsandbytes Load-time (no calibration) 8-bit/4-bit quantization, commonly used during training/experimentation rather than final edge deployment BitNet An architecture using ternary (1) weights via BitLinear layers QAT (Quantization-Aware Training) Training with quantization built in from the start — required for BitNet, since it can't be applied after the fact Core ML palettization Apple's k-means-based weight clustering quantization approach Memory-mapping Loading a model by paging it in from disk on demand, rather than reading the whole file into RAM up front — key to GGUF's fast cold starts
Term Meaning llama.cpp A C/C++ inference engine for GGUF models, no Python/CUDA required WebGPU The browser API enabling GPU-accelerated, client-side model inference transformers.js HuggingFace's JavaScript library for running models in the browser or Node, with WebGPU support ONNX Runtime Web The engine transformers.js's WebGPU acceleration is built on MLC-LLM A cross-platform compiler/runtime for on-device model deployment NNAPI Android's native neural network acceleration API vLLM / TGI GPU server-side serving engines optimized for high-throughput API deployment On-device deployment Running inference entirely on the user's own device, with no server call OTA (over-the-air) model update Shipping a new model file to deployed devices — should be versioned and rolled out like a code deploy
Term Meaning Speculative decoding A small draft model proposes tokens a large target model verifies in one pass Draft model The small, fast model proposing candidate tokens Target model The large model whose exact output is being produced, faster Assisted generation transformers' name for speculative decoding, via assistant_model=Universal assisted generation Speculative decoding with mismatched draft/target tokenizers, via lookbehind re-encoding Medusa Extra prediction heads added to the target model itself, avoiding a separate draft model N-gram speculation Speculating from pattern matches against existing text, with no model involved
Term Meaning Constrained / structured decoding Restricting generation to only tokens that keep output on a valid path through a schema Tool-call validity rate The fraction of tool calls that are well-formed and correctly targeted — tracked as its own metric CodeAgent A smolagents pattern where the model writes and executes Python to take actions ToolCallingAgent A smolagents pattern using classic JSON tool calling — generally more reliable for small models Escalation / routing Sending requests a small model can't confidently handle to a larger model or a human xsmall embedding model A very small (tens-of-MB) embedding model, suited to fully local RAG Context budget How much retrieved text is handed to the generator — tighter for small models than large ones Leaderboard trap Trusting a general benchmark score over task-specific evaluation on your own data
You've covered the whole track. Project Ideas Small Language Models Crash Course .