Designing an SLM-Based System
One complete worked example — an offline-first field-service app — walked through every layer of the SLM stack, with concrete numbers
Designing an SLM-Based System
TL;DR
This page is one worked design, not a generic checklist: a field-service mobile app that must work with no reliable connectivity on-site, answer questions against a local procedure manual, and occasionally escalate genuinely hard questions once the technician is back online. Every decision below follows from those three requirements.
| Property | Value |
|---|---|
| Level | Advanced |
| Reading time | ~20 minutes |
| Prerequisites | Failure Modes & Debugging |
| You will understand | How the whole track's decisions fit together in one real, concrete design |
The Requirements
A field-service company sends technicians to sites with unreliable or no connectivity — basements, rural installations, shielded equipment rooms. Technicians need to ask procedural questions ("what's the torque spec for this fitting," "what's the safe shutdown sequence for this model") against a large, frequently-updated procedure manual, and get an answer in seconds, without waiting for a signal.
| Requirement | Implication |
|---|---|
| Must work fully offline | On-device deployment, not an API call — no fallback to "just retry when you have signal" |
| Runs on company-issued mid-range phones | A real, tight RAM/CPU budget — not the newest flagship |
| Procedure manual changes monthly | The knowledge needs to update independently of the model itself |
| Some questions are genuinely hard (edge-case equipment, ambiguous symptoms) | Needs an escalation path for once connectivity returns |
Model Selection
Sizing the model to the constraint
A 1-3B instruct model
RecommendedSmall enough to run comfortably in the RAM budget of a mid-range company phone once quantized, fast enough for a response in a few seconds on CPU alone, and — critically for this task — doesn't need to carry deep world knowledge, since the procedure manual is supplied via retrieval, not memorized.
A frontier hosted model
Fails the core requirement outright — it needs connectivity, which is exactly what this app can't assume.
A model in the Llama-3.2 1B/3B-Instruct class, or a similarly-sized instruct model with GQA (see SLM Architectures) to keep the KV cache light on a phone-class CPU, fits this brief well.
Fine-Tuning Approach
The base instruct model already follows instructions reasonably; what it doesn't know is this company's specific procedure format and terminology. A modest full fine-tune (feasible at this size on a single GPU, per Fine-Tuning Small Models) on a few thousand real technician Q&A pairs teaches the model the house terminology and response format — short, structured, citation-style answers rather than long prose — without needing it to memorize the manual's actual content, which changes monthly and belongs in retrieval instead.
Watch for catastrophic forgetting here specifically: a narrow fine-tune on procedural Q&A risks degrading the model's general instruction-following on the (less common but real) open-ended questions technicians also ask. Mix in some general-purpose instruction data during fine-tuning and eval on both the target task and general capability.
Quantization Target
Two device tiers exist in the fleet — this app needs two shipped artifacts, not one:
| Tier | Quantization | Reasoning |
|---|---|---|
| Older/lower-RAM company phones | GGUF Q4_K_M | Fits the tighter RAM budget; the accuracy loss vs Q8_0 was confirmed acceptable via the re-scoring discipline from Evaluation for SLMs |
| Newer phones in the fleet | GGUF Q5_K_M or Q8_0 | More RAM headroom available; ship the higher-quality tier where the device can afford it |
Runtime Choice
llama.cpp on-device, wrapped in the mobile app via its GGUF support — chosen over a WebGPU/browser approach because this is a native company-issued app, not a web tool, and over MLC/Core ML/NNAPI's heavier per-platform build step because the company fleet is predominantly one platform and llama.cpp's single-file-plus-runtime simplicity matters more here than squeezing out the last bit of platform-native performance. See Inference Engines & Runtimes for the full decision framework this follows.
Local RAG Design
The procedure manual — the part that changes monthly — is handled entirely through retrieval, not baked into the fine-tune:
On-device retrieval for this app
This is exactly the separation of concerns from SLM RAG: the model carries how to answer, retrieval carries what's currently true — which is also what makes monthly manual updates cheap (re-index, don't retrain).
Evaluation Plan
A held-out set of real technician questions, covering both easy single-lookup questions and harder multi-step procedures, scored on the actual shipped artifact per tier:
Held-out eval, per quantization tier:
- Task accuracy against manual-grounded correct answers
- Instruction-following on multi-step procedures specifically
- Retrieval quality (was the right manual section retrieved at all)
- Re-run after every monthly manual re-index, and after every model updateThe Escalation Path
What happens when the small model isn't confident
Small model attempts the answer, grounded in retrieval
The default path for the large majority of questions
Low retrieval confidence or explicit 'not in the manual'
The model is trained to say so rather than guess
Queue the question locally
No connectivity yet — store it, don't block the technician
Sync and escalate once online
Routes to a larger hosted model or a human expert; the answer syncs back to the technician's device
Monitoring and Versioning Checklist
- Manual re-indexing is a data update — ship it independently of the model, with its own version number.
- Model updates (fine-tune revisions) follow the staged-rollout-plus-rollback discipline from Production & Operations — a bad fine-tune reaching every technician's phone at once is a real operational risk in a field-service context.
- Escalation rate is tracked explicitly — a rising rate signals either genuinely harder questions coming in, or a quality regression in the model or the retrieval index, and is worth distinguishing between.
- Opt-in, delayed-sync telemetry (question asked, answer accepted/escalated, not full transcripts by default) gives visibility without requiring constant connectivity or compromising the offline-first design.
Concept Checks
Check yourself
Why does the procedure manual go through retrieval instead of being baked into the fine-tune, given that fine-tuning is affordable at this model size?
Because the manual changes monthly, and retrieval lets that content update independently of the model — re-index, don't retrain. Baking it into the fine-tune would mean every manual update requires a new training run, a new evaluation pass, and a new staged rollout, turning a routine content update into a full model deployment.
Why does this design ship two different quantization tiers instead of one?
Because the device fleet has two real RAM tiers, and a single quantization choice would either be too aggressive for the newer phones (leaving quality on the table) or not aggressive enough for the older ones (failing to fit their tighter RAM budget). Matching the quantization level to the actual device population, confirmed via re-scoring each tier, serves both without compromising either.
Why does the escalation path queue a question locally instead of requiring immediate connectivity to escalate?
Because the core requirement is that the app must work with no reliable connectivity on-site — blocking the technician on an immediate escalation would violate that requirement the same way calling a hosted API directly would. Queuing locally and syncing once connectivity returns preserves the offline-first guarantee while still getting hard questions to a larger model or a human eventually.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Requirements drove every decision | Offline-first, mid-range hardware, monthly-changing knowledge, some genuinely hard questions |
| Model size | A 1-3B instruct model — small enough for the device, since world knowledge comes from retrieval |
| Fine-tuning | Teaches format and terminology, not manual content — content stays in retrieval |
| Two quantization tiers | Matched to the fleet's actual RAM tiers, each re-scored, not guessed |
| Local RAG | Manual updates monthly via re-indexing, independent of the model |
| Escalation queues locally | Preserves offline-first while still reaching a bigger model or human eventually |
| Versioning discipline | Manual updates and model updates are separate, both staged and monitored |
Next
You've seen every decision in this track applied to one real system. The last page collects every term used along the way: Glossary.