On-Device & Edge Deployment
The full stack from quantized model file to a shipped offline app, and the constraints — RAM, battery, download size, versioning — unique to running on someone else's device
On-Device & Edge Deployment
TL;DR
On-device deployment isn't just "the cheap option" — it's the one deployment shape a hosted API structurally cannot offer: zero network latency, zero per-token cost, and the strongest privacy story available, because there's no server call to intercept in the first place. It comes with its own constraints too: device RAM, battery/thermal budget, download size, and a model-update story that needs the same discipline as shipping app code.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~16 minutes |
| Prerequisites | Inference Engines & Runtimes |
| You will understand | Why on-device is a distinctive deployment shape, and the practical constraints that come with it |
The Full Stack, End to End
A fully local SLM stack
Model
Runtime
App
Every layer was covered on its own in earlier pages — the model in Quantization for Edge, the runtime in Inference Engines & Runtimes. This page is about what changes once you stack them and actually ship to a device you don't control.
Why On-Device Is Distinctive, Not Just Cheap
What a server call costs that an on-device call doesn't
Latency
RecommendedNo round trip to a server means no network latency at all — the only latency left is the model actually computing, which is the same latency a hosted API pays on top of the network hop.
Cost
RecommendedOnce the model is on the device, every inference is free at the margin. A hosted API bills per token forever.
Privacy
RecommendedThere is no server call to intercept, log, or breach, because there is no server call. This is a structurally stronger claim than "we don't log your data" — the data never left the device to be logged.
"Works offline" and "runs on-device" are related but not the same claim. An app can cache responses or ship a small offline fallback while still preferring a hosted model when connectivity exists. True on-device deployment means the primary path never needs a network call — decide deliberately which one you're actually building, since the constraints below only bite for the latter.
The Constraints That Come With It
Device RAM budget
A model has to fit alongside the OS, the app itself, and whatever else is running — not just fit in isolation. A phone with 6GB of RAM might realistically give an app 1-2GB of headroom, which rules out anything past a small, aggressively quantized model. Budget for the worst device you intend to support, not the one on your desk.
Battery and thermal impact
Sustained inference is real, continuous compute — noticeably different from the app's normal idle behavior. A chat-style assistant that's generating tokens for ten seconds at a time, repeatedly, will show up in battery diagnostics and can trigger thermal throttling on sustained use, which then slows the model down further. Profile actual battery drain during a realistic usage session, not just a single inference call.
Download size and first-run experience
A multi-hundred-megabyte model download on first launch is a real adoption barrier, especially on metered connections. Decide deliberately between bundling the model in the app binary (larger install, works immediately offline) versus fetching it once on first run (smaller install, needs a network connection at least once, needs a loading/progress experience).
Bundled: larger app store download, works the instant it's installed
Fetch-once: smaller install, needs one successful download before first use
(and a real UI for "downloading the model…", not a spinner)Model versioning and OTA updates
An on-device model update is a deploy, not a content update. Swapping the on-device model file changes the app's actual behavior — its accuracy, its failure modes, sometimes its output format. Treat it with the same rollout discipline as shipping new code: staged rollout to a fraction of users first, the ability to roll back to the previous model file, and monitoring for regressions after the swap — not a silent background download that flips behavior for everyone at once.
Offline-First Application Design
Designing for the constraint, not around it
Graceful degradation deserves more attention than it usually gets. Some fraction of any real device population will be older, more memory-constrained, or already under memory pressure from other apps. Decide upfront what the app does there — a smaller fallback model is often a better answer than a hard failure.
Concept Checks
Check yourself
Why is on-device deployment described as structurally more private than a hosted API with a strict no-logging policy?
Because a no-logging policy is a promise about what happens to data that still traveled to a server — it can be misconfigured, breached, or simply not honored. On-device deployment removes the network call entirely, so there's no data in transit and no server-side copy to protect in the first place. The privacy property doesn't depend on trusting a policy.
Why should an on-device model update follow the same rollout discipline as an app code deploy, rather than shipping as a quiet background download?
Because swapping the model file changes the app's actual behavior — accuracy, failure modes, sometimes output format — the same way a code change would. A silent update that regresses quality for every user simultaneously, with no staged rollout and no rollback path, is exactly the failure mode that code-deploy discipline (canary rollout, monitoring, rollback) exists to prevent.
An app bundles a 400MB model in its install versus fetching it once on first launch. What does each choice trade off?
Bundling produces a larger app store download but works immediately offline with no first-run network dependency. Fetching once keeps the install small but requires a real first-run download experience — a progress UI, retry handling, and a plan for what happens if that first download fails or the user is offline at install time.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| The full local stack | Quantized model file → runtime → app, all client-side |
| Why on-device is distinctive | Zero network latency, zero marginal cost, strongest available privacy story |
| RAM budget | Must fit alongside the OS and app, on the worst supported device — not just in isolation |
| Battery/thermal impact | Sustained inference is real continuous compute; profile it under realistic use |
| Bundle vs fetch-once | A real design decision trading install size against first-run network dependency |
| OTA model updates are deploys | Staged rollout and rollback, not a silent background swap |
| Graceful degradation | Plan the fallback for devices too constrained to run the model at all |
Next
The stack is running on-device — the next page covers a technique that uses a small model differently: making a larger model faster. Speculative Decoding.
Inference Engines & Runtimes
llama.cpp, transformers.js with WebGPU, mobile-native runtimes, and vLLM/TGI — matching the runtime to where the model actually has to run
Speculative Decoding
How a small draft model makes a large target model faster — the mechanism, why it works, and the alternatives that don't need a second model at all