Project Ideas
A starting brief for a modern SLM capstone — a fully local, in-browser AI assistant that never calls a server
Project Ideas
This is a brief, not a build guide
Like the HuggingFace project brief, this page sets up one well-scoped, genuinely modern project idea and leaves the building to you. Everything you need conceptually is in the Fundamentals track; this is where you'd point it.
The idea: an AI assistant that runs entirely in the browser tab, with no server at all
Why this, specifically, right now: for most of this field's history, "run an LLM" meant "call someone's server." What changed recently is that small, genuinely capable instruct models plus mature WebGPU inference in the browser make a different shape possible: an assistant that downloads its model once, then answers every question, retrieves over every document, and calls every tool entirely on the user's machine — zero API cost per query, zero latency to a server, and a privacy story no hosted product can match, because there is no server call to intercept. This is not a cheaper version of a hosted assistant; it's a product a hosted assistant structurally cannot be.
The whole thing, client-side
Why this shape, specifically
Design choices worth making deliberately
Everything runs client-side, nothing is a fallback to a server
RecommendedThe point of the project is to feel the actual constraint an on-device model lives under — memory, context length, latency — not to build a thin client for an API. Resist the urge to add a server-side escalation path until the local-only version is working and its limits are clear.
Pick the model for the browser, not for a leaderboard
RecommendedA 1-4B instruct model that quantizes cleanly and has a known-working transformers.js export beats a slightly stronger model that doesn't run smoothly on WebGPU. Runtime fit is a real requirement here, not an afterthought.
Treat the context window as small on purpose
Retrieve fewer, tighter passages than you would for a hosted large model answering the same question — this project is a good forcing function for actually internalizing why that matters, rather than reading about it.
Suggested stack
| Layer | Choice | Role |
|---|---|---|
| Model | A 1-4B instruct model, GGUF or ONNX export | The core reasoning/generation engine |
| Runtime | transformers.js with device: "webgpu" | In-browser inference, no server |
| Embeddings | A small ("xsmall"-class) embedding model | Local semantic search over the user's documents |
| Vector store | IndexedDB or an in-memory index rebuilt per session | Keeps retrieval fully client-side too |
| Tool calling | Constrained/structured decoding | Reliable JSON tool calls despite the model's smaller size |
| Deployment | Static site (no backend) | The whole point — nothing to operate |
Stretch directions, if the base project goes well
- Add the speculative decoding draft model from Speculative Decoding — still both models running locally, purely to cut generation latency.
- Distill a version of the model fine-tuned specifically on the assistant's own tool schema and typical questions, using the techniques from Knowledge Distillation, so it needs less prompting to call tools correctly.
- Package it as a browser extension instead of a standalone site, so it can retrieve over whatever page the user is currently looking at — the natural next step once the static-site version works, and still fully local.
Before you start
Work through What Is a Small Language Model? through Inference Engines & Runtimes at minimum — the rest of the fundamentals track fills in as you hit each stage above. If a step above uses a term or technique you haven't seen yet, that's the signal for which fundamentals page to read next, not a sign you're missing something.