Project Ideas
A starting brief for a production-shaped capstone — a monitored, cost-aware LLM gateway that ties the whole MLOps track together
Project Ideas
This is a brief, not a build guide
Like the HuggingFace, Small Language Models, and Deep Learning briefs, this page sets up one well-scoped project idea and leaves the building to you. Everything you need conceptually is in the Fundamentals track; this is where you'd point it.
The idea: a production-grade LLM gateway — routed, cached, monitored, and cost-aware
Why this, specifically: almost every real system built on LLMs today needs the same operational layer — something between the application code and the model providers that handles routing, failure, cost, and visibility — and almost nobody builds it deliberately until costs spike or an outage from one provider takes the whole product down with it. Building this gateway yourself, even a modest version, exercises nearly every idea in this track: serving, caching, monitoring, drift-adjacent quality tracking, and the LLM-specific routing problem, all in one coherent system with a clear, checkable purpose.
One request through the gateway
Why this shape, specifically
Design choices worth making deliberately
Build the cost dashboard before you build anything clever
RecommendedThe gateway's entire justification is visibility and cost control — if you can't see cost-per-request broken down by route and cache-hit-or-miss from day one, you can't tell later whether any of the cleverness (caching, routing) actually helped. Get the observability layer working first, even against a trivial single-provider passthrough.
Treat the fallback chain as a first-class feature, not an afterthought
RecommendedA gateway that routes to one provider with no fallback is barely different from calling that provider directly. The real value is in what happens when the primary route fails or rate-limits — test that path deliberately, don't just assume it works because you wrote the code.
Tune the semantic cache's similarity threshold with real (wrong) examples
Per the crash course's warning, a loose threshold returns confidently wrong cached answers. Deliberately construct near-miss query pairs that should and shouldn't share a cache entry, and use them as a real test set for the threshold, not a guessed constant.
Suggested stack
| Layer | Choice | Role |
|---|---|---|
| Gateway service | FastAPI (or similar), containerized | The routing/caching/reliability layer itself |
| Caching | Exact-match (hash) + semantic (embedding similarity) | Covered in Caching & Cost Optimization |
| Routing | A rule or cost-based router across at least two providers | The core LLM-specific operational problem from LLM Gateway & Routing |
| Observability | Prometheus + Grafana (or equivalent) | Per-request cost, latency, and provider tracked from Monitoring & Observability |
| Deployment | Docker, with health checks that exercise the real path | From Containerization & Packaging and Model Serving Patterns |
Stretch directions, if the base project goes well
- Add A/B testing between two routing strategies (A/B Testing & Experimentation) — does cost-based routing actually produce worse response quality than always using the strongest model, or not?
- Wire in a CI/CD validation gate (CI/CD for ML) that blocks a routing-rule change from deploying if it would have degraded cost or latency on a replayed traffic sample.
- Put a small local model from the Small Language Models track behind one of the gateway's routes, so the router can send easy requests to a free, local model and only pay for the ones that need it — the escalation pattern from that track, made real.
Before you start
Work through What Is MLOps? through Model Serving Patterns at minimum — the rest of the fundamentals track fills in as you hit each stage above. If a step above uses a term or technique you haven't seen yet, that's the signal for which fundamentals page to read next, not a sign you're missing something.
Glossary
Every MLOps term and abbreviation used across the fundamentals track, grouped by theme — from health checks and HPA to drift, gateways, and governance
Deep Learning Crash Course
All of deep learning on one page — neural networks, training loops, the transformer, fine-tuning, compression, alignment, and distributed training, from first principles