Designing a HuggingFace-Based System
A complete worked example — from a support-triage brief to a sized, justified system using pipeline(), QLoRA, trackio, evaluate, and a Gradio Space
Designing a HuggingFace-Based System
TL;DR
This page takes one realistic brief — a support/bug-triage classifier — and makes every decision with a reason attached: which base model size, why QLoRA over full fine-tuning, what rank, how big the eval set needs to be, and which deployment shape fits the traffic. It's the same project the Project Ideas brief sets up, worked through end to end as one coherent system rather than a menu of options.
| Property | Value |
|---|---|
| Level | Advanced — brings together every previous page |
| Reading time | ~30 minutes |
| Prerequisites | Failure Modes & Debugging |
| You will understand | How to go from a vague brief to a sized, justified HuggingFace system |
Part 1 — The Brief
We run a shared support inbox for a small SaaS product. Every incoming message needs a first decision — is this billing, a bug report, a feature request, or general/other — and how urgent it is, before it's routed to the right queue. We get a few hundred messages a day. A wrong category costs us a few minutes of a support agent's time re-routing it; a wrong urgency on something actually urgent costs more. We don't have a labeling team, but we have about two years of historical tickets we could mine for examples.
Turning that into numbers:
| Requirement | Value |
|---|---|
| Volume | ~300 messages/day |
| Categories | 4 classes: billing, bug, feature-request, general — plus a binary urgent/not-urgent flag |
| Latency target | Sub-second is nice, a few seconds is fine — nothing is blocking a human's screen |
| Cost of a wrong category | Low-moderate — a few minutes of manual re-routing |
| Cost of a wrong urgency flag | Moderate — a genuinely urgent ticket sits in the wrong queue |
| Labeled data available | None yet, but ~2 years of historical tickets to mine |
| Team | One or two engineers, no dedicated ML or labeling team |
| Success measure | Category + urgency accuracy on held-out tickets, beating the zero-shot baseline by a clear margin |
300 messages a day is a small-data, small-compute problem, and the design should look like one. No distributed training, no frontier-scale base model, no dedicated GPU serving. The two things that actually matter here are getting a workable labeled set from zero, and proving the fine-tune is worth the effort over a free zero-shot baseline.
Part 2 — Do the Arithmetic
One idea, the whole stack
DATA
Historical tickets available ~2 years, unlabeled
Target labeled set 600 tickets (150/category, roughly balanced)
Split 400 train / 100 val / 100 test
Labeling method Zero-shot pipeline() pre-labels, human spot-checks
and corrects — much faster than labeling from scratch
TRAINING COMPUTE (QLoRA, ~2B base model, 400 examples, 3 epochs)
Steps 400 × 3 ÷ batch size 8 ≈ 150 steps
Wall-clock on one consumer/rented GPU ≈ 15–30 minutes
This is a coffee-break job, not a multi-day run — accelerate/FSDP add
nothing here; distributing this would be pure overhead
INFERENCE VOLUME
300 requests/day ÷ 10 active hours ≈ 30/hour ≈ 0.01/second
Peak, generously 5× ≈ 0.05/second
COST SANITY CHECK
0.05 requests/second is not a load any deployment shape struggles with.
The deciding factor is NOT throughput — it's whether paying for an
always-on dedicated Endpoint makes sense at this volume. It does not.Two conclusions fall out immediately. First, this is not a scale problem — 0.05 requests/second rules out any argument for a dedicated Endpoint or distributed training. Second, the labeling bottleneck, not the model or the compute, is the actual constraint — which is why the design leans on the zero-shot baseline to bootstrap labels instead of starting from a blank spreadsheet.
Part 3 — The Decisions
Each choice names the requirement that drove it, and cross-links the page that covers it in depth.
Base model: small, not largest available
Sizing the base model
A large general-purpose model
Better zero-shot accuracy out of the box, but far more expensive to fine-tune, slower to iterate on, and total overkill for a 4-class-plus-flag classification task. The gap between "good enough" and "state of the art" here is not worth the cost.
A small open model (1–4B parameters)
RecommendedFine-tunes in minutes with QLoRA on a single GPU, iterates fast enough to try several hyperparameter settings in an afternoon, and is plenty capable for a well-scoped classification task once fine-tuned — the fine-tune closes most of the gap a bigger base model would have covered zero-shot.
This mirrors the reasoning on Fine-Tuning with PEFT: match model size to what the task and iteration speed need, not to what's available.
Baseline: zero-shot before anything is trained
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
result = classifier(
"My invoice charged me twice this month, please refund the duplicate.",
candidate_labels=["billing", "bug", "feature-request", "general"],
)Run this over the 100-ticket test split before touching peft. This does two jobs at once: it produces the number the fine-tune has to beat, and — critically for this brief's data problem — it pre-labels the training pool so a human is correcting rather than labeling from scratch. See Transformers & Pipelines for pipeline() mechanics.
Skipping the baseline is the single most common mistake on a first fine-tuning project. Without a zero-shot number scored on the same held-out set, there's no way to know whether the fine-tune helped or you just got used to its particular mistakes.
Fine-tuning: QLoRA, not full fine-tuning
Why QLoRA for this specific project
Full fine-tuning
Updates every parameter of even a small model — unnecessary memory and time cost for a task this narrow, and it produces a full multi-gigabyte checkpoint to store and version for every experiment.
LoRA
Would work fine at this scale — the base model is small enough that memory isn't the binding constraint the way it would be on a 27B+ model.
QLoRA
RecommendedCosts almost nothing extra over plain LoRA at this model size, and defaults to it anyway means the same recipe scales unmodified if the base model later grows — a reasonable hedge for a project that might outgrow its first model choice.
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=8,
lora_alpha=16,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
)
model = get_peft_model(model, config)Rank 8, not 16 or 64. This is a narrow, low-complexity task — one label out of four categories plus a binary flag — not open-ended generation. A higher rank buys capacity this task doesn't need and costs more to train; rank 8 is a reasonable starting point, with room to raise it only if validation accuracy plateaus below the baseline plus a healthy margin. Full reasoning on rank selection lives on Fine-Tuning with PEFT.
Tracking every run
import trackio
trackio.init(project="ticket-triage", config={"lr": 2e-4, "lora_r": 8, "epochs": 3})
trackio.log({"val_accuracy": 0.91, "val_loss": 0.24})
trackio.finish()With only 400 training examples, it's tempting to treat each run as "just a quick test" and skip tracking. Don't — at this data scale, a handful of hyperparameter changes (rank, learning rate, epoch count) can each meaningfully move the result, and without trackio there's no way to tell which change actually produced the model you shipped. See Production & Operations.
Evaluation: held-out, against the baseline, both metrics
import evaluate
metric = evaluate.load("f1")
category_f1 = metric.compute(predictions=category_preds, references=category_labels, average="macro")
urgency_f1 = metric.compute(predictions=urgency_preds, references=urgency_labels, average="binary")The 100-ticket test split is scored twice — once for the zero-shot baseline, once for the fine-tune — on category F1 and urgency F1 separately, since they're different problems with different costs of error. Macro F1 on category guards against the fine-tune looking good only because it nailed the most common class. Full treatment of building a test set that doesn't lie to you is on Evaluation.
100 held-out tickets is a small eval set, and that's an honest tradeoff, not an oversight. At this ticket volume, a bigger held-out split means less training data, and the team has no labeling budget to grow both at once. The mitigation is running eval consistently on the same fixed split across every experiment, so comparisons between runs stay fair even if the absolute numbers carry some noise.
Deployment: a Gradio Space, not an Endpoint
Matching deployment shape to 0.05 requests/second
Inference Endpoint
Bills for uptime regardless of traffic. At a few hundred requests a day, a dedicated always-on deployment sits idle the overwhelming majority of the time — the wrong shape for this volume, however "production" it sounds.
Inference Providers
Would work, but the model here is a project-specific fine-tune, not one hosted by a routed third-party provider — Inference Providers fits pre-hosted models better than a bespoke small fine-tune.
Gradio Space, mcp_server=True
RecommendedFree-to-low-cost at this traffic level, gives support staff a UI to paste a ticket into directly, and the same mcp_server=True argument makes it callable by an internal agent or automation with zero extra integration work.
import gradio as gr
def triage(ticket_text: str) -> dict:
"""Classify a support ticket into a category and urgency flag."""
category = category_model(ticket_text)
urgency = urgency_model(ticket_text)
return {"category": category, "urgent": urgency}
demo = gr.Interface(fn=triage, inputs="text", outputs="json")
demo.launch(mcp_server=True)See Deployment & Inference for the full shape comparison and how mcp_server=True turns this into an agent-callable tool.
The resulting design
Ticket triage system, end to end
Data
Baseline
Fine-tune
Evaluate
Ship
Part 4 — Build It in Stages
Don't build all five stages before measuring anything.
Staged rollout
Stage 1 — Zero-shot baseline only
No training yet. Run pipeline() over 100 labeled test tickets, record the number.
Stage 2 — Build the labeled set
Zero-shot pre-labels the remaining 500, a human corrects — much faster than labeling cold
Stage 3 — First QLoRA run
Rank 8, 3 epochs, tracked with trackio. Compare val accuracy to the baseline immediately.
Stage 4 — Score against the baseline
evaluate on the held-out test split, category and urgency F1 separately
Stage 5 — Ship the Space
Deploy only once the fine-tune clears the baseline by a real margin, not a rounding error
If Stage 4 shows the fine-tune barely beats the zero-shot baseline, that's a valid outcome — ship the free baseline instead. A fine-tune that costs training time and adds a model to maintain, for a marginal accuracy gain, is a worse system than just calling pipeline() directly. The evaluation step exists to make this decision honestly, not to justify training happening at all.
Part 5 — Change One Requirement
Now swap a single line of the brief: this is a content-moderation queue for user-submitted posts, not internal support tickets, and a missed violation can mean real harm reaching users before a human reviews it.
Volume and data availability are roughly unchanged. Look at how much of the design flips anyway:
| Decision | Support-triage version | Moderation version | Why it changed |
|---|---|---|---|
| Action on a positive | Auto-route to a queue | Hold for human review, never auto-publish | A missed violation reaching users is a different order of cost than a mis-routed ticket |
| Model size | Small (1–4B) | Larger, or an ensemble/second-pass check | Precision on the harmful class matters more than iteration speed here |
| Evaluation focus | Macro F1 across categories | Recall on the harmful class specifically, tolerating lower precision | Missing a real violation is far worse than a false positive that a human clears in seconds |
| Zero-shot baseline | The thing to beat | Also the fallback if the fine-tune's harmful-class recall regresses | Never ship a fine-tune that's worse than the baseline on the class that matters most |
| Deployment shape | Gradio Space | Space, but gated — no MCP auto-action tool, human-facing review UI only | An agent should not be able to auto-execute a moderation action end-to-end |
| Tracking | trackio, informal review | trackio, plus a required sign-off on any promoted model | Governance requirement once real harm is on the line |
| Revision pinning | Good practice | Strict — the exact promoted checkpoint pinned everywhere it's loaded | An accidental "main moved" bug here is a content-safety incident, not an inconvenience |
| Data mining approach | Unchanged | Unchanged | Both start from historical, human-labeled examples |
| QLoRA over full FT | Unchanged | Unchanged | Model size and cost tradeoff didn't change |
| Track every run with trackio | Unchanged | Unchanged | Reproducibility matters regardless of stakes |
Notice which rows did not change. QLoRA-over-full-FT, tracking every run, and mining historical labels are engineering fundamentals that serve either version. What changed is everything downstream of what a wrong classification actually costs — autonomy to act, which error type the evaluation optimizes for, and how tightly the deployed model is governed.
Concept Checks
Check yourself
Why does this design reject an Inference Endpoint even though it 'sounds more production-ready'?
Because an Endpoint bills for uptime regardless of traffic, and at 300 requests a day (roughly 0.05/second at peak) it would sit idle the vast majority of the time while still costing the same as if it were saturated. A Gradio Space matches the actual traffic pattern and costs far less at this volume — "production-ready" should describe fitness for the real traffic, not the most infrastructure-heavy option available.
Why run the zero-shot baseline before labeling any training data by hand?
Because it does two jobs at once: it produces the number the fine-tune has to beat, and it pre-labels the unlabeled historical tickets so a human is correcting predictions rather than labeling from a blank slate — much faster with no dedicated labeling team. Skipping it would mean starting the labeling process slower and having no baseline to know whether training was worth it afterward.
Why rank 8 instead of a higher LoRA rank for this task?
Because the task — one label out of four categories plus a binary urgency flag — has low complexity relative to open-ended generation, so it needs little extra trainable capacity. A higher rank costs more compute and storage for capacity this task won't use; rank is raised only if validation accuracy plateaus below what the baseline-plus-margin bar requires, not chosen high by default.
In the moderation variant, why does evaluation shift to recall on the harmful class instead of macro F1?
Because the two error types no longer cost the same: a missed violation reaching users is far more costly than a false positive that a human reviewer clears in seconds. Macro F1 treats all classes and error types symmetrically, which is right for the support-triage version but wrong once one specific error type carries disproportionate real-world harm — the metric has to reflect the actual cost structure of the task.
What's the argument for shipping the zero-shot baseline instead of the fine-tune, if the evaluation shows only a marginal improvement?
A fine-tune adds a model to train, track, version, and redeploy on every future data refresh, for a return that has to be worth that ongoing cost — not just better than zero, but clearly better. If Stage 4's evaluation shows the fine-tune barely clears the baseline, the simpler system (a stock model behind pipeline()) is the better engineering decision, and the evaluation step exists specifically to allow that conclusion rather than assume training was the point.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Size the model to the task | A small open model plus a good fine-tune beats a large model zero-shot, at a fraction of the cost, for a narrow task |
| Baseline before training | Zero-shot with pipeline() is both the bar to beat and a labeling accelerator |
| QLoRA by default at small scale | Costs almost nothing extra over LoRA and scales unmodified if the base model grows |
| Rank matches task complexity | Low-complexity classification needs a low rank; raise it only when evidence demands it |
| Track every run | Small datasets amplify the effect of each hyperparameter change |
| Evaluate on held-out data, against the baseline | The only way to know a fine-tune actually helped |
| Match deployment shape to real traffic | 0.05 requests/second rules out a dedicated Endpoint on cost grounds alone |
| What a wrong action costs reshapes the whole design | Autonomy, model size, evaluation metric, and governance all follow from it |
| Ship the simpler system if the complex one doesn't clearly win | A marginal fine-tune isn't worth the ongoing cost of maintaining it |
Next
That completes the worked example. Glossary closes out the track as a reference for every term used along the way.
Failure Modes & Debugging
The HuggingFace-stack bugs that show up again and again — tokenizer mismatches, CUDA OOM, silent revision drift, and the fix for each
Glossary
Every HuggingFace term and abbreviation used across the fundamentals track, grouped by theme — from the Hub and safetensors to LoRA, DPO, quantization, and MCP