Deployment & Inference
Three shapes for turning trained weights into something callable — Inference Providers, Spaces, and Inference Endpoints — and how to pick between them
Deployment & Inference
TL;DR
A model sitting in a repo is not running anywhere. HuggingFace gives you three shapes of "running," in increasing order of effort and control: Inference Providers (one API, zero infrastructure), a Space (a hosted app, optionally an MCP tool other agents can call), and an Inference Endpoint (your own dedicated, autoscaling deployment). Almost every project should start at the cheap end and move right only when traffic actually justifies it.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | Distributed Training |
| You will understand | The three deployment shapes, when each earns its cost, and how to make a Space callable by an agent |
The Three Shapes
From weights to a callable endpoint
They are not a ladder you have to climb in order — plenty of production systems live entirely on Inference Providers, and plenty of internal tools never leave a Space. Pick the shape that matches what you're actually building.
Choosing a shape
Inference Providers — no infrastructure at all
RecommendedA single, OpenAI-client-compatible API that routes your request to one of several backend providers (Cerebras, Groq, Together AI, Replicate, and others) hosting the model you asked for. You pay per request, with no HuggingFace markup on top of the provider's price. Best for prototyping, bursty or unpredictable traffic, and anything where owning a deployment would be wasted effort.
Space — a hosted app, and optionally a tool
A repo that runs code — almost always a Gradio app, sometimes Streamlit or a raw Docker container. Free compute tiers exist; paid tiers add GPUs and persistence. A Space is the right shape for a demo, an internal tool, or a UI a non-technical teammate needs to click through. Add one line and it becomes an MCP server other agents can call as a tool.
Inference Endpoint — a dedicated deployment
Your own autoscaling deployment of one model, on hardware you choose, billed for uptime rather than per request. The right shape once you have steady, latency-sensitive traffic that justifies paying for a warm deployment instead of a per-call rate. Premature for anything still finding its traffic pattern.
The most common mistake is skipping straight to an Inference Endpoint. It feels like "the real, production way" to deploy, but you are paying for uptime whether or not anything calls it. Start on Inference Providers or a Space, watch the actual traffic shape for a few weeks, and move to a dedicated Endpoint once the numbers justify owning the deployment — not before.
Inference Providers: One API, No Infrastructure
Inference Providers exposes over 200 models across text generation, image generation, embeddings, and more, through one consistent, pay-as-you-go API — and it is OpenAI-client compatible, so integrating it into existing code is usually a base_url change and nothing else.
from openai import OpenAI
client = OpenAI(
base_url="https://router.huggingface.co/v1",
api_key="hf_...",
)
response = client.chat.completions.create(
model="Qwen/Qwen3.8-27B",
messages=[{"role": "user", "content": "Summarize the tradeoffs of QLoRA in two sentences."}],
)
print(response.choices[0].message.content)Behind that one call, the router picks a provider hosting Qwen/Qwen3.8-27B — Cerebras, Groq, Together AI, or another — and returns the response in the shape you already expect. Function/tool calling works the same way as it does against any OpenAI-compatible API, so agent frameworks that already speak that interface need no special integration.
Zero vendor lock-in is the actual selling point. You are not tied to one provider's model catalog or one provider's uptime. If a provider hosting your model degrades, the router can serve the same request through another one — you wrote your integration once, against HuggingFace's interface, not against any single backend's.
Good for: prototyping, agent tool-calling, anything with unpredictable or spiky traffic, and teams that don't want to think about GPU provisioning at all. Bad for: workloads needing a fine-tuned model that isn't hosted by any provider (you'll need a Space or an Endpoint for that), or ultra-low, sub-100ms latency requirements where you need to control the exact hardware.
Spaces: A Hosted App, Optionally a Tool
A Space is a git repo that runs. Push a Gradio app, and the Hub builds and serves it — free CPU tiers for lightweight demos, paid GPU tiers for anything heavier.
import gradio as gr
def classify(text: str) -> str:
"""Classify a support message as billing, technical, or general."""
...
demo = gr.Interface(fn=classify, inputs="text", outputs="label")
demo.launch(mcp_server=True)That last argument is the detail worth dwelling on. mcp_server=True turns the Space into a Model Context Protocol server: every function becomes a callable tool, with a description auto-generated from its docstring, reachable by any MCP client — Claude, Cursor, or an agent you built yourself with smolagents. Thousands of MCP-compatible Spaces already exist on the Hub, filterable by the mcp-server tag.
from smolagents import CodeAgent, MCPClient, InferenceClientModel
with MCPClient({"url": "https://your-space.hf.space/gradio_api/mcp/"}) as tools:
agent = CodeAgent(tools=tools, model=InferenceClientModel())
agent.run("Classify this message: 'my invoice charged me twice this month'")That's the whole loop: fine-tune a small model, wrap it in a Gradio app, expose it as an MCP server, and an agent — yours or someone else's — can call it as a tool without you writing a custom client for it.
A Space that works locally can still fail to build on the Hub. The two most common causes are a missing entry in requirements.txt (your local environment had a package installed that the Space's fresh build doesn't) and a hardware tier too small for the model's memory footprint. Check the build logs on the Space's own page before assuming the code is wrong.
Good for: demos, internal tools, agent-callable tools, anything that benefits from a UI a non-engineer can use directly. Bad for: high-throughput production traffic — free and low tiers sleep when idle and cold-start on the next request, which shows up as latency spikes a dedicated Endpoint doesn't have.
Inference Endpoints: Your Own Dedicated Deployment
An Inference Endpoint is a specific model, deployed on hardware you choose, autoscaling within bounds you set, billed for the time it's running rather than per request.
| You get | You pay for |
|---|---|
| A private, dedicated deployment of exactly the model you specify (including your own fine-tune) | Uptime — the instance runs, and bills, whether or not it's handling traffic |
| Control over hardware (GPU type, replica count, autoscaling range) | The overhead of picking the right hardware tier for your model's memory and latency needs |
| Predictable, low-variance latency, since you're not sharing capacity with a router's other traffic | The operational responsibility of monitoring it, same as any service you own |
This is the shape that earns its cost when traffic is steady enough that a warm, dedicated instance beats paying per call, and latency-sensitive enough that routing through a shared provider's queue isn't acceptable. A support-triage model serving real-time traffic all day is a good fit. A batch job that runs once a night, or a feature still finding its usage pattern, is not — either of those is cheaper and simpler on Inference Providers or hf jobs.
Putting It Together
How a deployment shape typically evolves
Prototype on Inference Providers
No infrastructure, fast to iterate, cheap while traffic is near zero
Ship a Space
A UI for people, an MCP tool for agents, still no dedicated infrastructure to manage
Move to an Inference Endpoint
Once traffic is steady and latency-sensitive enough to justify owning a warm deployment
Nothing forces you through all three stages — a huge number of real systems stop at stage one or two permanently, and that's the correct outcome, not an unfinished one.
Concept Checks
Check yourself
Why is Inference Providers OpenAI-client-compatible worth mentioning specifically?
Because it means adopting it into an existing codebase is usually a one-line change — swap the base_url and API key — rather than a rewrite against a bespoke SDK. Any tooling, agent framework, or internal library already built against the OpenAI client shape works against Inference Providers with minimal integration cost.
A team wants to deploy a small internal tool that three colleagues will use a few times a day. What shape fits, and why not an Inference Endpoint?
A Space. Three colleagues using it occasionally is nowhere near the traffic that justifies paying for a dedicated, always-on deployment — an Inference Endpoint would bill for uptime through long idle stretches for no benefit. A Space (or Inference Providers behind a small app) costs little or nothing at that volume and is far less to operate.
What does `demo.launch(mcp_server=True)` actually change about a Gradio app?
It exposes the app's functions as MCP tools alongside the normal web UI — each function becomes callable by any MCP client, with its description generated from its docstring. The app still works exactly as a UI for a human; the MCP server is an additional interface on the same deployment, not a replacement for the UI.
Why can a Space that runs perfectly on your laptop fail to build on the Hub?
Because the Hub builds it fresh from requirements.txt in an isolated environment — any package your laptop happens to have installed but that isn't declared there will be missing on the Space. The other common cause is a hardware tier too small to hold the model in memory, which fails at load time rather than at build time. Both show up in the Space's build/runtime logs, which is the first place to look, not the code.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Three shapes | Inference Providers, Spaces, Inference Endpoints — increasing control and cost |
| Inference Providers | One routed, OpenAI-compatible API across many backend providers, no infra |
| Zero vendor lock-in | The router can serve a model through multiple providers behind one interface |
| Space | A hosted app — a UI for people, and with mcp_server=True, a tool for agents |
| MCP | The protocol that lets an agent call a Space's functions as tools |
| Inference Endpoint | A dedicated, autoscaling deployment of one model, billed for uptime |
| When to dedicate hardware | Steady, latency-sensitive traffic — not by default |
| Space build failures | Usually requirements.txt gaps or an undersized hardware tier |
Next
From serving text models, the same ideas extend to images, video, and audio: Diffusers & Multimodal.
Distributed Training
accelerate as the abstraction over DDP, FSDP, and DeepSpeed — when a single GPU is enough, and how to rent one instead of owning a cluster
Diffusers & Multimodal
Generating images, video, and audio with the diffusers library — pipelines, per-component quantization, and managing memory on models that don't fit