Inference Engines & Runtimes

The Decision, Up Front

The rest of this page goes through each in enough depth to actually choose.

llama.cpp: The CPU/Local Default

llama.cpp is a C/C++ inference engine built specifically for running GGUF models with no heavyweight dependencies — no Python interpreter, no CUDA toolkit, no framework installation. That makes it the natural choice whenever "local" means "someone's laptop" or "a lightweight self-hosted box," where you can't assume a GPU or a Python environment is available.

# Point it at a Hugging Face repo path and filename directly
llama-cli -hf TheBloke/some-model-GGUF:Q4_K_M -p "Explain memory mapping in one sentence."

It runs well on CPU (with AVX/AVX2/AVX512 vector instructions where the hardware supports them) and can also use GPU acceleration (CUDA and others) when one is present — the same binary scales from a phone-class ARM chip to a desktop with a discrete GPU. It also exposes an OpenAI-compatible API server (llama-server), so existing client code written against the OpenAI chat completions format can point at a local llama.cpp server with essentially no changes.

transformers.js + WebGPU: The Browser

This is the newest deployment shape to become genuinely practical: a model running inside a browser tab, with no server involved at any point after the page and model files are fetched.

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline(
  "feature-extraction",
  "mixedbread-ai/mxbai-embed-xsmall-v1",
  { device: "webgpu" },
);

const embeddings = await extractor(["Hello world!"], { pooling: "mean", normalize: true });

The device: "webgpu" option is the entire change from a normal transformers.js pipeline call — it's built on a collaboration with ONNX Runtime Web, which does the actual GPU-accelerated execution inside the browser sandbox. This works across modalities, not just text: text embedding (as above), automatic speech recognition (e.g. Whisper), and small vision-language models such as HuggingFaceTB/SmolVLM-256M-Instruct all run this way.

This is the runtime behind the fully local, in-browser assistant idea in Project Ideas — no backend to operate, no per-request cost, and the data never leaves the tab.

Mobile-Native Runtimes

Phones reward a real per-platform build step in exchange for the best performance the hardware allows:

Unlike llama.cpp or transformers.js, these generally aren't "point the runtime at a file and go" — they involve a real export/compile step tied to the specific target, and the resulting artifact is usually platform-specific rather than a single file that works everywhere.

vLLM / TGI: When "Small" Still Means "Served at Scale"

Not every small-model deployment is on-device. A 1-4B model served behind an API to thousands of concurrent users is still very much a small-model deployment — it just uses the GPU-serving stack instead of an edge runtime.

vLLM and Hugging Face's own TGI (Text Generation Inference) are built for throughput: batching many requests together, using AWQ/GPTQ/fp8-quantized weights (see Quantization for Edge), and optimizing for tokens-per-second across the whole fleet rather than latency on a single request. This is the right choice when the constraint driving "small" is cost per request at scale, rather than "must run on this specific device."

Matching Runtime to Constraint, Not to Familiarity

Concept Checks

Key Concepts Recap

Next

Runtime chosen — now the full on-device deployment story, from quantized file to shipped app: On-Device & Edge Deployment.