Transformers & Pipelines
pipeline() versus the Auto* classes, generation parameters, dtype, and device_map — the two ways to run a model and when each earns its cost
Transformers & Pipelines
TL;DR
transformers gives you two ways to run a model: pipeline(), a one-call task API that handles preprocessing, inference, and postprocessing for you, and the Auto* classes, which give you the raw tokenizer and model so you can control generation, batching, and sampling yourself. Reach for pipeline() first; drop to Auto* classes the moment you need something the pipeline doesn't expose.
| Property | Value |
|---|---|
| Level | Beginner |
| Reading time | ~20 minutes |
| Prerequisites | What Is HuggingFace? |
| You will understand | When to use pipeline() vs Auto* classes, and how to control generation, precision, and device placement |
pipeline() — the Fastest Path to a Working Model
from transformers import pipeline
classifier = pipeline("sentiment-analysis")
classifier(["Great documentation.", "This broke my afternoon."])
# [{'label': 'POSITIVE', 'score': 0.9998}, {'label': 'NEGATIVE', 'score': 0.9991}]One function call: it resolves the task string to a default model repo, downloads the model and tokenizer, builds the right pre- and post-processing, and gives you a callable that batches automatically when you pass a list. pipeline() covers dozens of tasks across text, vision, and audio, plus a unified any-to-any pipeline for multimodal chat-style input — a list of role/content messages where content can mix text, image, and audio in one call, the same shape as a chat completion API:
chat_pipe = pipeline("image-text-to-text", model="some-org/vision-chat-model")
chat_pipe([
{"role": "user", "content": [
{"type": "image", "image": "diagram.png"},
{"type": "text", "text": "What does this diagram show?"},
]},
])You never construct a tokenizer or a model object. That's the entire value proposition, and it's the right default for prototyping, a known task, or anything where you don't need custom pre/post-processing.
The Auto* Classes — When You Need Control
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.8-27B")
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.8-27B",
dtype="bfloat16",
device_map="auto",
)
inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=20, do_sample=False)
tokenizer.decode(outputs[0], skip_special_tokens=True)Auto* classes read the repo's config.json and resolve to the correct architecture class automatically — you never need to know that "Qwen/Qwen3.8-27B" is actually a Qwen3ForCausalLM under the hood. This is the layer you drop to for:
- Custom generation loops (streaming tokens, constrained decoding, speculative decoding)
- Batched generation with explicit control over sampling parameters
- Feeding the model into
peft(attaching a LoRA adapter) ortrl(an alignment trainer) - Any workflow where a pipeline's fixed pre/post-processing gets in the way
Side by Side: Same Task, Both Approaches
Sentiment classification, once via pipeline(), once by hand — so the tradeoff is concrete rather than abstract.
pipeline() vs Auto* classes for the same task
pipeline() — 3 lines
Recommendedfrom transformers import pipeline
classifier = pipeline("sentiment-analysis")
classifier("Solid release notes this time.")Handles tokenization, batching, and label mapping invisibly. Fastest to write, fastest to read, and correct by default. Loses: no access to raw logits, no control over batch size or padding strategy without extra arguments.
Auto* classes — explicit control
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tok = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
model = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased-finetuned-sst-2-english"
)
inputs = tok("Solid release notes this time.", return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits
probs = torch.softmax(logits, dim=-1)More code, but you now hold the raw logits — useful for calibration, custom thresholds, or feeding the score into something else downstream.
| Choose | When |
|---|---|
pipeline() | Prototyping, a known task, no need for raw outputs or custom batching |
Auto* classes | Custom generation logic, raw logits/hidden states, or feeding into peft/trl |
Generation Parameters Worth Understanding
Once you're calling model.generate() directly, these four arguments account for most of what "the output looks wrong" actually is:
| Parameter | What it controls | Effect |
|---|---|---|
max_new_tokens | Hard cap on generated length | Too low truncates mid-thought; too high wastes latency and cost on runs that were already done |
do_sample | Whether to sample or take the highest-probability token every step | False (greedy) is deterministic and reproducible; True enables the parameters below |
temperature | Sharpens or flattens the probability distribution before sampling | Near 0 → almost greedy; higher → more varied, more likely to wander off-topic |
top_p | Nucleus sampling — only sample from the smallest set of tokens whose probabilities sum to p | Lower top_p keeps sampling but trims the low-probability tail that produces garbled tokens |
model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.7,
top_p=0.9,
)temperature and top_p do nothing unless do_sample=True. With do_sample=False (the default in many configs), generation is greedy — deterministic, and immune to both parameters. Setting temperature=0.9 on a greedy generation call is a no-op, not a subtle tuning knob, and the silent no-op is a common source of "why isn't temperature changing anything" confusion.
dtype, Not torch_dtype
Transformers v5 renamed the load-time precision argument from torch_dtype to dtype. The old name still works but is deprecated — use the new one going forward.
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.8-27B",
dtype="bfloat16",
)| Precision | When |
|---|---|
bfloat16 | Default for training and most GPU inference — same exponent range as float32, so it's numerically stable without the memory cost of full precision |
float16 | Only where bfloat16 isn't supported by the hardware — narrower exponent range, more prone to overflow in some training setups |
float32 | Rarely needed for inference; occasionally needed for numerically sensitive training steps |
Not passing dtype at all does not mean "safe default." Without it, weights load in whatever precision they were saved in — sometimes float32, which is 2× the memory of bfloat16 for no accuracy benefit at inference time. Pin it deliberately rather than letting the checkpoint decide.
device_map="auto"
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.8-27B",
dtype="bfloat16",
device_map="auto",
)device_map="auto" hands GPU placement to accelerate, which is a transformers dependency you don't have to interact with directly for this. It inspects available devices (GPUs, and CPU/disk as a fallback) and places the model's layers to fit — including offloading layers to CPU RAM or even disk when the model doesn't fit in GPU memory alone. Without it, you'd write the placement code yourself: computing how much fits on each device, moving tensors, and keeping activations on the same device as the layer that produced them.
What device_map="auto" is doing for you
You write
accelerate inspects
accelerate places
This is the same mechanism Distributed Training builds on for multi-GPU training — device_map="auto" is its simplest, inference-time form.
Chat Templates, Briefly
Chat-tuned models expect input formatted a specific way — role markers, turn separators, a generation prompt. That formatting is model-specific, and getting it wrong is a common source of a chat model ignoring instructions or answering oddly. This deserves full treatment rather than a summary here — see Tokenizers for the complete walkthrough of apply_chat_template().
Concept Checks
Check yourself
You need raw logits to compute a custom confidence threshold. Which approach?
The Auto* classes. pipeline() returns already-postprocessed output (a label and a score derived from softmax), not the raw logits tensor. Load the tokenizer and model directly, run a forward pass, and read outputs.logits yourself.
You set temperature=1.2 but the output barely changed. Why might that be?
do_sample is probably False. Temperature only affects the probability distribution used during sampling — with greedy decoding, generation always picks the single highest-probability token regardless of temperature, so the parameter is silently ignored.
Why does pinning dtype explicitly matter even though the model 'works' without it?
Without an explicit dtype, weights load in whatever precision the checkpoint was saved in — often float32, double the memory of bfloat16 for no accuracy gain at inference time. It still runs, it just wastes GPU memory and can silently limit how large a model or batch size fits on the hardware you have.
What is device_map="auto" actually doing under the hood?
It hands placement decisions to accelerate, which inspects the available devices and free memory, then assigns the model's layers across GPUs — and offloads to CPU RAM or disk if the model doesn't fit in GPU memory alone. It replaces placement code you would otherwise have to write by hand.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
pipeline() | Task string in, prediction out — handles pre/post-processing for you |
| Any-to-any pipeline | Unified multimodal chat-style pipeline for mixed text/image/audio input |
Auto* classes | Raw tokenizer and model, resolved to the right architecture automatically from config.json |
max_new_tokens | Hard cap on generated length |
do_sample | Gate for whether temperature/top_p have any effect at all |
temperature / top_p | Control randomness and tail-trimming during sampling — inert without do_sample=True |
dtype | The v5 name for load-time weight precision; replaces deprecated torch_dtype |
device_map="auto" | Hands GPU/CPU placement to accelerate, including offload when needed |
Next
Every one of those calls starts by turning text into numbers — see exactly how, and the single mistake that breaks it silently: Tokenizers.