Evaluation
Metrics with the evaluate library, why a leaderboard score isn't your score, building a held-out test set, and LLM-as-judge for generative tasks
Evaluation
TL;DR
"It works when I try it" is not evaluation. HuggingFace's evaluate library gives you a single interface to hundreds of metrics, but the metric matters less than the discipline: score on held-out data you control, not training data and not someone else's benchmark. For generative and agentic tasks, where exact-match metrics stop meaning much, LLM-as-judge has become the practical default — with its own failure modes to watch for.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Quantization & Optimization |
| You will understand | How to measure a model honestly, and why most evaluation mistakes happen before a single metric is computed |
"It Works" Is Not a Measurement
Trying a model on a handful of prompts and liking the results is real signal, but it's an uncontrolled sample of size one, chosen by you, after you already had an opinion. It tells you the model can do the task, not how often, and it tells you nothing comparable across model versions, fine-tunes, or quantization settings.
The question evaluation answers is comparative, not absolute. "Is this model good?" is a much weaker question than "is this fine-tune better than the baseline, on the same data, by the same metric?" Almost every evaluation mistake traces back to skipping the comparison and asking the absolute question instead.
The evaluate Library
import evaluate
metric = evaluate.load("f1")
metric.compute(predictions=preds, references=labels, average="macro")
# {'f1': 0.812}One interface, hundreds of metrics — from standard ones (accuracy, F1, precision/recall, BLEU, ROUGE, perplexity) to community-contributed, task-specific ones. evaluate.load() pulls the metric implementation the same way load_dataset() pulls data — from the Hub, cached locally after the first call.
Classification/extraction metrics vs generation metrics
| Task shape | Typical metric | What it actually measures |
|---|---|---|
| Classification, extraction, structured output | Accuracy, F1, precision/recall | Exact or near-exact correctness against a known label |
| Translation, summarization (reference-based) | BLEU, ROUGE | Surface overlap with one or more reference texts |
| Language modeling | Perplexity | How well the model predicted the next token, on average |
| Open-ended generation, chat, agents | (see LLM-as-judge below) | Whether the output is good, which overlap metrics don't capture |
BLEU and ROUGE correlate poorly with actual quality for open-ended text. They reward surface overlap with a reference, so a paraphrase that says the same thing in different words scores worse than a shallow near-copy that's actually less useful. They're still reasonable for tasks with a fairly constrained "correct" phrasing (translation, close summarization), but treat a high score on genuinely open-ended generation with real skepticism.
The Leaderboard Trap
A model's published benchmark score is real, but it describes performance on that benchmark's distribution of questions — a specific mix of topics, difficulty, and phrasing. Your task is a different distribution: different domain vocabulary, different failure modes, different definition of "correct."
What a leaderboard number actually tells you
Useful for: comparing base models before you commit to one
A rough signal for narrowing down candidates worth trying on your own data — treat it as a filter, not a verdict.
Not useful for: predicting performance on your task
A model that tops a general reasoning leaderboard can perform worse than a smaller model on your specific, narrower task — leaderboards optimize for breadth, not for your distribution.
Not useful for: proving a fine-tune helped
Fine-tuning changes behavior on your data. A benchmark score computed before fine-tuning tells you nothing about whether the fine-tune moved the needle on the thing you actually changed it for.
The only score that answers "did this help my task" is one computed on your held-out data, with your metric.
Building a Test Set That Won't Lie to You
| Rule | Why |
|---|---|
| Never let it touch training data | Even indirect leakage — near-duplicates, the same source documents — inflates scores and hides real failures |
| Size it to the decision, not a fixed number | A few hundred well-chosen examples beats a few thousand noisy ones; enough that a meaningful quality change moves the score outside noise, however many that takes for your task's variance |
| Include the hard and the ambiguous, not just the easy | A test set of only clear-cut cases overstates how good the system looks in practice |
| Refresh it as the task drifts | A test set frozen a year ago on last year's data quietly stops measuring the thing you care about today |
Tie evaluation directly to fine-tuning. Always score the baseline (zero-shot, or the previous fine-tune) and the new fine-tune on the same held-out set, with the same metric, before believing the new one is actually better. See Fine-Tuning with PEFT — this is the step people skip most often, going straight from "trained" to "shipped" with no comparison in between.
LLM-as-Judge
For generative and agentic tasks — chat responses, summaries, agent trajectories — exact-match and overlap metrics stop being meaningful, because there's often no single correct string. The practical default is having another model judge the output against a rubric.
judge_prompt = """
Rate the response from 1-5 on whether it correctly and completely answers the question,
using only information in the provided context. Respond with just the number.
Question: {question}
Context: {context}
Response: {response}
"""LLM-as-judge failure modes to design around
Vague criteria produce vague judgments
"Rate the quality" gives the judge nothing concrete to check. Specific, checkable criteria — "does it cite a source," "does it contain a number that's actually in the context" — produce far more consistent scores.
Position bias
When comparing two responses side by side, judges tend to favor whichever position (first or second) they saw more often during their own training. Randomize the order across trials, or score responses independently rather than comparatively, to reduce this.
Self-preference bias
A judge model tends to rate output that resembles its own writing style more favorably — which quietly penalizes a fine-tune that deliberately writes differently. Where possible, use a judge from a different model family than the one being evaluated.
A judge prompt is still a prompt: it needs the same care, testing, and held-out validation as any other piece of the system — including, ideally, checking the judge's scores against a small set of human-labeled examples before trusting it at scale.
Concept Checks
Check yourself
A model tops a public leaderboard. Why might it still underperform a smaller, less celebrated model on your specific task?
The leaderboard measures performance on that benchmark's specific distribution of questions — its topics, phrasing, and definition of correct. Your task is a different distribution, with its own domain vocabulary and failure modes. A leaderboard score is a reasonable filter for picking candidates to try, but it's not a prediction of performance on data the benchmark never saw.
Why do BLEU and ROUGE often rate a shallow near-copy of the reference text higher than a correct paraphrase?
Both metrics score surface overlap with the reference — shared words and phrases — rather than whether the meaning is correct. A paraphrase that says the same thing in different words has low overlap and scores worse, even though it may be just as useful or more natural than a response that stays close to the reference's exact wording.
A team fine-tunes a model, ships it, and later can't explain why users report it's worse on edge cases. What evaluation step did they most likely skip?
Scoring the baseline and the fine-tune on the same held-out test set before shipping. Without that comparison, "the fine-tune worked" was never actually verified — it was assumed from a handful of manual checks that likely covered the easy cases, not the edge cases users are now hitting.
Why is asking one LLM to judge outputs from a fine-tune of the same base model a particularly risky setup?
Self-preference bias: a judge model tends to rate text that resembles its own style more favorably. If the fine-tune shares lineage with the judge, or was fine-tuned in a direction that diverges from the judge's natural style, the judge's scores can be systematically skewed — inflated for stylistic similarity, or unfairly penalized for a deliberate stylistic change. Using a judge from a different model family reduces this.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
evaluate | One interface to hundreds of metrics, loaded and cached like a Hub dataset |
| BLEU / ROUGE | Surface-overlap metrics — useful for constrained tasks, unreliable for open-ended generation |
| Perplexity | How well a model predicted the next token — a language-modeling metric, not a task-quality one |
| The leaderboard trap | A published score describes that benchmark's distribution, not your task |
| Held-out test set | Data that never touches training, sized and refreshed to actually catch regressions |
| Baseline vs fine-tune | Always score both on the same set before believing a fine-tune helped |
| LLM-as-judge | The practical default for generative/agentic evaluation, with its own biases to design around |
| Position bias | A judge favoring whichever response position it saw more often in training |
| Self-preference bias | A judge favoring output that resembles its own writing style |
Next
With a trustworthy way to measure a model, the next question is how to move its behavior beyond plain supervised fine-tuning: Alignment with TRL.
Quantization & Optimization
Why lowering numerical precision barely hurts quality, the format landscape (bnb, GGUF, AWQ/GPTQ, FP8), and matching the format to the serving engine
Alignment with TRL
SFT, reward modeling, DPO, and GRPO — what each stage past supervised fine-tuning teaches a model, and which one your data actually supports