SLM Agents & Tool Use

Where the Gap Actually Shows Up

Small models are often surprisingly competent at single-turn tasks — answering a question, summarizing a passage, classifying text. The gap to large models widens sharply on tasks that require holding a plan across several steps and producing output another program has to parse correctly. Tool use is exactly that: pick the right tool, produce syntactically valid arguments, and do it consistently across many calls, not just most of the time.

Fix One: Constrained Decoding

Instead of hoping the model produces valid output, make invalid output impossible to generate. Constrained (or structured) decoding restricts which tokens are allowed at each step of generation, so that the output is forced to conform to a grammar — typically a JSON Schema describing the tool's exact argument shape.

This single change closes most of the reliability gap that makes small models feel untrustworthy for tool use — the model doesn't need to get smarter about formatting, because formatting is no longer something it has discretion over.

Fix Two: Tool-Use Fine-Tuning

Constrained decoding guarantees the shape of a tool call is valid. It says nothing about whether the model picked the right tool, with the right arguments, for the situation. That's a task-competence problem, and it responds to the same fix covered in Fine-Tuning Small Models: train on examples of your exact tools and your exact typical call patterns.

A practical fine-tuning set for tool use looks like transcripts: a user message, the correct tool call (or sequence of calls), the tool's result, and the model's final response — enough examples covering your actual tool surface that the model learns the pattern of when each tool applies, not just memorizes individual examples.

Fix Three: Route What You Can't Fix

Some tasks are genuinely too hard for a small model regardless of constraints or fine-tuning — deep multi-step reasoning, ambiguous instructions needing real judgment, tools whose correct use depends on world knowledge the small model simply doesn't have. For those, the answer isn't a bigger fine-tuning set — it's not asking the small model to do it alone.

Request arrives
   │
   ▼
Small model attempts it (fast, cheap, constrained decoding on)
   │
   ├── Confident, tool calls valid, task fits the trained pattern → done
   │
   └── Low confidence, unfamiliar request shape, or repeated tool failures
              │
              ▼
       Escalate to a larger model (or a human)

This routing pattern is covered in full in Production & Operations — it's introduced here because tool use is where the need for it is most obvious: a malformed or wrong tool call is a much costlier mistake than a slightly-worse prose answer, so the threshold for escalating should be lower for agentic tasks than for plain generation.

Choosing an Agent Framework for a Small Model

smolagents offers two patterns worth distinguishing at this scale:

Concept Checks

Key Concepts Recap

Next

With tool use covered, the next page addresses a different gap — a small model's narrower world knowledge — and the technique built to fix it: SLM RAG.