[
EVALS GUIDE
]
Learn LLM evals: the free guide.
17 chapters on finding out whether your AI feature works and whether a change made it better. From the first error analysis to evals in CI, sourced throughout and without a vendor's tool ranking.
[
17 CHAPTERS
]
Written for engineers running LLM features in production. Free to read, no sign-up. Seven of the 17 chapters:
03
Designing and annotating eval sets
Which cases belong in the set, who labels them and when it gets frozen.
05
Building a calibrated LLM judge
Binary verdicts with a critique, measured against human labels.
07
Statistics for small eval sets
When a two-point difference is real and when it is noise.
09
Evals in CI
Every prompt or model change runs against the set before it ships.
16
Model selection
Finding the cheapest model that clears the bar.
17
The tooling landscape
Benchmark harnesses, eval frameworks and platforms, described without a ranking.
[
THE OUTCOME
]
That is where the guide takes you: every model against the same frozen set, quality and cost side by side, and a gate that decides what ships. Among the models that pass, the cheapest one wins.
Classifying insurance claims · 412 cases
Model
Quality
Gate
Frontier API (today)
95.1%
€8.40
PASS
Mistral Large
93.4%
€2.20
PASS
Qwen3 32B
92.6%
€0.70
PASS
Llama 3.3 70B
91.2%
€0.80
FAIL
Gemma 3 27B
88.0%
€0.40
FAIL
Illustrative example. Your report shows your task, your set and your models.
[
WITH KONTINENT
]
Rather not do it alone? We build the eval set with you.
The eval set comes from your operations.
Real cases, checked by the people who decide today what counts as correct. The set is frozen so every number stays comparable later.
Graders you can trust.
Where an answer can be checked in code, code checks it. Where an LLM judge is needed, it is calibrated against human judgments before it decides anything about a model.
The cheapest model that clears the bar.
The gateway puts 170+ models behind one OpenAI-compatible API. Each of them can be measured against your set, so you choose with numbers instead of gut feeling.
Questions about LLM evals
[
GET STARTED
]
Start with chapter one.
Or talk to us if you would rather not build your eval set alone. We build it with your domain experts and use it to find the model that solves your task at the lowest cost.