Two Kinds of Judge
You configure both kinds through
--benchmark-params, and judge_model names the judge Model. The --model-* options always refer to the Model under test, never to the judge.
Choose a Judge Model
- Keep the judge fixed. The judge decides the score. When you compare Models under test, use the same judge Model with the same inference parameters every time; scores from different judges are not directly comparable.
- Do not let a Model grade itself. Connection fields omitted from
judge_modelfall back to the Model under test, which is convenient for a quick trial. For a real evaluation, setid,base_url,api_key, andapi_protocolso the judge is independent of the Model under test. - Match the judge to the Benchmark. A mid-sized Model is enough for answer equivalence. Benchmarks with complex rubrics, image inputs, or agentic inspection need a stronger judge. See each Benchmark page for its requirements.
- Record the judge. The
requestinrun_info.jsonkeeps the run’s judge configuration. When you report scores, also state the judge Model’s ID, version, and inference parameters.
A Judge Failure Is Not a Wrong Answer
When a judge request fails, times out, or returns output that cannot be parsed, the attempt is recorded as an evaluation failure (FATAL, usually with codejudge_failed or judge_invalid_output) instead of a score of 0. A judge outage therefore never counts against the Model under test. Failed attempts can be retried as described in Error Handling.
Benchmarks That Use a Judge
The table lists the Benchmarks that use a judge. The last column says whenjudge_model is required; see each Benchmark page for what the judge does and which judge Model is recommended.
Next Steps
- Configure an LLM judge’s connection, timeouts, retries, and concurrency: LLM judge.
- Configure an agentic judge’s Harness, modalities, scheduling, and budget: Agentic judge.
- Add a judge to your own Benchmark: Integrate a Judge.
