Skip to main content
This page explains how to add an LLM judge or an agentic judge to a Benchmark’s evaluate(). For the user-facing parameters and behavior, see Judges overview; this page covers only the implementation contract.

Choose a Judge Type

Prefer an LLM judge whenever it is enough. An agentic judge needs several full agent runs per task, costs an order of magnitude more time and tokens, and is harder to reproduce. See Evaluation Modes and Artifacts for the mode lifecycles.

Shared Rules

  • Declare judge parameters through a config mixin. Do not define judge_model or similar fields in your Benchmark config. Use LLMJudgeConfig or AgentJudgeConfig so parameter names, validation, and ${ENV} resolution stay consistent across Benchmarks.
  • Configure the judge explicitly. Do not let the Model under test grade itself by default. judge_model_spec(req) fills missing connection fields from the Model under test only to make quick trials easier; the Benchmark page should ask users for a complete, independent judge configuration.
  • Separate judge failures from wrong answers. When a judge request fails, times out, or returns output that cannot be parsed, return a FATAL issue (judge_failed, or judge_invalid_output for malformed output) instead of a score of 0. See Results and Aggregation for the classification rules.
  • Keep the grading evidence. Write the judge’s verdicts, reasons, and enough diagnostics into the result for users to check per-task scores.
  • Document the judge on the Benchmark page. State what the judge does, its verdict format, the recommended judge Model, and any changed defaults, and link to the shared judge pages instead of repeating the shared parameters.

LLM Judge

Declare the Config

List LLMJudgeConfig first among the config’s bases and call LLMJudgeConfig.__post_init__(self) from __post_init__; it runs the base config’s __post_init__ first:
LLMJudgeConfig provides: You do not need to set an output limit: generate_text and generate_with_metadata add max_tokens 32768 when neither the call nor judge_model.params sets one (see Default max_tokens). Pass max_tokens per call only when the Benchmark needs another limit; judge_model.params still override it.

Use the Built-In Scorer

When grading compares only the final answer, use LLMJudgeScorer with evaluate_binary_scorer. The helper calls the scorer, checks the correct field, and maps exceptions to a FATAL judge_failed:
The default prompt uses the A/B/C protocol, where only A counts as correct. For your own verdict rules, pass prompt_template and response_parser to LLMJudgeScorer. The template can use the {question}, {gold_answer}, and {llm_response} placeholders. The parser receives the judge’s reply text and returns a bool; it should raise when it cannot parse the reply rather than return False.

Write a Custom Grading Flow

When rubrics, repeated sampling, or multi-stage grading do not fit a binary scorer, call generate_text or generate_with_metadata directly:
  • The client handles timeout and max_retries from judge_model.params, and sends the other keys with the request, so the scorer does not need its own timeout and retry logic.
  • When you fan out requests with asyncio.gather, the runtime already limits them to judge_concurrency.
  • To tell a truncated reply from a short one, use generate_with_metadata and check stop_reason == "length".
ResearchRubrics (one request per rubric item) and DeepResearch Bench (multi-stage grading) are useful references.

Agentic Judge

AgentJudgeScorer starts a judge Harness in the evaluation Environment and owns the mechanics: building the judge request, splitting windows, running them concurrently, retrying, time limits, and media-inspect. As with LLMJudgeScorer, the Benchmark supplies only the prompt and the parser.

Declare the Config

AgentJudgeConfig is an LLMJudgeConfig that adds judge_harness, judge_harness_params, and every judge_agent_* parameter, validated together:

Preflight the Judge

Do not probe the judge yourself. Before inference, the runner probes every Model that the config’s judge_requirements(req) returns, from the machine running AgentCompass, and stops the run on any failure, including an unreachable endpoint. See Modality Preflight.

Grade

  • build_unit(attempt, index, batch) returns a JudgeUnit for one window: the prompt, the judge’s working directory, and the judgment file path. Give each window its own directory, and do not reuse directories across retry passes.
  • parse(outcome, batch) turns a JudgeUnitOutcome into {item_id: verdict}. When outcome.judgment_text is empty, outcome.error is set, or the judgment misses items, return a failure marker for those items (such as {"passed": None, "error": ...}) instead of raising; outcome.failure_reason() explains the failure.
  • is_judged(verdict) tells valid verdicts from failure markers. Unjudged items are re-judged in the next pass; an item that is never judged keeps its first failure marker.
  • Time limits. Call evaluation_deadline() at the start of evaluate(); it sits a margin before the runtime’s evaluation watchdog. Pass it to scorer.score(..., deadline=deadline) so every judge run is cut to the remaining budget. Then wrap the whole judging step in call_with_timeout(..., timeout=remaining_budget(deadline)) as a backstop, and turn the TimeoutError it raises into judge_failed. call_with_timeout forwards positional arguments only, so bind keyword arguments with functools.partial.
  • Media. Put scorer.media_instructions() in the judge prompt: it explains how to use media-inspect, and returns an empty string when no media-inspect is configured. Write one prompt for the judge’s fixed modalities; do not add text-only or no-audio variants. If media-inspect cannot be installed in the evaluation Environment, scorer.score() raises, so the task’s judging fails. scorer.media_summary() returns ffmpeg availability and call counts for the result’s diagnostics.
  • Installation. A fresh Environment has nothing from inference installed, so a preinstalled judge Harness falls back to install_if_missing unless the judge params set install_strategy explicitly.

Aggregate the Result

  • If any item is still unjudged at the end, return a FATAL judge_failed for the task and no score.
  • A missing deliverable is a normal result. Tell the judge in the prompt to fail the items that depend on it, rather than recording a judge failure.
  • Save one record per judge run (window name, covered items, error, and outcome.diagnostics()), such as judge_windows in AA-Briefcase-Lite-AC. To keep the judge agent’s transcript, add outcome.transcript(): one entry per step with the text, reasoning, tool calls, and tool outputs, and inline base64 media replaced by its size.

Hide Which Side Is Under Test

When the judge compares against a baseline or reference, present the evidence under neutral labels and paths. Do not reveal in the prompt, file names, or directory names which side comes from the Model under test. GDPval-AC presents the output under test and the fixed baseline as A and B.

Reference Implementations

Before You Submit

  • A missing judge_model fails with a clear message before inference (through judge_model_required or a preflight).
  • Judge failures, unparseable output, and timeouts all return a FATAL issue and are never scored as 0.
  • Local tests check the parser on empty replies, malformed output, and missing items (keep test files under the ignored root tests/ directory).
  • The config sets judge_modalities, the runner’s preflight covers every judge Model through judge_requirements(req), and evaluate() uses evaluation_deadline().
  • The Benchmark page states the judge’s role, the recommended judge, and changed defaults, and links to the Judges overview.