evaluate(). For the user-facing parameters and behavior, see Judges overview; this page covers only the implementation contract.
Choose a Judge Type
Prefer an LLM judge whenever it is enough. An agentic judge needs several full agent runs per task, costs an order of magnitude more time and tokens, and is harder to reproduce. See Evaluation Modes and Artifacts for the mode lifecycles.
Shared Rules
- Declare judge parameters through a config mixin. Do not define
judge_modelor similar fields in your Benchmark config. UseLLMJudgeConfigorAgentJudgeConfigso parameter names, validation, and${ENV}resolution stay consistent across Benchmarks. - Configure the judge explicitly. Do not let the Model under test grade itself by default.
judge_model_spec(req)fills missing connection fields from the Model under test only to make quick trials easier; the Benchmark page should ask users for a complete, independent judge configuration. - Separate judge failures from wrong answers. When a judge request fails, times out, or returns output that cannot be parsed, return a FATAL issue (
judge_failed, orjudge_invalid_outputfor malformed output) instead of a score of 0. See Results and Aggregation for the classification rules. - Keep the grading evidence. Write the judge’s verdicts, reasons, and enough diagnostics into the result for users to check per-task scores.
- Document the judge on the Benchmark page. State what the judge does, its verdict format, the recommended judge Model, and any changed defaults, and link to the shared judge pages instead of repeating the shared parameters.
LLM Judge
Declare the Config
ListLLMJudgeConfig first among the config’s bases and call LLMJudgeConfig.__post_init__(self) from __post_init__; it runs the base config’s __post_init__ first:
LLMJudgeConfig provides:
You do not need to set an output limit:
generate_text and generate_with_metadata add max_tokens 32768 when neither the call nor judge_model.params sets one (see Default max_tokens). Pass max_tokens per call only when the Benchmark needs another limit; judge_model.params still override it.
Use the Built-In Scorer
When grading compares only the final answer, useLLMJudgeScorer with evaluate_binary_scorer. The helper calls the scorer, checks the correct field, and maps exceptions to a FATAL judge_failed:
A counts as correct. For your own verdict rules, pass prompt_template and response_parser to LLMJudgeScorer. The template can use the {question}, {gold_answer}, and {llm_response} placeholders. The parser receives the judge’s reply text and returns a bool; it should raise when it cannot parse the reply rather than return False.
Write a Custom Grading Flow
When rubrics, repeated sampling, or multi-stage grading do not fit a binary scorer, callgenerate_text or generate_with_metadata directly:
- The client handles
timeoutandmax_retriesfromjudge_model.params, and sends the other keys with the request, so the scorer does not need its own timeout and retry logic. - When you fan out requests with
asyncio.gather, the runtime already limits them tojudge_concurrency. - To tell a truncated reply from a short one, use
generate_with_metadataand checkstop_reason == "length".
Agentic Judge
AgentJudgeScorer starts a judge Harness in the evaluation Environment and owns the mechanics: building the judge request, splitting windows, running them concurrently, retrying, time limits, and media-inspect. As with LLMJudgeScorer, the Benchmark supplies only the prompt and the parser.
Declare the Config
AgentJudgeConfig is an LLMJudgeConfig that adds judge_harness, judge_harness_params, and every judge_agent_* parameter, validated together:
Preflight the Judge
Do not probe the judge yourself. Before inference, the runner probes every Model that the config’sjudge_requirements(req) returns, from the machine running AgentCompass, and stops the run on any failure, including an unreachable endpoint. See Modality Preflight.
Grade
build_unit(attempt, index, batch)returns aJudgeUnitfor one window: the prompt, the judge’s working directory, and the judgment file path. Give each window its own directory, and do not reuse directories across retry passes.parse(outcome, batch)turns aJudgeUnitOutcomeinto{item_id: verdict}. Whenoutcome.judgment_textis empty,outcome.erroris set, or the judgment misses items, return a failure marker for those items (such as{"passed": None, "error": ...}) instead of raising;outcome.failure_reason()explains the failure.is_judged(verdict)tells valid verdicts from failure markers. Unjudged items are re-judged in the next pass; an item that is never judged keeps its first failure marker.- Time limits. Call
evaluation_deadline()at the start ofevaluate(); it sits a margin before the runtime’s evaluation watchdog. Pass it toscorer.score(..., deadline=deadline)so every judge run is cut to the remaining budget. Then wrap the whole judging step incall_with_timeout(..., timeout=remaining_budget(deadline))as a backstop, and turn theTimeoutErrorit raises intojudge_failed.call_with_timeoutforwards positional arguments only, so bind keyword arguments withfunctools.partial. - Media. Put
scorer.media_instructions()in the judge prompt: it explains how to usemedia-inspect, and returns an empty string when nomedia-inspectis configured. Write one prompt for the judge’s fixed modalities; do not add text-only or no-audio variants. Ifmedia-inspectcannot be installed in the evaluation Environment,scorer.score()raises, so the task’s judging fails.scorer.media_summary()returnsffmpegavailability and call counts for the result’s diagnostics. - Installation. A
freshEnvironment has nothing from inference installed, so apreinstalledjudge Harness falls back toinstall_if_missingunless the judge params setinstall_strategyexplicitly.
Aggregate the Result
- If any item is still unjudged at the end, return a FATAL
judge_failedfor the task and no score. - A missing deliverable is a normal result. Tell the judge in the prompt to fail the items that depend on it, rather than recording a judge failure.
- Save one record per judge run (window name, covered items, error, and
outcome.diagnostics()), such asjudge_windowsin AA-Briefcase-Lite-AC. To keep the judge agent’s transcript, addoutcome.transcript(): one entry per step with the text, reasoning, tool calls, and tool outputs, and inline base64 media replaced by its size.
Hide Which Side Is Under Test
When the judge compares against a baseline or reference, present the evidence under neutral labels and paths. Do not reveal in the prompt, file names, or directory names which side comes from the Model under test. GDPval-AC presents the output under test and the fixed baseline as A and B.Reference Implementations
Before You Submit
- A missing
judge_modelfails with a clear message before inference (throughjudge_model_requiredor a preflight). - Judge failures, unparseable output, and timeouts all return a FATAL issue and are never scored as 0.
- Local tests check the parser on empty replies, malformed output, and missing items (keep test files under the ignored root
tests/directory). - The config sets
judge_modalities, the runner’s preflight covers every judge Model throughjudge_requirements(req), andevaluate()usesevaluation_deadline(). - The Benchmark page states the judge’s role, the recommended judge, and changed defaults, and links to the Judges overview.
