Skip to main content
An agentic judge is an agent that runs in the evaluation Environment. AgentCompass drives the judge Model with a judge Harness (openclaw by default). The judge opens, renders, or runs the deliverables the agent under test left behind, judges them item by item against the rubric, and writes its verdicts to a judgment file. This page covers the parameters and behavior shared by every agentic-judge Benchmark; see GDPval-AC and AA-Briefcase-Lite-AC for each Benchmark’s evidence bundle, rubric, and defaults. If the judge only needs to read a text answer, use the much cheaper LLM judge.

How It Works

  1. Prepare the evaluation Environment. After inference, the Benchmark grades in an evaluation Environment: either a new fresh Environment or the task Environment itself (reuse). It places the deliverables, reference files, and rubric in the judge’s working directory.
  2. Split the rubric into windows. A task’s rubric items are split into windows of judge_agent_rubric_window items, and each window goes to one judge run.
  3. Run judges concurrently. Up to judge_agent_concurrency judge runs proceed at once within a task. Each run writes its verdicts to its own judgment file, which the Benchmark parses.
  4. Re-judge unjudged items. A judge run that fails, times out, or writes an invalid judgment only affects the items it covered. After the first pass, every unjudged item is split into windows again and re-judged in a fresh judge session, for up to judge_agent_max_retries passes.
  5. Aggregate. A task that still has unjudged items after the retries is an evaluation failure (judge_failed), not a score of 0. A missing deliverable is a normal result: the judge fails the items that depend on it.

Models Involved

An agentic-judge run involves up to three Models: judge_model is the same parameter the LLM judge uses; only its role differs. An LLM judge calls the Model directly, while an agentic judge runs it as an agent on judge_harness. Connection fields omitted from judge_model or judge_agent_media.model fall back to the Model under test, so configure both completely to keep grading independent of the run.

How judge_model.params Applies

An agentic judge does not send requests through the shared LLM client, so the LLM judge defaults for timeout, max_retries, and the output limit do not apply to the judge Harness. judge_model.params reaches three places: The preflight sends only extra_headers and extra_query from judge_model.params, with a 60-second time limit and no retries. To limit the judge’s time, set judge_agent_timeout_seconds for one judge run; see Time Limits and Budget.

Parameters

Pass every parameter in --benchmark-params. The defaults below are the shared ones; a Benchmark can change them, and its own page is authoritative. Defaults changed by each Benchmark: judge_concurrency has no effect on an agentic judge; judge_agent_concurrency controls concurrency.

Judge Harness Params

AgentCompass assembles the judge Harness params in this order, each step overriding the previous ones:
  1. It starts from the judge Harness defaults. Only openclaw has them:
  2. When judge_harness is the same Harness as the one used for inference, it applies the run’s --harness-params, so the judge locates or installs the Harness the same way.
  3. It turns on image viewing when the judge inspects images: model_input for openclaw and view_image for stirrup.
  4. It writes judge_agent_max_turns to the Harness’s turn-limit param (max_turns or max_iterations).
  5. It turns on the Harness’s unattended-approval switch (such as dangerously_skip_permissions) so the judge runs without prompts.
  6. It applies judge_harness_params param by param: {"max_output_tokens": 64000} changes only max_output_tokens and keeps the other defaults. Any accepted name of a param replaces it, so the legacy {"max_tokens": 64000} has the same effect. To turn a default off, set it explicitly, such as {"setup_capsule_tag": ""} for an image that already has Node.js. These params may not contradict the image params from step 3: stirrup with {"view_image": false} is an error.
Only params the Harness actually declares are written. A fresh evaluation Environment starts from the image without anything installed during inference. So unless the judge params set install_strategy explicitly, preinstalled changes to install_if_missing, and the judge Harness is installed again in the evaluation Environment. openclaw installs through npm and needs Node.js, which the default setup_capsule_tag provides. See OpenClaw.

Modalities and the Judge Model

Each Benchmark fixes what its judge inspects when it is integrated; no parameter changes it. AA-Briefcase-Lite-AC and GDPval-AC inspect all four modalities: Text and images reach the judge Model through the judge Harness, so judge_harness must be one that can pass images: openclaw, stirrup, codex, or claude_code. The runner checks this, and that judge_harness_params do not contradict the image params, in preflight before inference starts. Harnesses cannot pass audio or video to the Model, so those go through media-inspect, and judge_agent_media is required: the default {} uses the judge Model, and null is a config error. The judge Model must accept images. The media-inspect Model must accept audio over the openai-chat protocol, and images too, because a video clip it rejects is sent again as frames:

Preflight

An agentic judge takes a long time, so before inference starts AgentCompass probes the judge Models from the machine running AgentCompass, as part of the Modality Preflight:
  1. It sends judge_model one text request to check the endpoint, key, and Model ID, then a small solid red image and asks for its color.
  2. It sends the Model that media-inspect uses (judge_agent_media.model, or the judge Model when unset) a text request and a short tone, which requires the openai-chat protocol. Because video can fall back to frames, it also sends that Model the red image.
Any failure stops the run with an error that names the setting to check: an endpoint this machine cannot reach, a rejected request, a Model that rejects the image or audio input, or one that cannot name the image’s color. A run without judge_model probes no judge.

media-inspect

When the Benchmark’s judge inspects audio or video, AgentCompass installs a media-inspect command in the evaluation Environment and explains its use in the judge prompt. The judge uses it to cut a clip of audio or video and ask a multimodal Model a specific question about it. media-inspect needs python3 in the evaluation Environment. With ffmpeg and ffprobe it can cut clips by time range; without them it can only inspect whole files up to max_request_mb. If media-inspect cannot be installed, for example because the evaluation Environment has no python3, that task’s judging fails.

Time Limits and Budget

  • The whole grading phase is bounded by evaluation_timeout_seconds, including every window and every retry pass.
  • Each judge run is limited to the smaller of judge_agent_timeout_seconds and the remaining evaluation budget; without judge_agent_timeout_seconds, it uses the remaining budget.
  • Judging stops shortly before the evaluation budget runs out (120 seconds or 10% of the budget, whichever is smaller), so a timeout is recorded as a judging failure instead of being cancelled by the runtime.
  • The evaluation phase runs at most task_concurrency × judge_agent_concurrency judge agents at once. If the judge endpoint times out or rate-limits, lower judge_agent_concurrency first.
See Timeouts for setting phase budgets.

Network

The judge Harness and media-inspect run inside the evaluation Environment. The judge endpoint, the media-inspect Model endpoint, and any package sources needed to install the judge Harness (such as npm) must all be reachable under evaluation_network_policy; see Network. The preflight runs from the host, so the judge and media-inspect Model endpoints must also be reachable from the machine running AgentCompass.

Tune Scheduling

Troubleshooting

Whether the judge agent’s transcript is saved depends on the Benchmark. See the “Evaluation Results” section of each Benchmark page for the grading evidence it keeps.