openclaw by default). The judge opens, renders, or runs the deliverables the agent under test left behind, judges them item by item against the rubric, and writes its verdicts to a judgment file. This page covers the parameters and behavior shared by every agentic-judge Benchmark; see GDPval-AC and AA-Briefcase-Lite-AC for each Benchmark’s evidence bundle, rubric, and defaults.
If the judge only needs to read a text answer, use the much cheaper LLM judge.
How It Works
- Prepare the evaluation Environment. After inference, the Benchmark grades in an evaluation Environment: either a new
freshEnvironment or the task Environment itself (reuse). It places the deliverables, reference files, and rubric in the judge’s working directory. - Split the rubric into windows. A task’s rubric items are split into windows of
judge_agent_rubric_windowitems, and each window goes to one judge run. - Run judges concurrently. Up to
judge_agent_concurrencyjudge runs proceed at once within a task. Each run writes its verdicts to its own judgment file, which the Benchmark parses. - Re-judge unjudged items. A judge run that fails, times out, or writes an invalid judgment only affects the items it covered. After the first pass, every unjudged item is split into windows again and re-judged in a fresh judge session, for up to
judge_agent_max_retriespasses. - Aggregate. A task that still has unjudged items after the retries is an evaluation failure (
judge_failed), not a score of 0. A missing deliverable is a normal result: the judge fails the items that depend on it.
Models Involved
An agentic-judge run involves up to three Models:judge_model is the same parameter the LLM judge uses; only its role differs. An LLM judge calls the Model directly, while an agentic judge runs it as an agent on judge_harness. Connection fields omitted from judge_model or judge_agent_media.model fall back to the Model under test, so configure both completely to keep grading independent of the run.
How judge_model.params Applies
An agentic judge does not send requests through the shared LLM client, so the LLM judge defaults for timeout, max_retries, and the output limit do not apply to the judge Harness. judge_model.params reaches three places:
The preflight sends only
extra_headers and extra_query from judge_model.params, with a 60-second time limit and no retries.
To limit the judge’s time, set judge_agent_timeout_seconds for one judge run; see Time Limits and Budget.
Parameters
Pass every parameter in--benchmark-params. The defaults below are the shared ones; a Benchmark can change them, and its own page is authoritative.
Defaults changed by each Benchmark:
judge_concurrency has no effect on an agentic judge; judge_agent_concurrency controls concurrency.
Judge Harness Params
AgentCompass assembles the judge Harness params in this order, each step overriding the previous ones:-
It starts from the judge Harness defaults. Only
openclawhas them: -
When
judge_harnessis the same Harness as the one used for inference, it applies the run’s--harness-params, so the judge locates or installs the Harness the same way. -
It turns on image viewing when the judge inspects images:
model_inputforopenclawandview_imageforstirrup. -
It writes
judge_agent_max_turnsto the Harness’s turn-limit param (max_turnsormax_iterations). -
It turns on the Harness’s unattended-approval switch (such as
dangerously_skip_permissions) so the judge runs without prompts. -
It applies
judge_harness_paramsparam by param:{"max_output_tokens": 64000}changes onlymax_output_tokensand keeps the other defaults. Any accepted name of a param replaces it, so the legacy{"max_tokens": 64000}has the same effect. To turn a default off, set it explicitly, such as{"setup_capsule_tag": ""}for an image that already has Node.js. These params may not contradict the image params from step 3:stirrupwith{"view_image": false}is an error.
fresh evaluation Environment starts from the image without anything installed during inference. So unless the judge params set install_strategy explicitly, preinstalled changes to install_if_missing, and the judge Harness is installed again in the evaluation Environment. openclaw installs through npm and needs Node.js, which the default setup_capsule_tag provides. See OpenClaw.
Modalities and the Judge Model
Each Benchmark fixes what its judge inspects when it is integrated; no parameter changes it. AA-Briefcase-Lite-AC and GDPval-AC inspect all four modalities:
Text and images reach the judge Model through the judge Harness, so
judge_harness must be one that can pass images: openclaw, stirrup, codex, or claude_code. The runner checks this, and that judge_harness_params do not contradict the image params, in preflight before inference starts. Harnesses cannot pass audio or video to the Model, so those go through media-inspect, and judge_agent_media is required: the default {} uses the judge Model, and null is a config error.
The judge Model must accept images. The media-inspect Model must accept audio over the openai-chat protocol, and images too, because a video clip it rejects is sent again as frames:
Preflight
An agentic judge takes a long time, so before inference starts AgentCompass probes the judge Models from the machine running AgentCompass, as part of the Modality Preflight:- It sends
judge_modelone text request to check the endpoint, key, and Model ID, then a small solid red image and asks for its color. - It sends the Model that
media-inspectuses (judge_agent_media.model, or the judge Model when unset) a text request and a short tone, which requires theopenai-chatprotocol. Because video can fall back to frames, it also sends that Model the red image.
judge_model probes no judge.
media-inspect
When the Benchmark’s judge inspectsaudio or video, AgentCompass installs a media-inspect command in the evaluation Environment and explains its use in the judge prompt. The judge uses it to cut a clip of audio or video and ask a multimodal Model a specific question about it.
media-inspect needs python3 in the evaluation Environment. With ffmpeg and ffprobe it can cut clips by time range; without them it can only inspect whole files up to max_request_mb. If media-inspect cannot be installed, for example because the evaluation Environment has no python3, that task’s judging fails.
Time Limits and Budget
- The whole grading phase is bounded by
evaluation_timeout_seconds, including every window and every retry pass. - Each judge run is limited to the smaller of
judge_agent_timeout_secondsand the remaining evaluation budget; withoutjudge_agent_timeout_seconds, it uses the remaining budget. - Judging stops shortly before the evaluation budget runs out (120 seconds or 10% of the budget, whichever is smaller), so a timeout is recorded as a judging failure instead of being cancelled by the runtime.
- The evaluation phase runs at most
task_concurrency × judge_agent_concurrencyjudge agents at once. If the judge endpoint times out or rate-limits, lowerjudge_agent_concurrencyfirst.
Network
The judge Harness andmedia-inspect run inside the evaluation Environment. The judge endpoint, the media-inspect Model endpoint, and any package sources needed to install the judge Harness (such as npm) must all be reachable under evaluation_network_policy; see Network. The preflight runs from the host, so the judge and media-inspect Model endpoints must also be reachable from the machine running AgentCompass.
Tune Scheduling
Troubleshooting
Whether the judge agent’s transcript is saved depends on the Benchmark. See the “Evaluation Results” section of each Benchmark page for the grading evidence it keeps.
