host_process). The harness first drives the model under test through the search loop to produce a final answer, then the judge model grades it. Judging is single-sided: the judge only compares the agent-under-test’s answer against the ground truth, without comparing to any baseline.
How it works
A GAIA run has two stages — inference and judging.Inference and judging
- Inference. The model under test acts as a search agent and, driven by the harness (default
naive_search_agent), completes multi-turn tool loops such as search / visit per task, producing a short natural-language final answer. - Judging. The judge model (
judge_model) receives “question + ground truth + answer under test” and grades it with the built-in A/B/C protocol. The judge compares only the final answer, ignoring reasoning and formatting differences; equivalent expressions are accepted. The judge and the model under test are two separate endpoints;judge_modelmust be specified explicitly.
The A/B/C verdict
The judge returns exactly one verdict, and only A counts as correct:- A — CORRECT: the answer semantically matches the ground truth (equivalent expressions and formatting allowed).
- B — INCORRECT: any deviation from the ground truth.
- C — INCOMPLETE / REPETITIVE / REFUSAL: an invalid answer (cut off mid-sentence, looping repetition, or an explicit refusal).
Parameters
Pass a JSON object via--benchmark-params '{...}', or a benchmark.params block in the YAML given to --config; the CLI wins on shared keys. See the Benchmark overview for merge precedence.
Parameter reference
| Parameter | Type | Default | Choices / values | Description |
|---|---|---|---|---|
judge_model | dict | null | id, base_url, api_key, api_protocol, params | Judge model spec, required (see Judge model spec). It decides grading, and is not the CLI —model-*. |
category | string / list | ”all" | "all”, 1, 2, 3 | Filter by GAIA difficulty level (1 = easiest to 3 = hardest); “all” = no filter, a list takes the union. Task counts by level — 1 (39), 2 (52), 3 (12); 103 in total. |
sample_ids follow Benchmark Parameters. Configure repeated attempts with --k and --attempt-strategy; see Metrics and Aggregation.
Judge model spec
judge_model is passed as a dict with the fields id, base_url, api_key, api_protocol, and params, pointing to the judge model’s own endpoint, with inference parameters under params.
We recommend fixing a single judge across all models under test. Grading directly decides the scores, so switching judges makes scores no longer comparable across models; likewise, the model under test should not serve as its own judge, as that is neither fair nor comparable. The judge need not be especially strong — the A/B/C criterion (semantic match) is relatively objective, so a mid-sized model suffices. AgentCompass recommends Qwen3.6-35B-A3B.
Run examples
The GAIA run command takes the formagentcompass run gaia <harness> <model>, whose three positional arguments are:
gaia— the benchmark id;<harness>— the harness that drives the model under test through the search loop, defaulting tonaive_search_agent; its own configuration is passed via--harness-params;<model>— the model under test, i.e. the agent that performs retrieval and answering; its access credentials are passed via--model-base-url/--model-api-key.
--benchmark-params for dataset and judge settings, --harness-params for agent and tool settings, and --execution-params for phase timeouts and multipliers. YAML uses benchmark.params, harness.params, and execution; explicit CLI values override YAML values.
In the examples below, --harness-params always passes the Serper and Jina keys required for retrieval directly via serper_api_key / jina_api_key (the search / visit tools of naive_search_agent depend on them); the examples differ only in --benchmark-params.
- Smoke test (single task end-to-end)
- Custom parameters
- AgentCompass recommended config
Use
sample_ids to evaluate a single task, verifying that the end-to-end inference and judging flow works; defaults for the rest.Outputs
A run writes per-task details and the aggregate viewssummary.md and metrics.json under the run directory.
Metric Contract and aggregate series
summary.md keeps the traditional metric and detail tables at k=1; at k>1, it shows the attempt plan plus headline and auxiliary series with independent Evaluated, Error, Unavailable, and Total counts. metrics.json preserves every series and breakdown.
The primary metric is binary correct. At k=1, correct.native@1 is the accuracy over evaluated observations and is true only when the judge returns verdict A. At k>1, the generic reducers can emit correct.avg@k and correct.pass@k, each with independent counts.
Per-task details (details/)
Each task has one JSON file. Its binary observation isattempts.<N>.metrics.correct, and the judge evidence for that attempt is recorded under attempts.<N>.meta.benchmark.scoring:
Only the parsed verdict is persisted here; the answer under test and ground truth are kept for tracing, while the full trajectory is written alongside in the same task file.