BrowseComp-ZH

BrowseComp-ZH (arxiv) evaluates a browsing agent’s web browsing ability in Chinese: given a question whose answer is a hard-to-find fact on the Chinese-language web — one that a single search cannot answer and that requires persistent, multi-step browsing to pin down — the agent produces a short final answer, which an LLM judge then grades as correct or not against the ground truth.

Like DeepSearchQA, BrowseComp-ZH uses single-sided judging. The judge only compares the agent-under-test’s answer against the ground truth, without comparing to any baseline. Both inference and judging run in the local process (host_process) — the harness first drives the model under test through the search loop to produce a final answer, then the judge model grades it.

How it works

A BrowseComp-ZH run has two stages — inference and judging.

Inference and judging

• Inference. The model under test acts as a search agent and, driven by the harness (default naive_search_agent), completes multi-turn tool loops such as search / visit per task, producing a short natural-language final answer.
• Judging. The judge model (judge_model) receives “question + ground truth + answer under test” and grades it with the built-in A/B/C protocol. The judge compares only the final answer, ignoring reasoning and formatting differences; equivalent expressions are accepted. The judge and the model under test are two separate endpoints; judge_model must be specified explicitly.

The A/B/C verdict

The judge returns exactly one verdict, and only A counts as correct:

• A — CORRECT: the answer semantically matches the ground truth (equivalent expressions and formatting allowed).
• B — INCORRECT: any deviation from the ground truth.
• C — INCOMPLETE / REPETITIVE / REFUSAL: an invalid answer (cut off mid-sentence, looping repetition, or an explicit refusal).

Parameters

Pass a JSON object via --benchmark-params '{...}', or a benchmark.params block in the YAML given to --config; the CLI wins on shared keys. See the Benchmark overview for merge precedence.

Parameter reference

Parameter	Type	Default	Choices / values	Description
judge_model	dict	null	id, base_url, api_key, api_protocol, params	Judge model spec, required (see Judge model spec). It decides grading, and is not the CLI —model-*.
category	string / list	”all"	"all”, 影视, 艺术, 地理, 音乐, 历史, 医学, 电子游戏, 科技, 体育, 政策法规, 学术论文	Filter tasks by category; “all” = no filter, a list takes the union. Task counts by category — 影视 (45), 艺术 (40), 地理 (37), 音乐 (32), 历史 (29), 医学 (26), 电子游戏 (23), 科技 (22), 体育 (18), 政策法规 (10), 学术论文 (7); 289 in total.

Shared Benchmark fields such as sample_ids follow Benchmark Parameters. Configure repeated attempts with --k and --attempt-strategy; see Metrics and Aggregation.

Judge model spec

judge_model is passed as a dict with the fields id, base_url, api_key, api_protocol, and params, pointing to the judge model’s own endpoint, with inference parameters under params.

We recommend fixing a single judge across all models under test. Grading directly decides the scores, so switching judges makes scores no longer comparable across models; likewise, the model under test should not serve as its own judge, as that is neither fair nor comparable. The judge need not be especially strong — the A/B/C criterion (semantic match) is relatively objective, so a mid-sized model suffices. AgentCompass recommends Qwen3.6-35B-A3B.

Run examples

The BrowseComp-ZH run command has this form:

agentcompass run browsecomp_zh <harness> <model>

Its three positional arguments are:

• browsecomp_zh — the benchmark id;
• <harness> — the harness that drives the model under test through the search loop, defaulting to naive_search_agent; its own configuration is passed via --harness-params;
• <model> — the model under test, i.e. the agent that performs retrieval and answering; its access credentials are passed via --model-base-url / --model-api-key.

Use --benchmark-params for dataset and judge settings, --harness-params for agent and tool settings, and --execution-params for phase timeouts and multipliers. YAML uses benchmark.params, harness.params, and execution; explicit CLI values override YAML values.

In the examples below, --harness-params always passes the Serper and Jina keys required for retrieval directly via serper_api_key / jina_api_key (the search / visit tools of naive_search_agent depend on them); the examples differ only in --benchmark-params.

Smoke test (single task end-to-end)

Use sample_ids to evaluate a single task, verifying that the end-to-end inference and judging flow works; defaults for the rest.

agentcompass run \
  browsecomp_zh \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "judge_model": {"id": "Qwen3.6-35B-A3B", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…"},
    "sample_ids": ["1"]
  }' \
  --harness-params '{
    "serper_api_key": "your-serper-key",
    "jina_api_key": "your-jina-key"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Custom parameters

Evaluate only a subset of categories to focus analysis on a specific domain; also demonstrates narrowing the toolset and iterations in --harness-params.

agentcompass run \
  browsecomp_zh \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "judge_model": {"id": "Qwen3.6-35B-A3B", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…"},
    "category": ["science", "geography"]
  }' \
  --harness-params '{
    "tools": ["search", "visit"],
    "max_iterations": 40,
    "serper_api_key": "your-serper-key",
    "jina_api_key": "your-jina-key"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 16

AgentCompass recommended config

Evaluate the full task set. --benchmark-params only needs the judge model judge_model; use --task-concurrency to raise cross-task concurrency.

agentcompass run \
  browsecomp_zh \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "judge_model": {"id": "Qwen3.6-35B-A3B", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…"}
  }' \
  --harness-params '{
    "serper_api_key": "your-serper-key",
    "jina_api_key": "your-jina-key"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 16

Outputs

A run writes per-task details and the aggregate views summary.md and metrics.json under the run directory.

Metric Contract and aggregate series

summary.md keeps the traditional metric and detail tables at k=1; at k>1, it shows the attempt plan plus headline and auxiliary series with independent Evaluated, Error, Unavailable, and Total counts. metrics.json preserves every series and breakdown.

The primary metric is binary correct. At k=1, the headline series correct.native@1 is the accuracy over evaluated observations: it is true only when the judge returns verdict A. At k>1, the generic reducers can emit correct.avg@k and correct.pass@k, each with independent counts.

Per-task details (details/)

Each task has one JSON file. Its binary observation is attempts.<N>.metrics.correct, and the judge evidence for that attempt is recorded under attempts.<N>.meta.benchmark.scoring:

Field	Meaning
evaluation_type	Fixed as llm_judge
correct	Whether the task is finally judged correct (judge verdict A)
model_answer	The final answer produced by the model under test
ground_truth	The reference answer

Only the parsed verdict is persisted here; the answer under test and ground truth are kept for tracing, while the full trajectory is written alongside in the same task file.