DeepResearch Bench

DeepResearch Bench (arxiv) evaluates a deep-research agent’s ability to write research reports: given an open-ended research query that requires web search and multi-step evidence gathering, the agent produces a complete Markdown research report, which is then graded by two frameworks — RACE ** for report quality and ** FACT ** for citation grounding. The dataset contains ** 100 tasks (50 Chinese, 50 English), written by domain experts and spanning 22 topics.

How it works

A DeepResearch Bench run has two stages — inference and scoring. The scoring stage comprises two mutually independent frameworks, RACE and FACT; use metrics to select one or both.

Inference and scoring

• Inference. The model under test acts as a research agent and, driven by the harness (default naive_search_agent), completes multi-turn tool loops such as search / visit per task, ultimately producing a Markdown research report as its answer for that task.
• Scoring. For RACE, the judge model (judge_model) compares the report under test against a reference report criterion by criterion to grade report quality; for FACT, fact_judge_model works with Jina Reader to fetch the cited pages and check whether the citations in the report support their claims. The judge and the model under test are two separate endpoints; judge_model must be specified explicitly.

Upstream ships an evaluator only and places no constraints on the inference side (no limits on tools, turns, or length); its leaderboard entries come from the real outputs of various deep-research products. Scores from this benchmark are therefore only comparable across runs that share the same harness, the same harness settings, and the same judge model — record all three alongside any number you cite.

The appended citation-format requirement

FACT can only verify citations that a report actually writes out, and an agent given only a query usually produces a report with no URLs anywhere — such a report scores zero on FACT, which says nothing about its real citation ability. So when require_citations is at its default of true, a citation-format requirement is appended after the query (one Chinese version and one English version, selected by the task’s language):

Deliver a complete research report in Markdown.
- Cite every non-obvious claim inline as [source title](https://source-url), placed immediately
  after the sentence it supports, using the real URL you retrieved it from.
- End the report with a numbered list of the sources you cited.

This requirement is appended only to the prompt sent to the model under test; the RACE judge always reads the bare query, so instruction_following is graded against the task’s own requirements rather than the one appended here. [title](url) is also one of the four citation forms natively supported by upstream’s extractor, not a format introduced by this integration. Set require_citations: false to fall back to upstream behaviour and send the bare query only.

RACE: reference-relative scoring

RACE gives no absolute score. Each task ships with a reference report written by a strong deep-research product, plus a weighted criteria tree. Scoring proceeds in two steps:

• Cleaning. Citation markers, reference lists, and footnotes are first removed from the report under test so the judge compares prose rather than bibliographies. Reports too long for a single call are split at paragraph boundaries and cleaned in parallel. The reference reports ship pre-cleaned and need no reprocessing. Set skip_cleaning: true to skip this step and grade the raw report.
• Judging. Within a single call, the judge scores both reports 0-10 against each criterion. Per-criterion scores are first folded into four dimension scores using the criterion weights, then composed into a task total using the dimension weights.

The number finally reported is the ratio target / (target + reference): 0.5 means it tied the reference report, above 0.5 means it beat the reference report, and below 0.5 means it lost to the reference report. The four dimensions — comprehensiveness, insight, instruction_following, and readability — are reported as the same ratio. A task whose judge never returns usable JSON within max_retries is recorded as eval_error; an absent primary score falls back to 0, and the independent error count preserves the diagnostic.

FACT: citation grounding

FACT checks whether every citation in the report truly supports the claim it accompanies. All four stages run against the raw report, citation markers intact:

• Extract. (fact, ref_idx, url) triples are pulled out of the report body; all four citation forms — [title](url), [15], text 15, and [15†L10] — are recognised.
• Deduplicate. Triples are grouped by URL; near-identical statements within a group are collapsed into one.
• Scrape. Each unique URL is fetched through Jina Reader. Fetched pages are cached under the AgentCompass data root and reused across runs (scrape_cache).
• Validate. Each statement is labelled supported, unsupported, or unknown against the fetched page.

Two exclusion rules match upstream: statements labelled unknown (dead link, paywall, page not found) leave both the numerator and the denominator; a report from which no citation could be extracted is dropped from the FACT averages entirely rather than scored zero.

Parameters

Pass a JSON object via --benchmark-params '{...}', or configure the fields under benchmarks.deepresearch_bench in a YAML file given to --config; explicit CLI values win on shared keys. See the Benchmark overview for merge precedence.

Parameter reference

Parameter	Type	Default	Choices / values	Description
judge_model	dict	null	id, base_url, api_key, api_protocol, params	Judge model spec, required (see Judge model spec). It decides RACE grading, and is not the CLI —model-*; it also serves as the default model for the cleaning and FACT stages.
metrics	list	[“race”, “fact”]	race, fact, or both	Which scoring frameworks to run. Both by default, matching upstream’s run_benchmark.sh; set it to [“race”] when you only want report quality.
jina_api_key	string	$JINA_API_KEY	Jina Reader key	Used by FACT to fetch cited pages. Required unless metrics is [“race”]; a missing key fails at config-building time.
fact_judge_model	dict	null	same as judge_model	Judge for the FACT stages; falls back to judge_model when unset.
cleaning_model	dict	null	same as judge_model	Model that performs cleaning before judging; falls back to judge_model when unset.
language	string	”all”	all / zh / en	Filter tasks by query language; all = no filter. 50 tasks each in Chinese and English.
category	string / list	”all"	"all”, a single topic name, or a list of topic names (22 listed below)	Filter tasks by topic; “all” = no filter. A list takes the union.
limit	int	0	0 = no limit	Run only the first N tasks after language and category filtering. Prefer sample_ids for a stable smoke-test selection.
data_dir	string	""	local repository or data/ directory	Use an existing DeepResearch Bench checkout instead of downloading the default archive.
dataset_zip_url	string	official repository archive	ZIP URL	Advanced dataset-source override used only when data_dir is empty.
require_citations	bool	true	true / false	Whether to append the citation-format requirement after the query (see The appended citation-format requirement). With false, only the bare query is sent and FACT usually has nothing to verify.
skip_cleaning	bool	false	true / false	Skip cleaning and grade the raw report. Saves one LLM call per task, but shifts the scores.
pass_threshold	float	0.5	0.0-1.0	Minimum active primary score required for passed=true. With RACE enabled, the default means “tied or beat the reference report”; in a FACT-only run, it means citation accuracy of at least 50%.
max_retries	int	10	≥ 1	Retry budget for one RACE judge call, covering both unparsable JSON and missing dimensions.
scrape_cache	bool	true	true / false	Whether to cache fetched pages under the data root and reuse them across runs.
max_urls	int	0	0 = no limit	Cap on unique URLs verified per task. A non-zero value bounds cost but drops some citations; the drop is logged.
max_url_content_chars	int	0	0 = no truncation	Truncate each fetched page to this length before validation.
clean_concurrency	int	4	≥ 1	Concurrent cleaning calls within one task; only takes effect when a long report is chunked. Cross-task concurrency is controlled by —task-concurrency.
scrape_concurrency	int	4	≥ 1	Concurrent Jina Reader fetches within one task.
fact_llm_concurrency	int	4	≥ 1	Concurrent FACT judge calls within one task, covering the extract, deduplicate, and validate stages.

Shared Benchmark fields such as sample_ids follow Benchmark Parameters. DeepResearchBench declares scalar primary metric score, auxiliary binary metric passed, and RACE dimensions and citation statistics as auxiliary scalar observations. At k>1, use the avg execution strategy; selecting pass for a scalar primary fails preflight. See Metrics and Aggregation.

All 22 category values (click to expand)

Science & Technology (16), Finance & Business (14), Software Development (10), Education & Jobs (8), Health (8), Literature (4), History (4), Hardware (4), Industrial (4), Art & Design (4), Games (2), Crime & Law (2), Entertainment (2), Sports & Fitness (2), Software (2), Transportation (2), Religion (2), Home & Hobbies (2), Travel (2), Food & Dining (2), Fashion & Beauty (2), Social Life (2). Numbers in parentheses are the task count per topic (100 total, half Chinese and half English). Case and spacing must match exactly.

Judge model spec

judge_model is passed as a dict with the fields id, base_url, api_key, api_protocol, and params, pointing to the judge model’s own endpoint, with inference parameters under params.

We recommend fixing a single judge ** across all models under test. RACE grading is the single biggest factor on the score, so switching judges makes scores no longer comparable across models; likewise, the model under test should not serve as its own judge, as that is neither fair nor comparable. Unlike benchmarks with relatively objective criteria such as DeepSearchQA, the RACE judge also needs a ** large enough context window: one judging call must hold two complete research reports plus the full criteria list, routinely exceeding 100k tokens; a judge that rejects the request outright will burn the whole retry budget, and the task ends up recorded as an error.

Choose one capable long-context judge and keep it fixed across the models under test. Upstream’s leaderboard uses gpt-5.5 for RACE and gpt-5.4-mini for FACT, so numbers produced with a different judge are internally comparable but cannot be aligned directly with that leaderboard.

Run examples

The DeepResearch Bench run command has this form:

agentcompass run deepresearch_bench <harness> <model>

Its three positional arguments are:

• deepresearch_bench — the benchmark id;
• <harness> — the harness that drives the model under test to write the report. naive_search_agent is the default; its own configuration is passed via --harness-params;
• <model> — the model under test, i.e. the agent that performs retrieval and writing; its access credentials are passed via --model-base-url / --model-api-key.

Use --benchmark-params for dataset and judge settings, --harness-params for agent and tool settings, and --execution-params for phase timeouts and multipliers. YAML uses benchmark.params, harness.params, and execution; explicit CLI values override YAML values.

jina_api_key appears once in each of the two blocks, for different purposes: in --harness-params it powers the agent’s visit tool for reading web pages, and in --benchmark-params it powers FACT’s fetching of cited pages for verification. The same key can be used for both, but FACT cannot run without the latter.

Smoke test (single task end-to-end)

Use sample_ids to evaluate a single task, verifying that the end-to-end inference, RACE, and FACT flow works; defaults for the rest.

agentcompass run \
  deepresearch_bench \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…"},
    "jina_api_key": "your-jina-key",
    "sample_ids": ["1"]
  }' \
  --harness-params '{
    "serper_api_key": "your-serper-key",
    "jina_api_key": "your-jina-key"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Custom parameters

Evaluate only the Chinese tasks and skip FACT, to focus on report quality; also demonstrates using a local dataset copy to skip the download. With metrics set to ["race"], no Jina key is required.

agentcompass run \
  deepresearch_bench \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "metrics": ["race"],
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…"},
    "language": "zh",
    "data_dir": "/path/to/deep_research_bench"
  }' \
  --harness-params '{
    "serper_api_key": "your-serper-key",
    "jina_api_key": "your-jina-key"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 8

AgentCompass recommended config

Report tasks search more broadly than QA ones, so max_iterations is widened from the default 50 to 80 and max_tool_response_length raised to 16384 so source URLs in the visit tool’s summaries are not truncated; the whole report has to be written in one generation, and its length cap comes from max_tokens in --model-params. fact_judge_model points at the same judge as judge_model; it is spelled out because the FACT stages make many calls with a short context each, so they can be moved to a cheaper model on their own (left unset, it falls back to judge_model).

agentcompass run \
  deepresearch_bench \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…"},
    "fact_judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…"},
    "jina_api_key": "your-jina-key"
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}",
    "max_iterations": 80,
    "max_tool_response_length": 16384
  }' \
  --model-params '{"max_tokens": 32768}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 8

Outputs

A run writes per-task details and the aggregate views summary.md and metrics.json under the run directory.

Metric Contract and aggregate series

summary.md keeps the traditional metric and detail tables at k=1; at k>1, it shows the attempt plan plus headline and auxiliary series with independent Evaluated, Error, Unavailable, and Total counts. metrics.json preserves every series and breakdown.

DeepResearchBench declares scalar primary metric score: it equals overall_score when RACE is enabled, or the task’s citation_accuracy for a FACT-only run. It also declares binary auxiliary metric passed and auxiliary scalar observations:

Metric	Meaning
score	Primary score selected from the active scoring mode
passed	Whether the attempt status is completed and its primary score reaches pass_threshold
overall_score	RACE total composed with the dimension weights
comprehensiveness	Coverage and completeness
insight	Depth of analysis
instruction_following	Adherence to the query’s explicit requirements
readability	Structure and writing quality
citation_accuracy	Supported citations divided by checked citations for that task
citations_checked	Citations that reached a verdict
citations_supported	Checked citations judged supported
citations_total	Citations extracted before checking
fact_scored	Numeric flag copied from a non-error FACT result to show whether that task was scored; omitted when FACT reports an error

FACT aggregation also derives avg_citations as total checked citations divided by scored FACT tasks, and avg_effective_citations as total supported citations divided by scored FACT tasks.

At k=1, every declared observation is emitted as a native series, including score.native@1 and passed.native@1. On a non-completed attempt, existing valid observations are preserved and absent scalar/binary observations fall back to 0/false; the task is counted as both evaluated and error. A completed attempt with no observation remains unavailable. At k>1, the avg execution strategy emits score.avg@k, passed.avg@k, and passed.pass@k alongside the auxiliary scalar averages. Selecting the pass execution strategy still fails during preflight because the primary metric is scalar; the auxiliary passed.pass@k series does not enable early stopping. RACE and FACT observations may be missing for different tasks, so each series reports its own evaluated, error, and unavailable counts; do not infer one shared denominator. overall_score remains RACE-only and is not derivable by averaging the four displayed dimensions because weighting occurs before normalization.

Per-task details (details/)

Each task has one JSON file. score, passed, and the auxiliary scalar observations are stored in attempts.<N>.metrics, while RACE and FACT evidence is recorded under attempts.<N>.meta.benchmark.scoring for item-by-item tracing. The scoring metadata also records the effective pass_threshold and, when a primary score is available, the passed verdict:

Field	Meaning
race.overall_score and the four dimensions	The task’s relative scores
race.raw	Pre-normalization weighted sums for both reports, plus any criterion matched only via fuzzy matching
race.cleaning	applied or skipped
race.article_chars / cleaned_article_chars	Report length before and after cleaning. A moderate drop is normal; a collapse means the cleaning call was truncated
race.judge_attempts	How many judge calls this task actually consumed
race.error	Why the task has no RACE score (empty_answer / cleaning_failed / judge_failed, etc.)
fact.n_citations / unique_urls	Extracted citation triples and distinct URLs after deduplication
fact.citations_checked / citations_supported	Statements that reached a verdict, and how many of them were supported
fact.scrape_failures / validate_failures	Unreachable pages, and validation calls that never returned usable JSON
fact.citations	Per-URL statements, verdicts, and errors, for tracing a score back to a specific page
fact.error	no_citations_found (task excluded from the FACT averages) / extraction_failed / empty_answer