ResearchClawBench

ResearchClawBench (arxiv) evaluates whether an autonomous research agent can complete an end-to-end scientific study. For each task, the agent receives the research question, related work, and task data, then produces a publication-quality report at report/report.md. A separate judge model scores the report against a weighted checklist derived from the target study.

How it works

Inference and judging

• Prepare the task. AgentCompass downloads the dataset when necessary, creates an isolated workspace, uploads the task’s data/ and related_work/ materials, and writes the complete instructions.
• Run the research agent. A search-oriented agent such as ResearchHarness drives the model through research, coding, analysis, and report writing. The required artifact is report/report.md; figures may be written anywhere under the task workspace.
• Judge the report. The benchmark reads the report and generated images, then asks judge_model to score every text or image checklist item on a 0–100 scale. Image criteria compare generated figures with the target-study figure, so the judge must support image input when such criteria are present.

Checklist score

Each checklist item has a weight. The task’s scalar score is the weighted mean of all item scores, from 0 to 100. pass_threshold also produces the auxiliary binary metric passed, which is true only when the attempt status is completed, the scorer has no top-level error, and score reaches the threshold.

Parameters

Pass benchmark configuration with --benchmark-params '{...}', or place it under benchmark.params in the YAML supplied to --config; command-line values take precedence.

Parameter reference

Parameter	Type	Default	Choices / values	Description
judge_model	dict	null	id, base_url, api_key, api_protocol, params	Judge model spec, required. It scores the checklist and is separate from the model under test.
category	string / list	”all"	"all”, one category, or a list	Filter tasks by the category prefix in the task id; a list takes the union.
pass_threshold	float	50.0	0–100	Minimum weighted checklist score required for passed=true.
max_generated_images	int	5	integer ≥ 0	Maximum generated figures sent to the judge for each image checklist item.

Judge model spec

judge_model is a complete model spec with the fields id, base_url, api_key, api_protocol, and params. Put judge inference options under params. Use the same judge configuration across all models being compared; changing the judge changes the scoring standard. Because some checklist items include target and generated figures, choose a multimodal judge that supports the configured API protocol.

Run examples

Run configuration is split into two JSON blocks: --benchmark-params carries ResearchClawBench configuration (judge model, filtering, and scoring), while --harness-params carries ResearchHarness configuration (service credentials and execution limits). Both can also be written under benchmark.params and harness.params in --config.

The examples pass Serper, Jina, and MinerU credentials as environment-variable references. Set SERPER_API_KEY, JINA_API_KEY, and MINERU_TOKEN before running.

Smoke test (single task end-to-end)

Verify the end-to-end flow works — sample_ids selects which case to run, with all other parameters using their defaults.

agentcompass run \
  researchclawbench \
  researchharness \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…", "api_protocol": "openai-chat"},
    "sample_ids": ["<task-id>"]
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}",
    "mineru_token": "${MINERU_TOKEN}"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Custom parameters

Filter by research category, raise the pass threshold, and set explicit ResearchHarness execution limits.

agentcompass run \
  researchclawbench \
  researchharness \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…", "api_protocol": "openai-chat"},
    "category": "<category>",
    "pass_threshold": 60
  }' \
  --harness-params '{"max_rounds": 600, "llm_request_timeout_seconds": 1800, "serper_api_key": "${SERPER_API_KEY}", "jina_api_key": "${JINA_API_KEY}", "mineru_token": "${MINERU_TOKEN}"}' \
  --execution-params '{"run_timeout_seconds": 14400}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

AgentCompass recommended config

Run a full evaluation with the AgentCompass recommended configuration. Only the judge model and ResearchHarness service credentials must be provided explicitly.

agentcompass run \
  researchclawbench \
  researchharness \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…", "api_protocol": "openai-chat"}
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}",
    "mineru_token": "${MINERU_TOKEN}"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --task-concurrency 16

Outputs

A run writes per-task details and the aggregate files summary.md and metrics.json under the run directory.

Metric Contract

ResearchClawBench declares scalar primary metric score, the 0–100 weighted checklist score, and auxiliary binary metric passed. At k=1, it emits score.native@1 and passed.native@1. At k>1 with the avg execution strategy, it emits score.avg@k, passed.avg@k, and passed.pass@k. Selecting the pass execution strategy still fails during preflight because the primary metric is scalar; the auxiliary passed.pass@k series does not enable early stopping. Every emitted series has its own counts and category breakdowns.

Per-task details (details/)

Each task JSON stores score and passed under attempts.<N>.metrics, together with the final report answer, trajectory, and Harness artifacts. Checklist evidence is stored under attempts.<N>.meta.benchmark.scoring, including total_score, total_weight, pass_threshold, passed, and every item’s type, weight, score, reasoning, and error information.