ResearchClawBench

ResearchClawBench (arxiv) evaluates whether an autonomous research agent can complete an end-to-end scientific study. For each task, the agent receives the research question, related work, and task data, then produces a publication-quality report at report/report.md. A separate judge model scores the report against a weighted checklist derived from the target study.

How it works

Inference and judging

• Prepare the task. AgentCompass downloads the dataset when necessary, creates an isolated workspace, uploads the task’s data/ and related_work/ materials, and writes the complete instructions.
• Run the research agent. A search-oriented agent such as ResearchHarness drives the model through research, coding, analysis, and report writing. The required artifact is report/report.md; figures may be written anywhere under the task workspace.
• Judge the report. The benchmark reads the report and generated images, then asks judge_model to score every text or image checklist item on a 0–100 scale. Image criteria compare generated figures with the target-study figure, so the judge must support image input when such criteria are present.

For Docker, Daytona, and Modal, a built-in recipe supplies the ResearchHarness runner image when no image, snapshot, or named image is explicitly configured. User-provided provider artifacts retain precedence. See Recipes.

Checklist score

Each checklist item has a weight. The task score is the weighted mean of all item scores, from 0 to 100. A task is marked correct when its score is at least pass_threshold and the harness run completed without error. The aggregate metric is mean_score, averaged across evaluated tasks.

Parameters

Pass benchmark configuration with --benchmark-params '{...}', or place it under benchmark.params in the YAML supplied to --config; command-line values take precedence.

Parameter reference

Parameter	Type	Default	Choices / values	Description
judge_model	dict	null	{id, base_url, api_key, api_protocol, params}	Judge model spec, required. It scores the checklist and is separate from the model under test.
category	string / list	”all"	"all”, one category, or a list	Filter tasks by the category prefix in the task id; a list takes the union.
pass_threshold	float	50.0	0–100	Minimum weighted checklist score required for correct=true.
max_generated_images	int	5	integer ≥ 0	Maximum generated figures sent to the judge for each image checklist item.

Judge model spec

judge_model is a complete model spec: {"id","base_url","api_key","api_protocol","params"}. Put judge inference options under params. Use the same judge configuration across all models being compared; changing the judge changes the scoring standard. Because some checklist items include target and generated figures, choose a multimodal judge that supports the configured API protocol.

Run examples

Run configuration is split into two JSON blocks: --benchmark-params carries ResearchClawBench configuration (judge model, filtering, and scoring), while --harness-params carries ResearchHarness configuration (service credentials and execution limits). Both can also be written under benchmark.params and harness.params in --config.

The examples pass Serper, Jina, and MinerU credentials as environment-variable references. Set SERPER_API_KEY, JINA_API_KEY, and MINERU_TOKEN before running.

Smoke test (single task end-to-end)

Verify the end-to-end flow works — sample_ids selects which case to run, with all other parameters using their defaults.

agentcompass run \
  researchclawbench \
  researchharness \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…", "api_protocol": "openai-chat"},
    "sample_ids": ["<task-id>"]
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}",
    "mineru_token": "${MINERU_TOKEN}"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Custom parameters

Filter by research category, raise the pass threshold, and set explicit ResearchHarness execution limits.

agentcompass run \
  researchclawbench \
  researchharness \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…", "api_protocol": "openai-chat"},
    "category": "<category>",
    "pass_threshold": 60
  }' \
  --harness-params '{
    "max_rounds": 600,
    "timeout": 14400,
    "llm_request_timeout_seconds": 1800,
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}",
    "mineru_token": "${MINERU_TOKEN}"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

AgentCompass recommended config

Run a full evaluation with the AgentCompass recommended configuration. Only the judge model and ResearchHarness service credentials must be provided explicitly.

agentcompass run \
  researchclawbench \
  researchharness \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "judge_model": {"id": "your-judge-model", "base_url": "https://your-judge-endpoint/v1", "api_key": "sk-…", "api_protocol": "openai-chat"}
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}",
    "mineru_token": "${MINERU_TOKEN}"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --task-concurrency 16

Outputs

A run writes aggregate metrics and per-task details under results/researchclawbench/<model>/<run>/.

Aggregate metrics (summary.md)

summary.md contains the run counts (Total, Evaluated, and Error) and the headline metric mean_score: the arithmetic mean of the 0–100 weighted checklist score for all evaluated tasks. Category-level mean scores are included when categories are present.

Per-task details (details/)

Each task JSON records the task score, correct verdict, final report answer, trajectory, and harness artifacts. The checklist grading is stored under attempts[*].meta.scoring, including total_score, total_weight, and every item’s type, weight, score, reasoning, and error information.