Choose a Benchmark

Select a registered benchmark and configure its complete benchmark-parameter schema.

Benchmarks define what is evaluated. Each benchmark owns its dataset, stable task ids, task preparation, scoring logic, and aggregate metrics. Select the benchmark as the first positional argument to agentcompass run:

agentcompass run <benchmark> <harness> "$MODEL_NAME"

Find a Benchmark

Use the live registry to see the benchmarks available in your installed AgentCompass revision:

agentcompass list benchmark

The sidebar links to benchmarks with dedicated task, parameter, compatibility, and run documentation. For any registry entry, including a newly integrated benchmark without a dedicated page, inspect its current configuration schema with:

agentcompass config docs benchmark <benchmark-id>

Configure Benchmark Parameters

The General Run Parameter Reference introduces --benchmark-params <json>. The <json> value is one JSON object containing the complete parameter override for the selected benchmark:

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --benchmark-params '{
    "sample_ids": ["<task-id>"],
    "k": 1,
    "<benchmark-specific-field>": "<value>"
  }'

The accepted object combines two schemas:

benchmark params
  ├─ shared fields from RuntimeBenchmarkConfig
  └─ fields defined by the selected benchmark config

Shared Benchmark Fields

Every benchmark config derived from RuntimeBenchmarkConfig supports these user-facing fields:

Field	Type	Default	Meaning and when to change it
sample_ids	list[str] | null	null	Runs only the listed stable task ids. Use it for a smoke test, failed-task rerun, or a controlled subset. Unknown ids fail before execution.
k	int	1	Number of independent attempts per selected task. Keep 1 for one attempt per task; increase only when the evaluation protocol requires repeated sampling.
avgk	bool	true	Enables average-at-k aggregation when the benchmark exposes it. Keep the default unless reproducing a report with different aggregation.
aggregation_mode	"micro_weighted" | "category_mean"	"micro_weighted"	micro_weighted weights tasks equally; category_mean averages category-level results equally. Match the official metric definition.
category_hierarchy	object | null	null	Overrides grouped metric hierarchy. Leave unset unless the benchmark documentation defines the required object shape.

The model id is not part of this JSON object. It remains the third positional argument to agentcompass run and is injected into the benchmark config by the runtime.

Selected Benchmark Fields

Each benchmark extends the shared schema with only the fields it owns. Depending on the integration, these can include:

Field family	Examples	Use
Dataset identity	version, repo_revision, split, data_path	Select an official release, pinned revision, or local dataset source.
Task filters	category, language, occupation	Select benchmark-defined subsets in addition to exact sample_ids.
Preparation	prepare_mode, workspace_root	Control benchmark-owned repository or task preparation.
Evaluation	eval_timeout, verifier_timeout_multiplier, pass thresholds	Match official verifier and scoring behavior.
Judge configuration	judge_model, judge scheduling or rubric fields	Configure benchmark-owned model judging when required.

These examples are not a universal schema. Print the complete field list, types, defaults, and descriptions for the selected benchmark directly from the installed code:

agentcompass config docs benchmark <benchmark-id>

The corresponding benchmark page explains valid values, recommended settings, required credentials, and interactions that cannot be expressed by type and default alone.

Build the JSON Object

For example, swebench_verified combines shared attempt and task-selection fields with its own preparation and evaluator fields:

{
  "sample_ids": ["astropy__astropy-12907"],
  "k": 1,
  "prepare_mode": "prebaked",
  "workspace_root": "/testbed",
  "eval_timeout": 1800
}

This expanded object demonstrates ownership; it is not a recommendation to repeat defaults in every command. Pass only the fields that must differ from the selected benchmark’s effective configuration.

--benchmark-params must be valid JSON, so keys and string values use double quotes. CLI values override matching keys from benchmark.params in configuration files. Inspect the merged built-in and configuration-file values before adding the final CLI override:

agentcompass config show \
  --benchmark <benchmark-id> \
  --config <config-file>

Images and Provider Settings

Heavyweight benchmarks usually attach task images, workspace roots, and resource hints to task metadata. Compatible recipes translate those requirements for Docker, Daytona, or Modal. Keep provider image, resource, and network overrides in --env-params; they are not benchmark parameters.