> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks

> Dataset loading, task selection, task preparation, scoring, and aggregation.

Benchmarks define what is being evaluated. They own the dataset, task ids, task filtering, material preparation, scoring logic, and aggregate metrics.

They should stay provider-neutral. If a benchmark task needs a Docker image, snapshot, workspace root, or provider resource hint, put that in task metadata and let recipes adapt it to the selected environment.

## What Benchmarks Own

<CardGroup cols={2}>
  <Card title="Task loading" icon="list-checks">
    Load `TaskSpec` objects from datasets, local files, services, or benchmark-specific repositories.
  </Card>

  <Card title="Task selection" icon="filter">
    Apply generic `sample_ids` filtering and benchmark-specific parameters such as category or split.
  </Card>

  <Card title="Task preparation" icon="package-open">
    Turn a raw task into a `PreparedTask` with prompt, media, files, tools, workspace, and metadata.
  </Card>

  <Card title="Evaluation" icon="badge-check">
    Score `RunResult`, write per-task detail payloads, and aggregate metrics into `summary.md`.
  </Card>
</CardGroup>

Benchmarks do not run agents, call model APIs, or manage provider sessions. Harnesses and environments handle those parts.

## Core Interface

| Method                                             | Responsibility                                            |
| -------------------------------------------------- | --------------------------------------------------------- |
| `load_tasks(req)`                                  | Return all candidate `TaskSpec` objects for the request.  |
| `select_tasks(tasks, req)`                         | Apply `sample_ids` and benchmark-specific filtering.      |
| `build_plan(task, req, environment)`               | Add benchmark-side execution metadata for one task.       |
| `prepare_task(task, env, req, plan)`               | Materialize task input for the harness.                   |
| `evaluate(task, prepared, result, req, plan, env)` | Score the harness result and return enriched `RunResult`. |
| `aggregate_metrics(results, req, config)`          | Produce run-level metrics.                                |

## Data Objects

| Object         | Meaning                                                                                   |
| -------------- | ----------------------------------------------------------------------------------------- |
| `TaskSpec`     | Stable dataset row with `task_id`, question, category, ground truth, and metadata.        |
| `TaskInput`    | Prompt, system prompt, media, files, workspace, tools, and messages exposed to harnesses. |
| `TaskOutput`   | Expected answer or requested output files.                                                |
| `PreparedTask` | Benchmark-resolved task material for one attempt.                                         |
| `RunResult`    | Harness output after benchmark scoring and metadata enrichment.                           |

## Usage: Select One Task

```bash theme={null}
agentcompass run \
  swebench_verified \
  mini_swe_agent \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"
```

`sample_ids` is generic runtime filtering. It matches `TaskSpec.task_id` and fails fast if a requested id does not exist.

## Usage: Run a Category

```bash theme={null}
agentcompass run \
  screenspot \
  qwen3vl_gui \
  qwen3-vl \
  --env <env-provider> \
  --benchmark-params '{"category":"desktop"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat
```

Category semantics are benchmark-defined. For ScreenSpot, `category` narrows the GUI grounding subset. For other benchmarks, use the reference page for supported params.

## Tag Vocabulary

The public documentation lists benchmark pages alphabetically. Capability labels are tags, not exclusive categories, so one benchmark can appear relevant to multiple dimensions.

| Tag              | Meaning                                                                                          |
| ---------------- | ------------------------------------------------------------------------------------------------ |
| `Agentic Coding` | Repository repair, code generation, scientific coding, or software-engineering workflows.        |
| `Deep Research`  | Browse-heavy, research-agent, general assistant, or judge-scored research tasks.                 |
| `GUI Grounding`  | Screenshot, visual grounding, or GUI target-localization tasks.                                  |
| `Productivity`   | Long-horizon deliverables, economically valuable tasks, skill execution, or workflow completion. |
| `Terminal`       | Command-line task execution in a shell or task-specific terminal workspace.                      |
| `Tool Use`       | Explicit use of tools, code execution, web actions, files, or terminal commands.                 |
| `Judge-Scored`   | Uses a judge model or benchmark-specific judging flow for scoring.                               |
| `Verified`       | Curated or verified subset of a larger benchmark.                                                |

See the [Benchmark reference](/reference/benchmarks/overview) for per-benchmark parameters and run examples.

## Developer Notes

Add a benchmark when the dataset, scoring, or task preparation is meaningfully different from existing benchmarks. Do not add a benchmark just to change provider settings; use an environment config or recipe instead.

## Implementation Checklist

* Define a stable `id` and human-readable `description`.
* Return stable public `task_id` values from `load_tasks`.
* Keep scoring fields out of harness-private assumptions.
* Put provider hints in task metadata, not provider SDK calls.
* Make single-task reproduction possible with `sample_ids`.
* Persist enough detail to debug failures without rerunning the task.

## Related Pages

* [Benchmark reference](/reference/benchmarks/overview)
* [Data Protocol](/developer/data_protocol)
* [Runtime Extensions](/developer/runtime_extensions)