> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Implementation Overview

Before implementing a Benchmark, decide who runs the agent, where evaluation occurs, and how results are aggregated.

There is no single Benchmark template for every use case. Most integrations delegate execution to a Harness; a Benchmark takes over execution only when the upstream interaction loop is part of the evaluation definition. Scoring can run in the AgentCompass process, the task Environment, or a separate evaluation Environment.

## Step 1: Choose the Execution Owner

This choice determines whether the Harness or the Benchmark runs the agent loop:

| Scenario                                                                          | <span style={{ display: "inline-block", minWidth: "10.5rem" }}>Implementation type</span> | Who implements the execution-stage `run_task()` | Harness in the command  |
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------- | ----------------------- |
| An existing Harness can run the prepared task                                     | `BaseBenchmark`                                                                           | Harness                                         | A registered Harness ID |
| The upstream Benchmark defines a dedicated interaction loop that cannot be reused | `HarnessFreeBenchmark`                                                                    | Benchmark                                       | `none`                  |

`HarnessFreeBenchmark` is a subclass of `BaseBenchmark`. They share the task-loading, preparation, artifact-collection, evaluation, and aggregation contracts; only the owner of the execution stage differs. Prefer a [Harness-driven Benchmark](/en/developer_guide/extensions/benchmark/code_implementation/harness_driven). Use a [Benchmark-driven implementation](/en/developer_guide/extensions/benchmark/code_implementation/benchmark_driven) only when the upstream interaction protocol itself belongs to the Benchmark definition.

## Step 2: Choose the Evaluation Location

This choice determines where `evaluate()` runs and is independent of the execution owner:

| Evaluator requirement                                              | `evaluation_environment_mode` | `env` passed to `evaluate()`    |
| ------------------------------------------------------------------ | ----------------------------- | ------------------------------- |
| Score only an answer or already collected data                     | `none`                        | `None`                          |
| Inspect the same workspace or process state left by task execution | `reuse`                       | The still-open task Environment |
| Run the verifier in an isolated Environment                        | `fresh`                       | A new evaluation Environment    |

Both `BaseBenchmark` and `HarnessFreeBenchmark` can use all three modes. `collect_artifacts()` is not a fourth mode; it is an optional lifecycle hook that extracts submissions before the task Environment closes. A `fresh` evaluator that depends on the task workspace usually needs to override this method. See [Evaluation Modes and Artifacts](/en/developer_guide/extensions/benchmark/code_implementation/evaluation_modes) for the implementation patterns.

## Step 3: Choose the Aggregation Strategy

This choice determines how attempt-level verdicts become request-level metrics. It does not change the execution owner or evaluation location:

| Result shape                                                                            | `aggregate_metrics()`                                |
| --------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| The official result is a mean of the Contract's reduced task observations               | Inherit the default aggregation from `BaseBenchmark` |
| The official result uses a ratio of sums, total, rank, medal, or another corpus formula | Override the method and return `MetricReport`        |

See [Results and Aggregation](/en/developer_guide/extensions/benchmark/code_implementation/results_and_aggregation) for the result fields and aggregation patterns. Regardless of the final combination, read [Shared Contracts](/en/developer_guide/extensions/benchmark/code_implementation/shared_contracts) first to define task fields, visibility boundaries, and per-task plans.

## Shared Lifecycle

Each request loads and selects tasks first, then performs these steps for every task and attempt:

```text theme={"system"}
load_tasks
  → select_tasks
  → build_plan
  → open the task Environment
  → prepare_task
  → Harness.run_task or Benchmark.run_task
  → collect_artifacts
  → evaluate
  → persist the attempt result
  → aggregate_metrics
```

The task Environment is always still running when `collect_artifacts()` is called. The evaluation mode determines where `evaluate()` runs. After all tasks finish, `aggregate_metrics()` reads the persisted structure and produces the summary.

## Method Responsibilities

| <span style={{ display: "inline-block", minWidth: "10.5rem" }}>Method</span> | Requirement or default                   | Responsibility                                                                       |
| ---------------------------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------ |
| `load_tasks()`                                                               | Required                                 | Convert a pinned dataset release into `TaskSpec` values                              |
| `select_tasks()`                                                             | Default implementation                   | Apply the shared task-selection logic; override only for special selection semantics |
| `build_plan()`                                                               | Default implementation                   | Resolve typed Benchmark state for one task attempt                                   |
| `prepare_task()`                                                             | Required                                 | Prepare execution input for the Harness or Benchmark inside the task Environment     |
| `collect_artifacts()`                                                        | Empty default                            | Extract submissions before the task Environment closes                               |
| `evaluate()`                                                                 | Required                                 | Preserve execution status and write declared observations to `RunResult.metrics`     |
| `aggregate_metrics()`                                                        | Default Contract-driven aggregation      | Convert reduced task observations into a `MetricReport`                              |
| `run_task()`                                                                 | Required only for `HarnessFreeBenchmark` | Run the inference or interaction loop owned by the Benchmark                         |

After the implementation works, add the user-facing entry point through [Documentation Update](/en/developer_guide/extensions/benchmark/documentation_update), then follow [Validation and Alignment](/en/developer_guide/extensions/benchmark/validation_and_alignment) with real data, Environments, and official results.