> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Harness-Driven Benchmark

When a Harness owns the agent loop, use `BaseBenchmark` to prepare tasks and score the `RunResult` returned by the Harness.

This path fits question answering, code editing, browsing, and other Benchmarks that can reuse an existing Harness. The Benchmark neither calls the Model directly nor reimplements the Harness session or tool loop.

## Implement a Complete Exact-Match Example

Create `example_exact_match.py` under `src/agentcompass/benchmarks/` with the following content:

```python theme={"system"}
from __future__ import annotations

from dataclasses import dataclass, replace

from agentcompass.benchmarks.config import RuntimeBenchmarkConfig
from agentcompass.runtime import (
    BENCHMARKS,
    BaseBenchmark,
    BenchmarkPlan,
    EnvironmentSession,
    EnvironmentSpec,
    ExecutionPlan,
    PreparedTask,
    RunRequest,
    RunResult,
    TaskInput,
    TaskOutput,
    TaskSpec,
    TaskStatus,
)
from agentcompass.runtime.config import config_field, parse_bool
from agentcompass.runtime.metrics import make_metric_contract


@dataclass(slots=True)
class ExampleExactMatchConfig(RuntimeBenchmarkConfig):
    case_sensitive: bool = config_field(
        default=False,
        description="Compare answers with case sensitivity.",
    )

    def __post_init__(self) -> None:
        RuntimeBenchmarkConfig.__post_init__(self)
        self.case_sensitive = parse_bool(self.case_sensitive, "case_sensitive")


@dataclass(slots=True)
class ExampleExactMatchPlan(BenchmarkPlan):
    expected: str = ""
    case_sensitive: bool = False


@BENCHMARKS.register()
class ExampleExactMatchBenchmark(BaseBenchmark):
    id = "example_exact_match"
    description = "One-task exact-match Benchmark used by the developer tutorial."
    config_class = ExampleExactMatchConfig
    evaluation_environment_mode = "none"
    metric_contract = make_metric_contract(
        primary="correct",
        binary=("correct",),
        labels={"correct": "Accuracy"},
    )

    def load_tasks(self, req: RunRequest) -> list[TaskSpec]:
        _ = req
        return [
            TaskSpec(
                task_id="capital-france",
                question=(
                    "What is the capital of France? "
                    "Answer with only the city name."
                ),
                category="geography",
                ground_truth="Paris",
                metadata={"dataset_revision": "tutorial-v1"},
            )
        ]

    def build_plan(
        self,
        task: TaskSpec,
        req: RunRequest,
        environment: EnvironmentSpec,
    ) -> ExampleExactMatchPlan:
        _ = environment
        config = self.build_config(req)
        if not isinstance(config, ExampleExactMatchConfig):
            raise TypeError(
                "example_exact_match requires ExampleExactMatchConfig"
            )
        return ExampleExactMatchPlan(
            expected=str(task.ground_truth),
            case_sensitive=config.case_sensitive,
        )

    async def prepare_task(
        self,
        task: TaskSpec,
        env: EnvironmentSession,
        req: RunRequest,
        plan: BenchmarkPlan,
    ) -> PreparedTask:
        _ = env, req
        self._require_plan(plan)
        return PreparedTask(
            task_id=task.task_id,
            category=task.category,
            ground_truth=None,
            input=TaskInput(prompt=task.question),
            output=TaskOutput(answer="Return only the city name."),
            metadata={
                "dataset_revision": task.metadata["dataset_revision"],
            },
        )

    async def evaluate(
        self,
        task: TaskSpec,
        prepared: PreparedTask,
        result: RunResult,
        req: RunRequest,
        plan: ExecutionPlan,
        env: EnvironmentSession | None = None,
    ) -> RunResult:
        _ = req, env
        benchmark_plan = self._require_plan(plan.benchmark_plan)
        candidate = str(result.final_answer or "").strip()
        expected = benchmark_plan.expected.strip()
        if not benchmark_plan.case_sensitive:
            candidate = candidate.casefold()
            expected = expected.casefold()

        evaluated = (
            result.status == TaskStatus.COMPLETED and not result.error
        )
        correct = evaluated and candidate == expected
        return replace(
            result,
            task_id=prepared.task_id,
            category=prepared.category,
            metrics={"correct": correct},
        )

    @staticmethod
    def _require_plan(plan: BenchmarkPlan) -> ExampleExactMatchPlan:
        if not isinstance(plan, ExampleExactMatchPlan):
            raise TypeError(
                "example_exact_match requires ExampleExactMatchPlan"
            )
        return plan
```

This implementation completes the three abstract `BaseBenchmark` methods: `load_tasks()`, `prepare_task()`, and `evaluate()`. `build_plan()` keeps the answer on the evaluation side instead of exposing `ground_truth` to the Harness. `evaluate()` uses `dataclasses.replace()` to preserve the status, error, trajectory, artifacts, and other result fields written by the Harness.

## Export and Inspect the Registration

Add this import to `src/agentcompass/benchmarks/__init__.py`:

```python theme={"system"}
from .example_exact_match import ExampleExactMatchBenchmark
```

Inspect registration and the configuration schema:

```bash theme={"system"}
uv run agentcompass list benchmark
uv run agentcompass config docs benchmark example_exact_match
```

The first command should list `example_exact_match`. The second should show `case_sensitive` and the shared `RuntimeBenchmarkConfig` fields. If the component is missing, inspect `__init__.py`, duplicate IDs, and the complete import traceback first.

## Run with a Harness

The following command uses `example_answer` from the [Harness implementation tutorial](/en/developer_guide/extensions/harness/code_implementation). That Harness returns the configured `final_answer` directly, so this smoke run does not require a working Model endpoint:

```bash theme={"system"}
uv run agentcompass run example_exact_match example_answer unused-model \
  --env host_process \
  --benchmark-params '{"sample_ids":["capital-france"]}' \
  --harness-params '{"answer":"Paris"}' \
  --task-concurrency 1 \
  --no-enable-analysis \
  --results-dir results-dev \
  --run-name benchmark-smoke
```

In `run_info.json`, inspect `request` → `benchmark` → `id` and `resolved_execution_plans`, then inspect the single attempt record under `details/*.json`. With the answer `Paris`, expect `status: "completed"` and `metrics.correct: true`. Change the answer to `Lyon`; `status` should remain `completed`, while `metrics.correct` becomes `false`. This demonstrates that execution status and the Benchmark verdict are separate dimensions.

## Extend the Pattern for Real Data

* Put dataset loading, revision validation, and stable task conversion in `load_tasks()`, not at module-import time.
* Put per-task evaluator state, timeouts, and paths in a typed `BenchmarkPlan`; do not pass them between attempts through a shared mutable dictionary.
* Put prompts, workspaces, and public attachments in `PreparedTask`; hidden tests, answers, and reference patches must not enter Harness-visible fields.
* When evaluation needs the task Environment or an isolated verifier, do not keep adding logic to this `none` example. Use the `reuse` or `fresh` patterns from [Evaluation Modes and Artifacts](/en/developer_guide/extensions/benchmark/code_implementation/evaluation_modes).
* When scoring is not a simple boolean, declare scalar `score` as the Contract primary and write it under `RunResult.metrics`; see [Results and Aggregation](/en/developer_guide/extensions/benchmark/code_implementation/results_and_aggregation).

For a production Harness-driven Benchmark with controller-side evaluation, inspect [`browsecomp.py`](https://github.com/open-compass/AgentCompass/blob/main/src/agentcompass/benchmarks/browsecomp.py).