> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Code Implementation

> Implement the dataset, task, evaluator, dependency, recipe, and network contracts for a benchmark.

Implement the smallest complete benchmark path while preserving the official task and scoring semantics.

## 1. Record the Upstream Contract

Record these inputs before designing the adapter:

| Area         | Required evidence                                                                       |
| ------------ | --------------------------------------------------------------------------------------- |
| Dataset      | Release, revision, split, task count, license, access and cache requirements            |
| Evaluator    | Source revision, patch or answer format, metric, timeout and failure semantics          |
| Official run | Model, harness version, prompt, inference settings, retries and attempts per task       |
| Environment  | Task image, workspace, CPU, memory, storage, GPU and network behavior                   |
| Outputs      | Official aggregate, category metrics, completion denominator and available trajectories |

Never use a later project implementation, future tests, reference patches, hidden answers, or unrestricted task-time
network access to solve a task. Such access contaminates the evaluation even when the model discovers it independently.

## 2. Define the Task and Config Contracts

Create the implementation under `src/agentcompass/benchmarks/`. A benchmark normally defines:

* A `RuntimeBenchmarkConfig` subclass for benchmark-owned public parameters.
* A typed `BenchmarkPlan` for per-task preparation and evaluator state.
* A `BaseBenchmark` subclass registered with `BENCHMARKS`.
* Small version-specific adapters when multiple releases differ.

Reuse generic benchmark controls such as `sample_ids`, `k`, `avgk`, `aggregation_mode`, and `category_hierarchy`; do not
redefine them with slightly different semantics. Validate versions, aliases, revisions, splits, and unknown task ids
before opening an environment.

`load_tasks()` must return deterministic `TaskSpec` objects with stable public task ids. Put task images, resource hints,
workspace metadata, evaluator inputs, and upstream identifiers in `TaskSpec.metadata`. Do not call provider SDKs or
perform module-import-time downloads.

## 3. Build a Provider-Neutral Plan

Use `build_plan()` for benchmark-owned task and evaluator state. Keep it provider-neutral and do not mutate the
`RunRequest`. The runtime and recipes compose it into an `ExecutionPlan` later.

Choose the evaluation environment mode deliberately:

| Mode    | Use when                                                                            |
| ------- | ----------------------------------------------------------------------------------- |
| `none`  | Evaluation runs on the AgentCompass process and needs no task sandbox               |
| `reuse` | Evaluation must inspect the existing task environment after agent execution         |
| `fresh` | The official contract requires isolated verification in a newly created environment |

If versions require different modes, resolve them explicitly from benchmark config instead of duplicating the entire
implementation.

## 4. Prepare Only the Harness Contract

`prepare_task()` converts a `TaskSpec` into `PreparedTask`. Expose only what a compatible harness needs:

* `TaskInput.prompt` and optional system prompt or messages.
* Files, media, tools, and the resolved workspace.
* `TaskOutput` answer or requested output files.
* Stable metadata needed for execution and reproduction.

Keep scoring-only information out of prompts and harness-specific fields out of benchmark metadata. Make preparation
safe to repeat during retries and resumed runs.

Use `collect_artifacts()` when a patch or output must be copied before the task environment closes, especially when
verification runs in a fresh environment. Do not combine artifact collection with scoring.

## 5. Preserve Official Evaluation Semantics

Reuse the official evaluator or verifier when practical, pin its revision, and keep the compatibility wrapper small.
The evaluator must distinguish:

* Agent failure or timeout.
* Environment or harness failure.
* Artifact collection failure.
* Verifier crash or timeout.
* Valid evaluated failure or zero score.
* Verified success.

Preserve official timeout and status rules. A patch may pass a verifier after the agent exceeded the official time
budget; if the benchmark defines that timeout as failure, store the verifier evidence but apply the official final rule.

Prefer a benchmark-owned timeout multiplier only when the upstream dataset expresses relative task budgets. If a shared
verifier timeout override also exists, document and test their precedence so two controls never have indistinguishable
meaning.

## 6. Add Dependencies at the Correct Target

| Dependency target                     | Placement                                                  |
| ------------------------------------- | ---------------------------------------------------------- |
| Framework-essential or broadly shared | Default `pyproject.toml` dependencies                      |
| Benchmark-specific driver import      | Named optional extra and `DependencySpec`                  |
| Harness-specific runtime              | Harness extra or isolated harness installer                |
| Task runtime or verifier              | Task image, sandbox setup, or pinned evaluator environment |
| External CLI or service               | Documented prerequisite                                    |

Automatic dependency installation is disabled by default. Missing optional imports must produce an actionable manual
installation command. Do not install at module import time or resolve a specialized integration by downgrading common
framework packages.

## 7. Add Provider Recipes Only When Required

Recipes map task metadata onto provider settings. They must copy the `ExecutionPlan`, remain deterministic, and preserve
this precedence:

```text theme={"system"}
explicit provider-native selector
  > explicit environment image
  > task metadata
  > recipe fallback
```

Build resource defaults from task metadata, then overlay explicit environment values field by field. Preserve explicit
workspace, resource, timeout, label, credential, and network settings. Resolve the winning image or native selector
before removing mutually exclusive fields.

Recipes must not create sandboxes, execute commands, call models, or score results. Audit sibling provider and version
recipes when changing shared precedence behavior.

## 8. Resolve Network Phases Explicitly

Treat setup, agent execution, and verification as separate policy phases. Use the official benchmark behavior as the
default and apply restrictions through environment enforcement, never through prompt instructions.

Trusted harness installation normally completes under the setup policy before a stricter run policy is applied. If the
user explicitly restricts setup, fail clearly when required dependencies are unavailable rather than silently opening
network access.

## 9. Register and Inspect the Component

Export the module from `src/agentcompass/benchmarks/__init__.py` and verify discovery and generated config documentation:

```bash theme={"system"}
uv run agentcompass list benchmark
uv run agentcompass config docs benchmark <benchmark-id>
```

The benchmark must define a stable `id` and non-empty `description`.