Code Implementation

Implement the dataset, task, evaluator, dependency, recipe, and network contracts for a benchmark.

Implement the smallest complete benchmark path while preserving the official task and scoring semantics.

1. Record the Upstream Contract

Record these inputs before designing the adapter:

Area	Required evidence
Dataset	Release, revision, split, task count, license, access and cache requirements
Evaluator	Source revision, patch or answer format, metric, timeout and failure semantics
Official run	Model, harness version, prompt, inference settings, retries and attempts per task
Environment	Task image, workspace, CPU, memory, storage, GPU and network behavior
Outputs	Official aggregate, category metrics, completion denominator and available trajectories

Never use a later project implementation, future tests, reference patches, hidden answers, or unrestricted task-time network access to solve a task. Such access contaminates the evaluation even when the model discovers it independently.

2. Define the Task and Config Contracts

Create the implementation under src/agentcompass/benchmarks/. A benchmark normally defines:

• A RuntimeBenchmarkConfig subclass for benchmark-owned public parameters.
• A typed BenchmarkPlan for per-task preparation and evaluator state.
• A BaseBenchmark subclass registered with BENCHMARKS.
• Small version-specific adapters when multiple releases differ.

Reuse generic benchmark controls such as sample_ids, k, avgk, aggregation_mode, and category_hierarchy; do not redefine them with slightly different semantics. Validate versions, aliases, revisions, splits, and unknown task ids before opening an environment.

load_tasks() must return deterministic TaskSpec objects with stable public task ids. Put task images, resource hints, workspace metadata, evaluator inputs, and upstream identifiers in TaskSpec.metadata. Do not call provider SDKs or perform module-import-time downloads.

3. Build a Provider-Neutral Plan

Use build_plan() for benchmark-owned task and evaluator state. Keep it provider-neutral and do not mutate the RunRequest. The runtime and recipes compose it into an ExecutionPlan later.

Choose the evaluation environment mode deliberately:

Mode	Use when
none	Evaluation runs on the AgentCompass process and needs no task sandbox
reuse	Evaluation must inspect the existing task environment after agent execution
fresh	The official contract requires isolated verification in a newly created environment

If versions require different modes, resolve them explicitly from benchmark config instead of duplicating the entire implementation.

4. Prepare Only the Harness Contract

prepare_task() converts a TaskSpec into PreparedTask. Expose only what a compatible harness needs:

• TaskInput.prompt and optional system prompt or messages.
• Files, media, tools, and the resolved workspace.
• TaskOutput answer or requested output files.
• Stable metadata needed for execution and reproduction.

Keep scoring-only information out of prompts and harness-specific fields out of benchmark metadata. Make preparation safe to repeat during retries and resumed runs.

Use collect_artifacts() when a patch or output must be copied before the task environment closes, especially when verification runs in a fresh environment. Do not combine artifact collection with scoring.

5. Preserve Official Evaluation Semantics

Reuse the official evaluator or verifier when practical, pin its revision, and keep the compatibility wrapper small. The evaluator must distinguish:

• Agent failure or timeout.
• Environment or harness failure.
• Artifact collection failure.
• Verifier crash or timeout.
• Valid evaluated failure or zero score.
• Verified success.

Preserve official timeout and status rules. A patch may pass a verifier after the agent exceeded the official time budget; if the benchmark defines that timeout as failure, store the verifier evidence but apply the official final rule.

Prefer a benchmark-owned timeout multiplier only when the upstream dataset expresses relative task budgets. If a shared verifier timeout override also exists, document and test their precedence so two controls never have indistinguishable meaning.

6. Add Dependencies at the Correct Target

Dependency target	Placement
Framework-essential or broadly shared	Default pyproject.toml dependencies
Benchmark-specific driver import	Named optional extra and DependencySpec
Harness-specific runtime	Harness extra or isolated harness installer
Task runtime or verifier	Task image, sandbox setup, or pinned evaluator environment
External CLI or service	Documented prerequisite

Automatic dependency installation is disabled by default. Missing optional imports must produce an actionable manual installation command. Do not install at module import time or resolve a specialized integration by downgrading common framework packages.

7. Add Provider Recipes Only When Required

Recipes map task metadata onto provider settings. They must copy the ExecutionPlan, remain deterministic, and preserve this precedence:

explicit provider-native selector
  > explicit environment image
  > task metadata
  > recipe fallback

Build resource defaults from task metadata, then overlay explicit environment values field by field. Preserve explicit workspace, resource, timeout, label, credential, and network settings. Resolve the winning image or native selector before removing mutually exclusive fields.

Recipes must not create sandboxes, execute commands, call models, or score results. Audit sibling provider and version recipes when changing shared precedence behavior.

8. Resolve Network Phases Explicitly

Treat setup, agent execution, and verification as separate policy phases. Use the official benchmark behavior as the default and apply restrictions through environment enforcement, never through prompt instructions.

Trusted harness installation normally completes under the setup policy before a stricter run policy is applied. If the user explicitly restricts setup, fail clearly when required dependencies are unavailable rather than silently opening network access.

9. Register and Inspect the Component

Export the module from src/agentcompass/benchmarks/__init__.py and verify discovery and generated config documentation:

uv run agentcompass list benchmark
uv run agentcompass config docs benchmark <benchmark-id>

The benchmark must define a stable id and non-empty description.