Shared Contracts

Convert dataset records into stable TaskSpec values first, then decide which fields the Harness may see and which state must remain evaluation-only.

Distinguish the Four Data Carriers

Carrier	Primary consumers	Store here	Do not store here
TaskSpec	Benchmark, Planner, Recipe	Stable task ID, question, category, upstream metadata, per-task evaluation mode, and network policies	Provider sessions or an already-open Environment
BenchmarkPlan	Benchmark and Planner for the current task attempt	Resolved configuration, workspace paths, verifier timeout, and typed evaluator state	Provider SDK clients or mutable global state
PreparedTask	Harness or HarnessFreeBenchmark.run_task()	Prompt, messages, files, media, tools, workspace, and expected output	Hidden answers, reference patches, private tests, or grading secrets
RunResult	Benchmark, runtime, and result consumers	Execution status, answer, trajectory, artifacts, scores, and publishable evaluation evidence	Non-serializable objects or credentials

A Harness can read the entire PreparedTask, including its ground_truth and metadata. Do not copy TaskSpec.metadata into it without filtering.

Keep evaluator-only data in TaskSpec.ground_truth or a typed BenchmarkPlan, and set PreparedTask.ground_truth to None. These objects still belong to the runtime and result-audit boundary, so they must not contain credentials or other secret material that must never be persisted.

Write a reference answer to RunResult.ground_truth only when it is safe to publish with the result.

Define Public Configuration

Define Benchmark parameters with RuntimeBenchmarkConfig and config_field(), and normalize types early in __post_init__():

from dataclasses import dataclass

from agentcompass.benchmarks.config import RuntimeBenchmarkConfig
from agentcompass.runtime.config import config_field, parse_bool


@dataclass(slots=True)
class ExampleExactMatchConfig(RuntimeBenchmarkConfig):
    case_sensitive: bool = config_field(
        default=False,
        description="Compare answers with case sensitivity.",
    )

    def __post_init__(self) -> None:
        RuntimeBenchmarkConfig.__post_init__(self)
        self.case_sensitive = parse_bool(self.case_sensitive, "case_sensitive")

Do not download data, install dependencies, or read credentials during module import. Access data through an explicit loader or dependency-preparation path. If a revision, split, or access requirement is invalid, fail with an actionable message.

Load Deterministic Tasks

load_tasks() should pin the upstream revision and produce stable task_id values. The following metadata contains only reproducibility information that may appear in logs and execution input; the answer remains separate in ground_truth:

def load_tasks(self, req: RunRequest) -> list[TaskSpec]:
    _ = req
    return [
        TaskSpec(
            task_id="capital-france",
            question="What is the capital of France? Answer with only the city name.",
            category="geography",
            ground_truth="Paris",
            metadata={"dataset_revision": "tutorial-v1"},
        )
    ]

The inherited select_tasks() method already applies the runtime’s shared task-selection logic. Override it only when the Benchmark needs semantics beyond ordinary ID filtering. Whichever rule you use, keep the returned order deterministic.

Build a Typed Plan for Each Attempt

When evaluator state must be derived from both configuration and the task, define a BenchmarkPlan subclass and resolve it once for the current attempt in build_plan():

from dataclasses import dataclass

from agentcompass.runtime import BenchmarkPlan, EnvironmentSpec, RunRequest, TaskSpec


@dataclass(slots=True)
class ExampleExactMatchPlan(BenchmarkPlan):
    expected: str = ""
    case_sensitive: bool = False


def build_plan(
    self,
    task: TaskSpec,
    req: RunRequest,
    environment: EnvironmentSpec,
) -> ExampleExactMatchPlan:
    _ = environment
    config = self.build_config(req)
    if not isinstance(config, ExampleExactMatchConfig):
        raise TypeError("example_exact_match requires ExampleExactMatchConfig")
    return ExampleExactMatchPlan(
        expected=str(task.ground_truth),
        case_sensitive=config.case_sensitive,
    )

build_plan() must not open an Environment, call a Model, or mutate RunRequest. After the initial ExecutionPlan is built, Recipes adjust the plan according to their own contracts. Benchmark documentation therefore must not assume that the runtime enforces one framework-wide precedence rule for Recipe fields. When provider mapping is required, document and test its preservation rules in the corresponding Recipe Integration.

Prepare Execution Input

prepare_task() may create a workspace or upload public material in the task Environment, but its return value may contain only data visible during execution:

async def prepare_task(
    self,
    task: TaskSpec,
    env: EnvironmentSession,
    req: RunRequest,
    plan: BenchmarkPlan,
) -> PreparedTask:
    _ = env, req
    self._require_plan(plan)
    return PreparedTask(
        task_id=task.task_id,
        category=task.category,
        ground_truth=None,
        input=TaskInput(prompt=task.question),
        output=TaskOutput(answer="Return only the city name."),
        metadata={"dataset_revision": task.metadata["dataset_revision"]},
    )

Use the supplied EnvironmentSession when you need to create files or directories; do not bypass the Environment and call a provider SDK directly. Retries may invoke this method again, so preparation must be safe to repeat. Otherwise, explicitly clean up the workspace you created before execution.

Registration and Dependencies

Register the implementation with @BENCHMARKS.register() and import its module from src/agentcompass/benchmarks/__init__.py:

from .example_exact_match import ExampleExactMatchBenchmark

From the repository root, inspect component discovery and the parameter schema:

uv run agentcompass list benchmark
uv run agentcompass config docs benchmark example_exact_match

Dependencies required by the framework in every installation belong in the default project dependencies. A Python driver used only by this Benchmark belongs in a dedicated optional dependency group with a declared DependencySpec. Runtime dependencies needed by the task or verifier belong in the corresponding Environment and must be pinned there. Successful registration proves only that the module imports; it does not validate data, credentials, the verifier, or a real run.