Overview

Understand the ownership, workflow, and completion criteria for an AgentCompass benchmark integration.

A benchmark integration is a reproducible evaluation contract, not only a dataset loader. It owns task identity, preparation, scoring, aggregation, and the evidence required to compare an AgentCompass run with the official result.

Before editing, establish the upstream contract from primary sources: the official repository, dataset release, paper, technical report, evaluator, task images, and leaderboard artifacts. Pin revisions whenever upstream state can change.

Integration Boundary

Belongs in the benchmark	Belongs elsewhere
Dataset discovery, versioning, task selection, stable task ids	Model calls and agent loops → harness
TaskSpec, benchmark plan, PreparedTask, expected outputs	Commands, files, sandbox lifecycle → environment
Evaluation mode, official verifier, score, aggregation	Provider image/snapshot/workspace mapping → recipe
Benchmark-specific parameters and metadata	Concurrency, retries, persistence → runtime

Mark the upstream harness as the recommended alignment configuration, but do not reject another harness merely because its id differs. Compatibility should follow the prepared-task, model protocol, and environment capabilities.

Integration Workflow

1. Implement the benchmark contract, including task identity, evaluator semantics, dependencies, recipes, and network phases.
2. Update public documentation with runnable commands and benchmark-owned parameters.
3. Validate and align the integration from focused checks through a complete official-split comparison.

Completion Gate

Do not request review until all applicable statements are true:

• Official task, evaluator, model and harness versions are identifiable.
• Stable task ids support exact sample_ids selection.
• Errors, valid zero scores, timeouts and evaluator failures remain distinguishable.
• Explicit environment settings win over inferred recipe defaults.
• Every claimed environment has a real smoke result.
• Restricted network behavior has enforcement-level evidence when applicable.
• A full-run report states coverage, failures, score, official reference and material differences.
• Public documentation matches the implemented defaults and compatibility matrix.