Overview

Understand the ownership, workflow, and completion criteria for an AgentCompass Benchmark integration.

A Benchmark integration is a reproducible evaluation contract, not only a dataset loader. It owns task identity, preparation, scoring, aggregation, and the evidence required to compare an AgentCompass run with the official result.

Before editing, establish the upstream contract from primary sources: the official repository, dataset release, paper, technical report, evaluator, task images, and leaderboard artifacts. Pin revisions whenever upstream state can change.

Integration Boundary

Belongs in the Benchmark	Belongs elsewhere
Dataset discovery, versioning, task selection, stable task IDs	Model calls and agent loops → Harness
TaskSpec, Benchmark plan, PreparedTask, expected outputs	Commands, files, sandbox lifecycle → Environment
Evaluation mode, official verifier, score, aggregation	Provider image/snapshot/workspace mapping → Recipe
Benchmark-specific parameters and metadata	Concurrency, retries, persistence → runtime

Mark the upstream Harness as the recommended alignment configuration, but do not reject another Harness merely because its ID differs. Compatibility should follow the prepared-task format, Model protocol, and Environment capabilities.

Integration Workflow

1. Implement the Benchmark contract, choosing execution ownership, evaluation mode, and aggregation strategy before entering the relevant subpage.
2. Update public documentation with runnable commands and Benchmark-owned parameters.
3. Validate and align the integration from focused checks through a complete official-split comparison.

Completion Gate

Do not request review until all applicable statements are true:

• Official task, evaluator, Model, and Harness versions are identifiable.
• Errors, valid zero scores, timeouts and evaluator failures remain distinguishable.
• Every claimed Environment has a real smoke result.
• A full-run report states coverage, failures, score, official reference and material differences.
• Public documentation matches the implemented defaults and compatibility matrix.