Architecture Overview

AgentCompass separates Model, Benchmark, Harness, Environment, Recipe, Analyzer, and runtime responsibilities, connecting them through request, plan, attempt, and result contracts.

AgentCompass resolves CLI or Python input into one or more RunRequest objects, discovers tasks through a Benchmark, builds an ExecutionPlan for each semantic attempt, and executes that plan through the shared runtime. The runtime persists evaluated attempts as task details and aggregates those details into a request summary.

From request to result and ownership

Component registries do not contain ModelSpec. A Model is a request value describing an endpoint and inference settings. The runtime component registry discovers Benchmark, Harness, Environment, Recipe, and Analyzer implementations; its source location appears in the Source Map.

The exact Environment close point depends on evaluation_environment_mode: evaluation may run in the task Environment, without an Environment, or in a fresh Environment. Cleanup remains in finally paths in all three cases.

Layer	Owns	Primary source
Model	Endpoint identity, API protocol, credentials, and inference parameters	src/agentcompass/runtime/models/model.py
Benchmark	Dataset loading, stable task identity, task preparation, evaluation, and metric aggregation	src/agentcompass/benchmarks/ and BaseBenchmark in src/agentcompass/runtime/base.py
Harness	agent or Model execution loop, session lifecycle, trajectory and usage normalization	src/agentcompass/harnesses/ and BaseHarness in src/agentcompass/runtime/base.py
Environment	Commands, files, endpoints, enforceable network policy, resources, and sandbox lifecycle	src/agentcompass/environments/ and BaseEnvironment in src/agentcompass/runtime/base.py
Recipe	Pure, task-specific adaptation of an ExecutionPlan	src/agentcompass/recipes/, BaseRecipe, and Planner
runtime	Cross-component orchestration, attempts, retries, limits, cancellation, cleanup, and persistence	src/agentcompass/runtime/
Analyzer	Optional interpretation of an evaluated attempt without changing the authoritative Benchmark score	src/agentcompass/analyzers/ and src/agentcompass/runtime/analysis.py

Place policy in the component that owns it. For example, a Benchmark may request isolated evaluation through its plan, but Environment code owns how a sandbox is opened and closed, while the runtime owns when those operations occur.

Execution scopes. AgentCompass has two nested units that should not be conflated:

• An attempt is one of the Benchmark’s k executions. The runtime calls Planner.plan once at the start of each attempt, so every attempt receives a newly resolved plan.
• A runtime retry repeats failed work within that same attempt according to ExecutionSpec.max_retries and retry_pattern_list. It reuses the already resolved ExecutionPlan; it does not call Planner.plan again.

Likewise, RunResult is the execution-level object returned by a Harness or harness-free Benchmark. Benchmark.evaluate turns it into an evaluated attempt, _run_attempts groups evaluated attempts into one task record, and UnifiedEvaluationRuntime.finalize aggregates task records into the request result.

Invariants to preserve

• Import shared contracts from agentcompass.runtime; do not couple one extension to another extension’s private module.
• Treat planning as deterministic. Sandbox creation, network calls, package installation, and file mutation belong to explicit lifecycle phases.
• Preserve compatible explicit user values when defaults or Recipes fill missing settings.
• Enforce network, filesystem, resource, and secret boundaries in the runtime or Environment provider, not in prompts.
• Preserve the distinction between run errors, evaluation errors, valid zero scores, and skipped work.
• Record enough resolved state to explain a result, and keep secrets redacted at every persistence boundary.

Continue by task

Your task	Read
Follow a CLI or SDK request through concrete functions and find the owner of a change	Source Map
Change shared contracts, merge precedence, per-attempt planning, or Recipe behavior	Runtime Contracts and Planning
Work on preparation, inference, artifact collection, evaluation, attempts, retries, scheduling, cancellation, or resource release	Execution, Scheduling, and Cleanup
Change detail files, summaries, analysis output, result reuse, or compatibility	Results and Reuse

When implementing a new component, continue with Benchmark Integration, Harness Integration, or Environment Integration. When preparing a change for review, use the General Contributing Workflow.