Overview

Understand AgentCompass system boundaries, runtime contracts, execution flow, and the impact of changing each component.

AgentCompass is a composable evaluation runtime. Before changing it, locate the behavior in the system map and identify the contracts it produces and consumes. This prevents a benchmark-specific requirement from leaking into a harness, environment provider, or the shared runtime.

System Overview

The CLI and Python SDK converge on the same orchestration runtime. run wraps one RunRequest; launch resolves an ordered set of named requests into one Orchestration. Registries resolve the requested components, the planner creates a per-task ExecutionPlan, and the shared scheduler coordinates request priority, task concurrency, environment, benchmark, harness, evaluation, result, and analysis lifecycles.

The diagram shows the common harness-backed path. A HarnessFreeBenchmark can own the inference loop, but it must preserve the same task, environment, result, evaluation, and persistence contracts.

Runtime Shape

The runtime is built around a small set of typed objects. These objects are the stable seams between modules.

Orchestration
  ├─ global task concurrency, deadline, provider limits, and progress
  └─ ordered OrchestratedRun[]
       └─ RunRequest
            ├─ ModelSpec
            ├─ BenchmarkSpec
            ├─ HarnessSpec
            ├─ EnvironmentSpec
            ├─ ExecutionSpec
            └─ OutputSpec
                 │
                 ├─ load and select -> TaskSpec
                 ├─ plan            -> ExecutionPlan
                 ├─ prepare         -> PreparedTask
                 ├─ run             -> RunResult
                 └─ evaluate        -> normalized RunResult -> persisted artifacts

Contract	Producer	Primary consumers	Change impact
Orchestration	CLI/SDK launch resolver	Global scheduler, progress renderer, runtime bootstrap	Cross-request priority, concurrency, cancellation, failure isolation, CLI/SDK parity
RunRequest	CLI, Python SDK, config loader	Registry, planner, runtime, every component	Public configuration, serialization, CLI/SDK parity, reproducibility
TaskSpec	Benchmark loader	Task selection, planner, benchmark preparation, analyzers	Dataset identity, recipes, retries, result paths
ExecutionPlan	Planner and recipes	Environment, benchmark, harness, runtime	Provider settings, resources, network phases, evaluation environment
PreparedTask	Benchmark	Harness or harness-free inference loop	Prompt, files, media, tools, workspace, expected outputs
EnvironmentSession	Environment provider	Benchmark preparation, harness, verifier	Command and file semantics across every sandbox provider
RunResult	Harness, then benchmark evaluator	Persistence, metrics, analyzers	Status semantics, score correctness, trajectories, summaries

Changing one of these contracts is a runtime change, not a local component change. Audit every producer and consumer, update public exports, and preserve serialization compatibility where existing result artifacts depend on it.

Component Ownership

Component	Owns	Must not own	Main source
Benchmark	Dataset, stable task identity, preparation, scoring, aggregation, evaluator semantics	Agent loop, provider SDK, generic sandbox lifecycle	src/agentcompass/benchmarks/
Harness	Agent or model execution loop, harness setup, trajectory and usage normalization	Dataset loading, benchmark score, provider image selection	src/agentcompass/harnesses/
Model	Endpoint identity, API protocol, credentials, inference parameters	Benchmark prompts, agent lifecycle, scoring	ModelSpec and protocol clients
Environment	Commands, files, endpoints, sandbox lifecycle, resources, enforceable network policy	Benchmark rules, model decisions, score interpretation	src/agentcompass/environments/
Recipe	Deterministic, per-task plan adaptation for benchmark/provider compatibility	Side effects, sandbox creation, inference, scoring	src/agentcompass/recipes/
Runtime	Cross-component orchestration, concurrency, retries, network phase transitions, cleanup, persistence	Benchmark-specific shortcuts or provider-specific business rules	src/agentcompass/runtime/
Analyzer	Post-run interpretation of normalized results	Changing the authoritative benchmark score or execution outcome	src/agentcompass/analyzers/

Use ownership to decide where a change belongs. For example, a task image stored in dataset metadata belongs to the benchmark contract; mapping that image to a Daytona snapshot or Modal named image belongs to a recipe; creating and closing the sandbox belongs to the environment provider.

Execution Lifecycle

The runtime creates a plan for every selected task because images, resources, workspaces, and verifier requirements can vary between tasks.

Cleanup belongs in finally paths. A failure in preparation, harness startup, model execution, artifact collection, or verification must not leak a container, cloud sandbox, proxy, client, or background process.

Planning and Precedence

Configuration is normalized before execution. Recipes then adapt a copy of the per-task plan without mutating the original request. Explicit user intent must survive every adaptation layer.

For provider selection fields, use this precedence:

explicit provider-native selector
  > explicit environment image
  > task metadata
  > recipe fallback

For resource dictionaries, start with task defaults and overlay explicit user values field by field. For scalar defaults such as a workspace root, use setdefault() rather than unconditional assignment.

This policy has a deliberate trade-off: automatic recipes make official task images convenient, while user-first precedence keeps custom prebuilt images and provider-native snapshots possible. Reject incompatible explicit values with an actionable error; do not silently replace them with a configuration that happens to run.

Design Principles

Depend on Contracts, Not Implementations

Components communicate through runtime contracts and public registries. Import shared types from agentcompass.runtime instead of reaching into another component’s private module. This keeps a harness reusable across benchmarks and an environment provider reusable across harnesses.

Separate Policy From Mechanism

Benchmarks define evaluation policy; environments provide execution mechanisms; the runtime sequences them. A benchmark may require isolated verification, but it should express that through its evaluation plan rather than creating a Docker container directly.

Preserve Explicit User Intent

Defaults and recipes may fill missing values, never overwrite an explicit compatible choice. Convenience that changes a user-supplied image, resource limit, workspace, network policy, model parameter, or timeout is a correctness bug.

Keep Planning Pure and Side Effects Scoped

Task loading, plan construction, and recipe application should be deterministic. Network calls, package installation, sandbox creation, and file mutation belong to explicit lifecycle phases where errors, limits, and cleanup are visible.

Enforce Security at the Environment Boundary

Prompts and agent instructions are not security controls. Network isolation, resource limits, filesystem boundaries, and secret handling must be enforced by the provider or runtime. Setup, agent execution, and verification policies are separate because trusted preparation and untrusted execution have different requirements.

Make Reproducibility Observable

Record versions, revisions, resolved plans, model and harness settings, evaluator behavior, failures, and task coverage. A close score without matching settings and denominator is not alignment evidence.

Fail Early, Preserve Failure Meaning

Validate unsupported versions, protocols, providers, policies, and task ids before expensive work begins. Preserve the difference between environment errors, harness errors, model errors, agent timeouts, evaluator errors, and valid zero scores; downstream analysis depends on those distinctions.

Keep Specialized Dependencies Optional

The default installation should contain framework-essential and broadly shared packages. Benchmark- or harness-specific driver dependencies belong in optional extras and the trusted dependency workflow. Task-runtime dependencies belong in the task image or controlled setup phase.

Bound Concurrency and External Pressure

Task concurrency, provider-open rate limits, model endpoint capacity, and resource quotas are independent constraints. Keep async provider calls non-blocking, apply bounded concurrency at the owning layer, and avoid global mutable state in per-task sessions.

Change Impact Map

Before editing, use this table to identify the minimum code and validation surface.

Intended change	Primary owner	Also inspect	Typical regression if misplaced
New dataset version or evaluator	Benchmark	Recipes, docs, result alignment	Old and new scoring semantics become mixed
New agent framework or CLI	Harness	Protocol support, install strategy, result normalization	Benchmark becomes coupled to one agent
New sandbox provider	Environment	Config schema, network, resources, limits, recipes	Provider-specific behavior leaks into benchmarks
Task image or workspace mapping	Recipe	Benchmark task metadata, environment config	Explicit user overrides stop working
New setup/run/verifier phase behavior	Runtime	Every environment and affected benchmark	One provider or evaluation mode bypasses the policy
New shared request or result field	Runtime contract	CLI, SDK, persistence, analyzers, all components	Serialization or old result analysis breaks
New CLI flag	CLI and config	RunRequest, Python SDK, User Guide	CLI and SDK resolve different runs
New metric or analysis	Benchmark or analyzer	Result schema, summary rendering	Authoritative score and post-analysis are conflated

When a reusable runtime or provider capability is required by a benchmark, split the work: land the foundational change first, then rebase the benchmark integration onto it. This keeps platform behavior reviewable without hiding it inside a single benchmark pull request.

Public Surfaces and Compatibility

The stable developer-facing surfaces are:

• agentcompass for Python SDK entry points and supported external recipe types.
• agentcompass.runtime for shared contracts and built-in component implementation.
• Component registries for discovery by stable id.
• Config dataclasses for public component parameters.
• Persisted detail, progress, run-info, and summary artifacts for downstream tooling.

Avoid importing private implementation paths from external integrations. When a contract must change, prefer an additive field with a compatible default, update CLI and SDK construction together, and test both new runs and analysis of an existing result directory.

Continue With an Integration Guide

• General Contributing Workflow
• Benchmark Integration
• Harness Integration
• Environment Integration