> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Understand AgentCompass system boundaries, runtime contracts, execution flow, and the impact of changing each component.

AgentCompass is a composable evaluation runtime. Before changing it, locate the behavior in the system map and identify
the contracts it produces and consumes. This prevents a benchmark-specific requirement from leaking into a harness,
environment provider, or the shared runtime.

## System Overview

The CLI and Python SDK converge on the same orchestration runtime. `run` wraps one `RunRequest`; `launch` resolves an
ordered set of named requests into one `Orchestration`. Registries resolve the requested components, the planner creates
a per-task `ExecutionPlan`, and the shared scheduler coordinates request priority, task concurrency, environment,
benchmark, harness, evaluation, result, and analysis lifecycles.

```mermaid theme={"system"}
flowchart TB
  subgraph interfaces["User interfaces"]
    CLI["CLI"]
    SDK["Python SDK"]
    CONFIG["Configuration and environment variables"]
  end

  ORCHESTRATION["Orchestration<br/>ordered requests · global limits · deadline"]
  REQUEST["RunRequest<br/>model · benchmark · harness · environment · execution · output"]

  subgraph resolution["Resolution and planning"]
    REGISTRY["Component registries"]
    DEPENDENCY["Optional dependency resolution"]
    DATASET["Benchmark task loading and selection"]
    PLANNER["Planner"]
    RECIPE["Recipes<br/>provider-aware plan adaptation"]
    PLAN["ExecutionPlan<br/>one resolved plan per task"]
  end

  subgraph task["Per-task runtime"]
    ENV["Environment provider<br/>open session"]
    PREPARE["Benchmark<br/>prepare task"]
    HARNESS["Harness<br/>start · run · close"]
    MODEL["Model endpoint"]
    COLLECT["Benchmark<br/>collect artifacts"]
    VERIFY["Benchmark<br/>evaluate / verify"]
    CLEANUP["Environment provider<br/>close session"]
  end

  subgraph outputs["Persistence and observability"]
    RESULT["RunResult"]
    STORE["Details · progress · logs · summary"]
    ANALYZER["Analyzers"]
  end

  CLI --> ORCHESTRATION
  SDK --> ORCHESTRATION
  CONFIG --> ORCHESTRATION
  ORCHESTRATION --> REQUEST
  REQUEST --> REGISTRY
  REQUEST --> DEPENDENCY
  REGISTRY --> DATASET
  DATASET --> PLANNER
  RECIPE --> PLANNER
  PLANNER --> PLAN
  PLAN --> ENV
  ENV --> PREPARE
  PREPARE --> HARNESS
  MODEL <--> HARNESS
  HARNESS --> COLLECT
  COLLECT --> VERIFY
  VERIFY --> RESULT
  RESULT --> STORE
  RESULT --> ANALYZER
  VERIFY --> CLEANUP
```

The diagram shows the common harness-backed path. A `HarnessFreeBenchmark` can own the inference loop, but it must
preserve the same task, environment, result, evaluation, and persistence contracts.

## Runtime Shape

The runtime is built around a small set of typed objects. These objects are the stable seams between modules.

```text theme={"system"}
Orchestration
  ├─ global task concurrency, deadline, provider limits, and progress
  └─ ordered OrchestratedRun[]
       └─ RunRequest
            ├─ ModelSpec
            ├─ BenchmarkSpec
            ├─ HarnessSpec
            ├─ EnvironmentSpec
            ├─ ExecutionSpec
            └─ OutputSpec
                 │
                 ├─ load and select -> TaskSpec
                 ├─ plan            -> ExecutionPlan
                 ├─ prepare         -> PreparedTask
                 ├─ run             -> RunResult
                 └─ evaluate        -> normalized RunResult -> persisted artifacts
```

| Contract             | Producer                          | Primary consumers                                         | Change impact                                                                        |
| -------------------- | --------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `Orchestration`      | CLI/SDK launch resolver           | Global scheduler, progress renderer, runtime bootstrap    | Cross-request priority, concurrency, cancellation, failure isolation, CLI/SDK parity |
| `RunRequest`         | CLI, Python SDK, config loader    | Registry, planner, runtime, every component               | Public configuration, serialization, CLI/SDK parity, reproducibility                 |
| `TaskSpec`           | Benchmark loader                  | Task selection, planner, benchmark preparation, analyzers | Dataset identity, recipes, retries, result paths                                     |
| `ExecutionPlan`      | Planner and recipes               | Environment, benchmark, harness, runtime                  | Provider settings, resources, network phases, evaluation environment                 |
| `PreparedTask`       | Benchmark                         | Harness or harness-free inference loop                    | Prompt, files, media, tools, workspace, expected outputs                             |
| `EnvironmentSession` | Environment provider              | Benchmark preparation, harness, verifier                  | Command and file semantics across every sandbox provider                             |
| `RunResult`          | Harness, then benchmark evaluator | Persistence, metrics, analyzers                           | Status semantics, score correctness, trajectories, summaries                         |

Changing one of these contracts is a runtime change, not a local component change. Audit every producer and consumer,
update public exports, and preserve serialization compatibility where existing result artifacts depend on it.

## Component Ownership

| Component   | Owns                                                                                                 | Must not own                                                     | Main source                      |
| ----------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -------------------------------- |
| Benchmark   | Dataset, stable task identity, preparation, scoring, aggregation, evaluator semantics                | Agent loop, provider SDK, generic sandbox lifecycle              | `src/agentcompass/benchmarks/`   |
| Harness     | Agent or model execution loop, harness setup, trajectory and usage normalization                     | Dataset loading, benchmark score, provider image selection       | `src/agentcompass/harnesses/`    |
| Model       | Endpoint identity, API protocol, credentials, inference parameters                                   | Benchmark prompts, agent lifecycle, scoring                      | `ModelSpec` and protocol clients |
| Environment | Commands, files, endpoints, sandbox lifecycle, resources, enforceable network policy                 | Benchmark rules, model decisions, score interpretation           | `src/agentcompass/environments/` |
| Recipe      | Deterministic, per-task plan adaptation for benchmark/provider compatibility                         | Side effects, sandbox creation, inference, scoring               | `src/agentcompass/recipes/`      |
| Runtime     | Cross-component orchestration, concurrency, retries, network phase transitions, cleanup, persistence | Benchmark-specific shortcuts or provider-specific business rules | `src/agentcompass/runtime/`      |
| Analyzer    | Post-run interpretation of normalized results                                                        | Changing the authoritative benchmark score or execution outcome  | `src/agentcompass/analyzers/`    |

Use ownership to decide where a change belongs. For example, a task image stored in dataset metadata belongs to the
benchmark contract; mapping that image to a Daytona snapshot or Modal named image belongs to a recipe; creating and
closing the sandbox belongs to the environment provider.

## Execution Lifecycle

The runtime creates a plan for every selected task because images, resources, workspaces, and verifier requirements can
vary between tasks.

```mermaid theme={"system"}
sequenceDiagram
  participant R as Runtime
  participant B as Benchmark
  participant P as Planner and recipes
  participant E as Environment
  participant H as Harness
  participant V as Evaluator
  participant S as Result store

  R->>B: load_tasks and select_tasks
  loop each selected task, under concurrency limits
    R->>P: build ExecutionPlan
    R->>E: open under setup network policy
    R->>B: prepare_task
    R->>H: start_session
    R->>E: switch to run network policy
    R->>H: run_task
    R->>E: restore setup policy
    R->>B: collect_artifacts
    alt reuse task environment
      R->>E: switch to verifier network policy
      R->>V: evaluate
      R->>E: close task environment
    else fresh verifier environment
      R->>E: close task environment
      R->>E: open fresh verifier environment
      R->>V: evaluate
      R->>E: close verifier environment
    else driver-side evaluation
      R->>E: close task environment
      R->>V: evaluate without environment
    end
    R->>S: persist detail and progress
  end
  R->>S: aggregate metrics, analysis, and summary
```

Cleanup belongs in `finally` paths. A failure in preparation, harness startup, model execution, artifact collection, or
verification must not leak a container, cloud sandbox, proxy, client, or background process.

## Planning and Precedence

Configuration is normalized before execution. Recipes then adapt a copy of the per-task plan without mutating the
original request. Explicit user intent must survive every adaptation layer.

For provider selection fields, use this precedence:

```text theme={"system"}
explicit provider-native selector
  > explicit environment image
  > task metadata
  > recipe fallback
```

For resource dictionaries, start with task defaults and overlay explicit user values field by field. For scalar
defaults such as a workspace root, use `setdefault()` rather than unconditional assignment.

This policy has a deliberate trade-off: automatic recipes make official task images convenient, while user-first
precedence keeps custom prebuilt images and provider-native snapshots possible. Reject incompatible explicit values
with an actionable error; do not silently replace them with a configuration that happens to run.

## Design Principles

### Depend on Contracts, Not Implementations

Components communicate through runtime contracts and public registries. Import shared types from `agentcompass.runtime`
instead of reaching into another component's private module. This keeps a harness reusable across benchmarks and an
environment provider reusable across harnesses.

### Separate Policy From Mechanism

Benchmarks define evaluation policy; environments provide execution mechanisms; the runtime sequences them. A benchmark
may require isolated verification, but it should express that through its evaluation plan rather than creating a Docker
container directly.

### Preserve Explicit User Intent

Defaults and recipes may fill missing values, never overwrite an explicit compatible choice. Convenience that changes a
user-supplied image, resource limit, workspace, network policy, model parameter, or timeout is a correctness bug.

### Keep Planning Pure and Side Effects Scoped

Task loading, plan construction, and recipe application should be deterministic. Network calls, package installation,
sandbox creation, and file mutation belong to explicit lifecycle phases where errors, limits, and cleanup are visible.

### Enforce Security at the Environment Boundary

Prompts and agent instructions are not security controls. Network isolation, resource limits, filesystem boundaries,
and secret handling must be enforced by the provider or runtime. Setup, agent execution, and verification policies are
separate because trusted preparation and untrusted execution have different requirements.

### Make Reproducibility Observable

Record versions, revisions, resolved plans, model and harness settings, evaluator behavior, failures, and task coverage.
A close score without matching settings and denominator is not alignment evidence.

### Fail Early, Preserve Failure Meaning

Validate unsupported versions, protocols, providers, policies, and task ids before expensive work begins. Preserve the
difference between environment errors, harness errors, model errors, agent timeouts, evaluator errors, and valid zero
scores; downstream analysis depends on those distinctions.

### Keep Specialized Dependencies Optional

The default installation should contain framework-essential and broadly shared packages. Benchmark- or harness-specific
driver dependencies belong in optional extras and the trusted dependency workflow. Task-runtime dependencies belong in
the task image or controlled setup phase.

### Bound Concurrency and External Pressure

Task concurrency, provider-open rate limits, model endpoint capacity, and resource quotas are independent constraints.
Keep async provider calls non-blocking, apply bounded concurrency at the owning layer, and avoid global mutable state in
per-task sessions.

## Change Impact Map

Before editing, use this table to identify the minimum code and validation surface.

| Intended change                       | Primary owner         | Also inspect                                             | Typical regression if misplaced                     |
| ------------------------------------- | --------------------- | -------------------------------------------------------- | --------------------------------------------------- |
| New dataset version or evaluator      | Benchmark             | Recipes, docs, result alignment                          | Old and new scoring semantics become mixed          |
| New agent framework or CLI            | Harness               | Protocol support, install strategy, result normalization | Benchmark becomes coupled to one agent              |
| New sandbox provider                  | Environment           | Config schema, network, resources, limits, recipes       | Provider-specific behavior leaks into benchmarks    |
| Task image or workspace mapping       | Recipe                | Benchmark task metadata, environment config              | Explicit user overrides stop working                |
| New setup/run/verifier phase behavior | Runtime               | Every environment and affected benchmark                 | One provider or evaluation mode bypasses the policy |
| New shared request or result field    | Runtime contract      | CLI, SDK, persistence, analyzers, all components         | Serialization or old result analysis breaks         |
| New CLI flag                          | CLI and config        | `RunRequest`, Python SDK, User Guide                     | CLI and SDK resolve different runs                  |
| New metric or analysis                | Benchmark or analyzer | Result schema, summary rendering                         | Authoritative score and post-analysis are conflated |

When a reusable runtime or provider capability is required by a benchmark, split the work: land the foundational change
first, then rebase the benchmark integration onto it. This keeps platform behavior reviewable without hiding it inside
a single benchmark pull request.

## Public Surfaces and Compatibility

The stable developer-facing surfaces are:

* `agentcompass` for Python SDK entry points and supported external recipe types.
* `agentcompass.runtime` for shared contracts and built-in component implementation.
* Component registries for discovery by stable id.
* Config dataclasses for public component parameters.
* Persisted detail, progress, run-info, and summary artifacts for downstream tooling.

Avoid importing private implementation paths from external integrations. When a contract must change, prefer an additive
field with a compatible default, update CLI and SDK construction together, and test both new runs and analysis of an
existing result directory.

## Continue With an Integration Guide

* [General Contributing Workflow](/en/developer_guide/contributing)
* [Benchmark Integration](/en/developer_guide/benchmark_integration)
* [Harness Integration](/en/developer_guide/harness_integration)
* [Environment Integration](/en/developer_guide/environment_integration)