> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Runtime Contracts and Planning

This page explains how configuration becomes a `RunRequest`, how the Planner derives an `ExecutionPlan` for each task attempt, and how those types define component boundaries. See [Execution, Scheduling, and Cleanup](/en/developer_guide/architecture/execution_lifecycle) for sequencing and [Results and Reuse](/en/developer_guide/architecture/results_and_reuse) for persisted formats.

Public exports are assembled in `src/agentcompass/runtime/__init__.py`, concrete dataclasses live under `src/agentcompass/runtime/models/`, and component interfaces live in `src/agentcompass/runtime/base.py`. Configuration and planning are implemented primarily in these files:

```text theme={"system"}
src/agentcompass/runtime/config/loader.py
src/agentcompass/launcher.py
src/agentcompass/runtime/orchestration.py
src/agentcompass/runtime/planner.py
src/agentcompass/runtime/runner.py
```

## Request and configuration resolution

`RunRequest` is the complete input for one run and contains eight sections:

```text theme={"system"}
RunRequest
  ├─ model: ModelSpec
  ├─ benchmark: BenchmarkSpec
  ├─ harness: HarnessSpec
  ├─ environment: EnvironmentSpec
  ├─ execution: ExecutionSpec
  ├─ runtime: RunRuntimeSpec
  ├─ output: OutputSpec
  └─ metadata: RunMetadata
```

| Section       | Meaning                                                                              | Typical consumers                                                  |
| ------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------ |
| `model`       | Model ID, endpoint, API key, protocol, inference params, and key wrapping            | Harness or `HarnessFreeBenchmark`                                  |
| `benchmark`   | Benchmark ID and Benchmark-specific params                                           | Registry, Benchmark, and result identity                           |
| `harness`     | Harness ID and Harness-specific params                                               | Registry and Harness; `id="none"` selects a harness-free Benchmark |
| `environment` | Environment ID, provider params, phase network policies, and evaluation mode         | Planner and Environment provider                                   |
| `execution`   | Task concurrency, Recipes, analysis, Environment retention, and runtime retry policy | Orchestrator, Planner, runtime, and analyzers                      |
| `runtime`     | Result reuse controls for this request                                               | `RunStore` and orchestration validation                            |
| `output`      | Result namespace and requested run ID                                                | `RunStore`                                                         |
| `metadata`    | Loaded config path and trusted external Recipe directories                           | Launcher, queue serialization, and run-local Recipe registry       |

`ModelSpec` is a value object, not a registered component. There is no `MODELS` registry. Registries resolve Benchmark, Harness, Environment, Recipe, and Analyzer classes; the selected Harness or harness-free Benchmark consumes `req.model` directly. When you add a field to `ModelSpec`, audit request construction, redaction, persistence signatures, and every Harness that consumes it. Do not add a registry entry for a normal endpoint configuration.

Do not confuse `RunRuntimeSpec` with `ResolvedRuntimeOptions`. The former belongs to one request and currently controls reuse. The latter lives in the runtime's `models/orchestration.py` module and controls process-wide behavior such as result paths, deadline, cleanup grace, provider limits, Environment opening rate, progress, and logging.

`load_run_config` reads existing configuration files in this order:

```text theme={"system"}
user config
  -> nearest project config.yaml
  -> each explicit config_path, in argument order
```

`deep_merge` recursively combines mappings without mutating either input. A later scalar or list replaces the earlier value, while nested mappings merge by key. A whole-field `${VAR}` reference resolves from the environment; embedding `${VAR}` inside a larger string is rejected. Run config component sections are `benchmarks`, `harnesses`, and `environments`, with each selected component's parameters flat under its ID. A `models` section is rejected because Model is supplied through `ModelSpec`.

Single-run entry points such as `build_run_request`, `run_evaluation`, and the `run` CLI use the following precedence. Each arrow points from lower to higher precedence:

| Value                                      | Resolution order                                                               |
| ------------------------------------------ | ------------------------------------------------------------------------------ |
| Benchmark, Harness, and Environment params | Selected run-config component entry → explicit helper or CLI params            |
| `ExecutionSpec` fields                     | Dataclass defaults → run config `execution` → explicit non-`None` arguments    |
| Process runtime options                    | Runtime defaults → run config `runtime` → explicit non-`None` arguments        |
| Model                                      | Explicit Model arguments only                                                  |
| Output and per-request reuse               | Explicit arguments; configured runtime reuse applies when reuse is unspecified |

If a caller passes a constructed `RunRequest` to `run_evaluation_request`, `_merge_request_with_config` treats request component params as overrides over the selected config entries. It preserves the request's Model, execution settings, output settings, and explicit reuse value, then records the loaded config path and resolved Recipe directories in `RunMetadata`.

The orchestration path uses `resolve_orchestration` to resolve each named request from an `OrchestrationSpec`:

| Value                                        | Low-to-high precedence                                                                                       |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Component params                             | Selected run-config component entry → orchestration `defaults` → named request                               |
| Model fields                                 | Orchestration `defaults.model` → named request `model`                                                       |
| Execution fields other than task concurrency | Dataclass defaults → run config `execution` → orchestration `defaults.execution` → named request `execution` |
| Per-request runtime fields                   | `RunRuntimeSpec` defaults and configured reuse → orchestration `defaults.runtime` → named request `runtime`  |
| Output fields                                | `OutputSpec` defaults → orchestration `defaults.output` → named request `output`                             |
| Orchestration runtime                        | Runtime defaults → run config `runtime` → orchestration-file `runtime` → SDK/CLI runtime overrides           |

Global `task_concurrency` is resolved separately: explicit resolver argument → orchestration value → run-config execution value → `ExecutionSpec` default. The `from_requests()` class method of `Orchestration` writes the resolved value into every request's `ExecutionSpec`, keeping the single-request and multi-request paths consistent. Resolution also rejects unknown orchestration fields and validates Benchmark, Harness, and Environment IDs.

An `Orchestration` contains ordered `OrchestratedRun` objects. Each object holds a stable orchestration key, declaration index, request name, and complete `RunRequest`. `RequestOutcome` represents one request's terminal state, and `OrchestrationResult` preserves every outcome in declaration order.

## Task and plan contracts

A Benchmark progressively turns dataset material into runtime input:

```text theme={"system"}
BaseBenchmark.load_tasks
  -> TaskSpec
  -> Planner.plan
  -> ExecutionPlan
  -> BaseBenchmark.prepare_task
  -> PreparedTask
```

| Contract        | Producer                            | Required meaning                                                                                            |
| --------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `TaskSpec`      | `BaseBenchmark.load_tasks`          | Stable `task_id`, question, category, ground truth, metadata, and optional phase-policy hints               |
| `BenchmarkPlan` | `BaseBenchmark.build_plan`          | Benchmark-private, task-specific preparation and evaluation settings                                        |
| `HarnessPlan`   | `BaseHarness.build_plan`            | Harness runtime settings after config validation                                                            |
| `ExecutionPlan` | `Planner.plan` and matching Recipes | Resolved task/evaluation Environments, phase policies, subplans, execution settings, and applied Recipe IDs |
| `PreparedTask`  | `BaseBenchmark.prepare_task`        | Harness-facing input, expected output, ground truth, category, and metadata                                 |

`PreparedTask.input` is a `TaskInput` containing a prompt plus optional system prompt, media, files, workspace, tools, and messages. `PreparedTask.output` is a `TaskOutput` containing an expected answer and output-file declarations. Keep dataset-native records in `TaskSpec.metadata`; expose only execution-ready material through `PreparedTask`.

An `ExecutionPlan` is the resolved plan for one task attempt, not an immutable plan shared across the entire task. During execution, the `_run_attempts` method of `UnifiedEvaluationRuntime` calls `Planner.plan` inside the semantic-attempt loop and outside the runtime-retry loop. Each semantic attempt therefore receives a new plan, while runtime retries allowed by `max_retries` reuse that attempt's plan. Recipe code must not depend on side effects from an earlier planning call. See the [Source Map](/en/developer_guide/architecture/source_map) for the owning file and surrounding call chain.

Planning performs these steps in order:

1. Resolve the evaluation Environment mode.
2. Resolve baseline, run, and evaluation network-policy inputs.
3. Deep-copy the request's `EnvironmentSpec` into a task-local spec.
4. Call `benchmark.build_plan` to create the Benchmark subplan.
5. Call `harness.build_plan`, or create the base `HarnessPlan` for a harness-free Benchmark.
6. Apply each allowed, matching Recipe in registry order.
7. Fill missing policies with `NetworkPolicy()`, whose default mode is `public`.
8. Create or clear `evaluation_environment` according to the final evaluation mode.

The initial evaluation mode uses this precedence:

```text theme={"system"}
RunRequest.environment.evaluation_environment_mode
  > TaskSpec.evaluation_environment_mode
  > BaseBenchmark.resolve_evaluation_environment_mode(req)
```

For each phase, an explicit request network policy wins over a `TaskSpec` hint. Matching Recipes may then transform the value; only a value that remains missing falls back to `public`. Recipes must preserve compatible explicit user choices. After planning, the runtime calls `_resolve_execution_plan_network_policies` so the selected provider can resolve and validate the policies it will enforce. Only a redacted audit view is persisted before the Environment opens.

An orchestration dry run resolves configuration, validates component compatibility, and prints complete requests, but it neither loads tasks nor calls `Planner.plan`. To inspect a task plan, open `resolved_execution_plans` in `run_info.json`, locate the task by `task_id`, enter `attempts`, and select the attempt number. You can also consume the `execution_plan_resolved` progress event. The persisted view contains only Environment IDs, phase policies, the evaluation mode, and applied Recipe IDs. Add safe fields to `_resolved_execution_plan_payload` when you need more diagnostics; do not persist an unrestricted dataclass dump.

## Recipe planning

Built-in Recipes register through `RECIPES` in `src/agentcompass/runtime/registry.py`. The `build_run_recipe_registry` function in `src/agentcompass/runtime/recipes.py` copies those entries into a run-local registry, then appends validated Recipe classes loaded from trusted `RunMetadata.recipe_dirs` directories.

`ExecutionSpec.enabled_recipes` controls the candidate set:

* An empty list considers every Recipe in the run-local registry and lets `matches` select applicable entries.
* A non-empty list is a whitelist of Recipe IDs.
* Every matching Recipe receives the plan returned by the preceding Recipe, and its ID is appended to `ExecutionPlan.applied_recipes`.

The Planner iterates in registry insertion order and does not sort candidates. Avoid overlapping Recipes whose correctness depends on an implicit order. If two adaptations must compose, make the relationship explicit and test the combined plan.

`BaseRecipe` exposes two methods:

```python theme={"system"}
def matches(self, req: RunRequest, task: TaskSpec, plan: ExecutionPlan) -> bool: ...

def apply(
    self,
    plan: ExecutionPlan,
    req: RunRequest,
    task: TaskSpec,
) -> ExecutionPlan: ...
```

`matches` and `apply` run during planning. They must not open an Environment, install software, call a Model, mutate task files, or evaluate an answer. `apply` should return a copied plan and preserve compatible explicit user values. For a reusable example, see `clone_execution_plan` in `common.py` under `src/agentcompass/recipes/swebench_verified/`.

A Recipe reads stable requirements from `TaskSpec` or the Benchmark subplan, then adapts Environment parameters and Benchmark/Harness execution details in the plan. The Environment provider owns sandbox creation and policy enforcement; the Benchmark owns scoring.

## Environment session contract

`BaseEnvironment.open` returns an `EnvironmentSession`. The session defines async command execution, upload and download, text I/O, directory transfer, endpoint discovery, file checks, and optional dynamic network switching. See `EnvironmentSession` in `src/agentcompass/runtime/base.py` for exact signatures.

The command contract is explicit: `exec(..., shell=False)` accepts `list[str]`, while `exec(..., shell=True)` accepts a string. Providers should preserve the semantics of `ExecResult.returncode`, `stdout`, `stderr`, and `timed_out`. See [Execution, Scheduling, and Cleanup](/en/developer_guide/architecture/execution_lifecycle) for Environment opening, reuse, and cleanup.

## Compatibility and result-type boundaries

`RunResult` represents one execution or evaluation attempt, not a request-level summary. A Harness records status, answer, trajectory, artifacts, and execution errors. A Benchmark's `evaluate` method owns `correct`, `score`, and evaluation errors, while `aggregate_metrics` owns request-level metrics and returns a validated `MetricResult`. The runtime shapes attempts into task details and request summaries; the Orchestrator uses `RequestOutcome` and `OrchestrationResult` for named requests and the orchestration's terminal state.

`TaskStatus.COMPLETED`, `RUN_ERROR`, `EVAL_ERROR`, and `SKIPPED` carry different meanings. `TaskStatus.ERROR` serializes as `run_error_or_eval_error` when the failing stage cannot be narrowed further. Do not turn a valid `score=0` into an error or drop the error stage during normalization. See [Results and Reuse](/en/developer_guide/architecture/results_and_reuse) for the persisted shape of each layer.

When you change a shared contract, inspect these boundaries:

* Runtime dataclass construction, validation, and public exports.
* CLI, SDK, and orchestration code that constructs request and result objects.
* Benchmark, Harness, Environment, Recipe, and Analyzer producers and consumers.
* `RunRequest.to_task_payload`, `to_persistence_params`, detail shaping, redaction, and reuse loading.
* Deserialization during analysis and summary reconstruction of an existing result directory.

Prefer additive fields with validated defaults. If a persisted format changes, support existing artifacts or state the schema boundary and migration explicitly.