Runtime Contracts and Planning

This page explains how configuration becomes a RunRequest, how the Planner derives an ExecutionPlan for each task attempt, and how those types define component boundaries. See Execution, Scheduling, and Cleanup for sequencing and Results and Reuse for persisted formats.

Public exports are assembled in src/agentcompass/runtime/__init__.py, concrete dataclasses live under src/agentcompass/runtime/models/, and component interfaces live in src/agentcompass/runtime/base.py. Configuration and planning are implemented primarily in these files:

src/agentcompass/runtime/config/loader.py
src/agentcompass/launcher.py
src/agentcompass/runtime/orchestration.py
src/agentcompass/runtime/planner.py
src/agentcompass/runtime/runner.py

Request and configuration resolution

RunRequest is the complete input for one run and contains eight sections:

RunRequest
  ├─ model: ModelSpec
  ├─ benchmark: BenchmarkSpec
  ├─ harness: HarnessSpec
  ├─ environment: EnvironmentSpec
  ├─ execution: ExecutionSpec
  ├─ runtime: RunRuntimeSpec
  ├─ output: OutputSpec
  └─ metadata: RunMetadata

Section	Meaning	Typical consumers
model	Model ID, endpoint, API key, protocol, inference params, and key wrapping	Harness or HarnessFreeBenchmark
benchmark	Benchmark ID and Benchmark-specific params	Registry, Benchmark, and result identity
harness	Harness ID and Harness-specific params	Registry and Harness; id="none" selects a harness-free Benchmark
environment	Environment ID, provider params, phase network policies, and evaluation mode	Planner and Environment provider
execution	Task concurrency, Recipes, analysis, Environment retention, and runtime retry policy	Orchestrator, Planner, runtime, and analyzers
runtime	Result reuse controls for this request	RunStore and orchestration validation
output	Result namespace and requested run ID	RunStore
metadata	Loaded config path and trusted external Recipe directories	Launcher, queue serialization, and run-local Recipe registry

ModelSpec is a value object, not a registered component. There is no MODELS registry. Registries resolve Benchmark, Harness, Environment, Recipe, and Analyzer classes; the selected Harness or harness-free Benchmark consumes req.model directly. When you add a field to ModelSpec, audit request construction, redaction, persistence signatures, and every Harness that consumes it. Do not add a registry entry for a normal endpoint configuration.

Do not confuse RunRuntimeSpec with ResolvedRuntimeOptions. The former belongs to one request and currently controls reuse. The latter lives in the runtime’s models/orchestration.py module and controls process-wide behavior such as result paths, deadline, cleanup grace, provider limits, Environment opening rate, progress, and logging.

load_run_config reads existing configuration files in this order:

user config
  -> nearest project config.yaml
  -> each explicit config_path, in argument order

deep_merge recursively combines mappings without mutating either input. A later scalar or list replaces the earlier value, while nested mappings merge by key. A whole-field ${VAR} reference resolves from the environment; embedding ${VAR} inside a larger string is rejected. Run config component sections are benchmarks, harnesses, and environments, with each selected component’s parameters flat under its ID. A models section is rejected because Model is supplied through ModelSpec.

Single-run entry points such as build_run_request, run_evaluation, and the run CLI use the following precedence. Each arrow points from lower to higher precedence:

Value	Resolution order
Benchmark, Harness, and Environment params	Selected run-config component entry → explicit helper or CLI params
ExecutionSpec fields	Dataclass defaults → run config execution → explicit non-None arguments
Process runtime options	Runtime defaults → run config runtime → explicit non-None arguments
Model	Explicit Model arguments only
Output and per-request reuse	Explicit arguments; configured runtime reuse applies when reuse is unspecified

If a caller passes a constructed RunRequest to run_evaluation_request, _merge_request_with_config treats request component params as overrides over the selected config entries. It preserves the request’s Model, execution settings, output settings, and explicit reuse value, then records the loaded config path and resolved Recipe directories in RunMetadata.

The orchestration path uses resolve_orchestration to resolve each named request from an OrchestrationSpec:

Value	Low-to-high precedence
Component params	Selected run-config component entry → orchestration defaults → named request
Model fields	Orchestration defaults.model → named request model
Execution fields other than task concurrency	Dataclass defaults → run config execution → orchestration defaults.execution → named request execution
Per-request runtime fields	RunRuntimeSpec defaults and configured reuse → orchestration defaults.runtime → named request runtime
Output fields	OutputSpec defaults → orchestration defaults.output → named request output
Orchestration runtime	Runtime defaults → run config runtime → orchestration-file runtime → SDK/CLI runtime overrides

Global task_concurrency is resolved separately: explicit resolver argument → orchestration value → run-config execution value → ExecutionSpec default. The from_requests() class method of Orchestration writes the resolved value into every request’s ExecutionSpec, keeping the single-request and multi-request paths consistent. Resolution also rejects unknown orchestration fields and validates Benchmark, Harness, and Environment IDs.

An Orchestration contains ordered OrchestratedRun objects. Each object holds a stable orchestration key, declaration index, request name, and complete RunRequest. RequestOutcome represents one request’s terminal state, and OrchestrationResult preserves every outcome in declaration order.

Task and plan contracts

A Benchmark progressively turns dataset material into runtime input:

BaseBenchmark.load_tasks
  -> TaskSpec
  -> Planner.plan
  -> ExecutionPlan
  -> BaseBenchmark.prepare_task
  -> PreparedTask

Contract	Producer	Required meaning
TaskSpec	BaseBenchmark.load_tasks	Stable task_id, question, category, ground truth, metadata, and optional phase-policy hints
BenchmarkPlan	BaseBenchmark.build_plan	Benchmark-private, task-specific preparation and evaluation settings
HarnessPlan	BaseHarness.build_plan	Harness runtime settings after config validation
ExecutionPlan	Planner.plan and matching Recipes	Resolved task/evaluation Environments, phase policies, subplans, execution settings, and applied Recipe IDs
PreparedTask	BaseBenchmark.prepare_task	Harness-facing input, expected output, ground truth, category, and metadata

PreparedTask.input is a TaskInput containing a prompt plus optional system prompt, media, files, workspace, tools, and messages. PreparedTask.output is a TaskOutput containing an expected answer and output-file declarations. Keep dataset-native records in TaskSpec.metadata; expose only execution-ready material through PreparedTask.

An ExecutionPlan is the resolved plan for one task attempt, not an immutable plan shared across the entire task. During execution, the _run_attempts method of UnifiedEvaluationRuntime calls Planner.plan inside the semantic-attempt loop and outside the runtime-retry loop. Each semantic attempt therefore receives a new plan, while runtime retries allowed by max_retries reuse that attempt’s plan. Recipe code must not depend on side effects from an earlier planning call. See the Source Map for the owning file and surrounding call chain.

Planning performs these steps in order:

1. Resolve the evaluation Environment mode.
2. Resolve baseline, run, and evaluation network-policy inputs.
3. Deep-copy the request’s EnvironmentSpec into a task-local spec.
4. Call benchmark.build_plan to create the Benchmark subplan.
5. Call harness.build_plan, or create the base HarnessPlan for a harness-free Benchmark.
6. Apply each allowed, matching Recipe in registry order.
7. Fill missing policies with NetworkPolicy(), whose default mode is public.
8. Create or clear evaluation_environment according to the final evaluation mode.

The initial evaluation mode uses this precedence:

RunRequest.environment.evaluation_environment_mode
  > TaskSpec.evaluation_environment_mode
  > BaseBenchmark.resolve_evaluation_environment_mode(req)

For each phase, an explicit request network policy wins over a TaskSpec hint. Matching Recipes may then transform the value; only a value that remains missing falls back to public. Recipes must preserve compatible explicit user choices. After planning, the runtime calls _resolve_execution_plan_network_policies so the selected provider can resolve and validate the policies it will enforce. Only a redacted audit view is persisted before the Environment opens.

An orchestration dry run resolves configuration, validates component compatibility, and prints complete requests, but it neither loads tasks nor calls Planner.plan. To inspect a task plan, open resolved_execution_plans in run_info.json, locate the task by task_id, enter attempts, and select the attempt number. You can also consume the execution_plan_resolved progress event. The persisted view contains only Environment IDs, phase policies, the evaluation mode, and applied Recipe IDs. Add safe fields to _resolved_execution_plan_payload when you need more diagnostics; do not persist an unrestricted dataclass dump.

Recipe planning

Built-in Recipes register through RECIPES in src/agentcompass/runtime/registry.py. The build_run_recipe_registry function in src/agentcompass/runtime/recipes.py copies those entries into a run-local registry, then appends validated Recipe classes loaded from trusted RunMetadata.recipe_dirs directories.

ExecutionSpec.enabled_recipes controls the candidate set:

• An empty list considers every Recipe in the run-local registry and lets matches select applicable entries.
• A non-empty list is a whitelist of Recipe IDs.
• Every matching Recipe receives the plan returned by the preceding Recipe, and its ID is appended to ExecutionPlan.applied_recipes.

The Planner iterates in registry insertion order and does not sort candidates. Avoid overlapping Recipes whose correctness depends on an implicit order. If two adaptations must compose, make the relationship explicit and test the combined plan.

BaseRecipe exposes two methods:

def matches(self, req: RunRequest, task: TaskSpec, plan: ExecutionPlan) -> bool: ...

def apply(
    self,
    plan: ExecutionPlan,
    req: RunRequest,
    task: TaskSpec,
) -> ExecutionPlan: ...

matches and apply run during planning. They must not open an Environment, install software, call a Model, mutate task files, or evaluate an answer. apply should return a copied plan and preserve compatible explicit user values. For a reusable example, see clone_execution_plan in common.py under src/agentcompass/recipes/swebench_verified/.

A Recipe reads stable requirements from TaskSpec or the Benchmark subplan, then adapts Environment parameters and Benchmark/Harness execution details in the plan. The Environment provider owns sandbox creation and policy enforcement; the Benchmark owns scoring.

Environment session contract

BaseEnvironment.open returns an EnvironmentSession. The session defines async command execution, upload and download, text I/O, directory transfer, endpoint discovery, file checks, and optional dynamic network switching. See EnvironmentSession in src/agentcompass/runtime/base.py for exact signatures.

The command contract is explicit: exec(..., shell=False) accepts list[str], while exec(..., shell=True) accepts a string. Providers should preserve the semantics of ExecResult.returncode, stdout, stderr, and timed_out. See Execution, Scheduling, and Cleanup for Environment opening, reuse, and cleanup.

Compatibility and result-type boundaries

RunResult represents one execution or evaluation attempt, not a request-level summary. A Harness records status, answer, trajectory, artifacts, and execution errors. A Benchmark’s evaluate method owns correct, score, and evaluation errors, while aggregate_metrics owns request-level metrics and returns a validated MetricResult. The runtime shapes attempts into task details and request summaries; the Orchestrator uses RequestOutcome and OrchestrationResult for named requests and the orchestration’s terminal state.

TaskStatus.COMPLETED, RUN_ERROR, EVAL_ERROR, and SKIPPED carry different meanings. TaskStatus.ERROR serializes as run_error_or_eval_error when the failing stage cannot be narrowed further. Do not turn a valid score=0 into an error or drop the error stage during normalization. See Results and Reuse for the persisted shape of each layer.

When you change a shared contract, inspect these boundaries:

• Runtime dataclass construction, validation, and public exports.
• CLI, SDK, and orchestration code that constructs request and result objects.
• Benchmark, Harness, Environment, Recipe, and Analyzer producers and consumers.
• RunRequest.to_task_payload, to_persistence_params, detail shaping, redaction, and reuse loading.
• Deserialization during analysis and summary reconstruction of an existing result directory.

Prefer additive fields with validated defaults. If a persisted format changes, support existing artifacts or state the schema boundary and migration explicitly.