> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Execution, Scheduling, and Cleanup

The shared runtime executes tasks within one request, while the `Orchestrator` schedules work across named requests. Before changing lifecycle behavior, identify whether its scope is one attempt, one task, one request, or the entire orchestration.

The main implementation lives in these files:

```text theme={"system"}
src/agentcompass/runtime/runner.py
src/agentcompass/runtime/orchestration.py
src/agentcompass/runtime/limits.py
src/agentcompass/runtime/base.py
```

## Prepare the request

`UnifiedEvaluationRuntime.prepare` performs request-level work before task workers run:

1. reserve a new output directory and initialize progress reporting;
2. write the initial `run_info.json`;
3. run dependency and component compatibility preflight if it has not already completed;
4. call `Benchmark.load_tasks`, validate task IDs, and call `Benchmark.select_tasks`;
5. materialize reusable normal detail files into the new run;
6. load those details and remove their task IDs from the pending queue.

Task loading and selection happen once per request. `Planner.plan` and `Benchmark.prepare_task` belong to the attempt scope and run again later when needed.

## Run one attempt

For each selected task, `_run_attempts` reads `RunRequest.execution.attempts`, resolves the registered strategy, and asks `AttemptScheduler` which attempt indices still need work. One attempt follows this logical sequence:

```mermaid theme={"system"}
sequenceDiagram
  participant R as Runtime
  participant P as Planner
  participant E as Task Environment
  participant B as Benchmark
  participant H as Harness
  participant V as Evaluator
  participant F as Fresh Eval Environment
  participant A as Analyzer
  participant S as RunStore

  R->>P: build ExecutionPlan
  R->>S: record plan by task and attempt
  R->>E: open under baseline policy
  R->>B: prepare_task
  opt Harness-backed
    R->>H: start_session
  end
  R->>E: switch to run policy when needed
  alt Harness-backed
    R->>H: run_task
    H-->>R: raw RunResult
    R->>H: close_session
  else HarnessFreeBenchmark
    R->>B: run_task
    B-->>R: raw RunResult
  end
  R->>B: collect_artifacts
  alt evaluation mode = reuse
    R->>E: switch to evaluation policy
    R->>V: evaluate with task Environment
    R->>E: restore baseline and close
  else evaluation mode = none
    R->>E: close
    R->>V: evaluate with env=None
  else evaluation mode = fresh
    R->>E: close
    R->>F: open and switch to evaluation policy
    R->>V: evaluate with fresh session
    R->>F: restore baseline and close
  end
  R->>A: analyze evaluated result when enabled
  R->>R: add to in-memory attempts map
```

The runtime builds and records the `ExecutionPlan` before it opens an Environment. It builds a new plan for the next semantic attempt but reuses the current plan for runtime retries within that attempt.

`Benchmark.prepare_task` receives the original `TaskSpec`, the open task `EnvironmentSession`, the complete request, and `plan.benchmark_plan`, and returns a `PreparedTask`. The normal Harness path then adjusts the input, starts the Harness session, switches to the run network policy, executes the task, and closes the Harness session in a nested `finally` block.

For a `HarnessFreeBenchmark`, the request uses the placeholder Harness ID `none`. The runtime skips Harness construction and calls `HarnessFreeBenchmark.run_task` under the same network-policy boundary. Both paths return a raw `RunResult` at this stage; neither scores the task during inference.

After receiving a `RunResult`, the runtime calls `Benchmark.collect_artifacts` while the task Environment is still alive under the run policy. This is the last shared hook that can read agent-created files before the task Environment may be released. It can attach or normalize artifacts, but it does not aggregate request metrics.

`ExecutionPlan.evaluation_environment_mode` determines the Environment passed to `Benchmark.evaluate`:

| Mode    | Task Environment                         | `env` passed to `evaluate()`    | Network and cleanup behavior                                                                      |
| ------- | ---------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------- |
| `reuse` | remains open through evaluation          | task `EnvironmentSession`       | switch to evaluation policy, evaluate, restore baseline, then close or retain                     |
| `none`  | closes before evaluation unless retained | `None`                          | evaluate in the driver process after the task Environment close/retain decision                   |
| `fresh` | closes before evaluation unless retained | newly opened evaluation session | open from the plan, switch to evaluation policy, evaluate, restore baseline, then close or retain |

`Benchmark.evaluate` returns the authoritative score for each attempt. After evaluation, `analyze_task` selects eligible implementations by dataset, data requirements, configuration, and analyzer family. An analysis failure is recorded separately and does not overwrite the Benchmark score or the original failure.

Here, “retain” means `ExecutionSpec.keep_environment=True`. It only skips Environment close calls; it does not change which session reaches the evaluator.

## Separate semantic attempts from runtime retries

The two loops solve different problems:

| Mechanism        | Configuration owner                                  | Plan behavior                    | Result behavior                                                                      |
| ---------------- | ---------------------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------ |
| Semantic attempt | `ExecutionSpec.attempts.k` and registered strategy   | call `Planner.plan` again        | checkpoint one evaluated payload under its stable attempt index                      |
| Runtime retry    | `ExecutionSpec.max_retries` and `retry_pattern_list` | reuse the current attempt's plan | write failed tries to `retry_details/`; only the terminal try enters the task detail |

The retry budget resets for each semantic attempt and is shared by every retryable stage in that attempt. With `retry_pattern_list=None`, any non-empty error can match. A list retries only errors matching at least one regular expression; an empty list matches none.

The failure location determines retry scope:

* preparation, Environment, inference, artifact, or general attempt failures rerun the whole attempt with a new task Environment and the same plan;
* an error result from inference also reruns the whole attempt when it matches;
* a `reuse` evaluation failure reruns the whole attempt because the evaluator shared task Environment state;
* a `none` evaluation failure retries only `Benchmark.evaluate` with the same prepared task and run result;
* a `fresh` evaluation failure reopens the evaluation Environment and retries with the same prepared task and run result.

`RunStore.save_retry_detail` writes the discarded diagnostic payload, retry number, stage, scope, matched pattern, and redacted plan outside `details/`, so retries do not change the countable task set. If retries are exhausted, the runtime preserves the exception trace or returned error in the result.

Each evaluated result is checkpointed under its stable attempt index. The `avg` strategy completes all configured attempts; the `pass` strategy may stop after the first valid successful primary observation. The scheduler restores existing compatible checkpoints before dispatch, so an interrupted `k>1` task resumes only its missing attempts. When the task is terminal, the runtime calls `RunStore.save_partial_result` and then removes its internal attempt checkpoints.

## Orchestrate requests and apply limits

The stages above all belong to one request. Cross-request scheduling belongs to the `Orchestrator`. `Orchestrator._run_orchestration` starts three cooperating groups:

* `_prepare_in_order` prepares requests in declaration order;
* `task_concurrency` copies of `_worker` execute tasks through one global pool;
* `_finalize_ready` starts aggregation when a request becomes eligible to finalize.

Preparation is pipelined. A later request starts preparation only after earlier non-terminal requests are prepared and their total pending work falls below the worker count. `_select_state` chooses the first prepared request with pending tasks in declaration order:

```text theme={"system"}
dispatch pending tasks from the earliest request
  -> when its pending queue is empty, dispatch from the next request
  -> active tasks may overlap tasks from a later request
```

This is ordered priority, not a barrier between complete requests. Round-robin or eager interleaving would change user-visible scheduling semantics. A request becomes eligible to finalize after its own pending and active work drains and every earlier request has cleared its pending queue; finalization may overlap other active tasks. For direct runtime compatibility, `UnifiedEvaluationRuntime.execute` retains a bounded `TaskExecutor` path. Public single-request helpers wrap the request in an `Orchestration`, so they use the central worker pool.

AgentCompass applies independent limits at different ownership layers:

| Limit                            | Scope                                                               | Enforcement point                                                          |
| -------------------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `Orchestration.task_concurrency` | total active Benchmark tasks in one orchestration                   | number of `Orchestrator._worker` tasks                                     |
| provider capacity                | concurrent attempts for an Environment ID in this process           | process-global permit for that Environment ID around `_run_single_attempt` |
| Environment open QPS             | rate of `BaseEnvironment.open` calls for a provider in this process | wrapper installed by `BaseEnvironment.__init_subclass__`                   |
| orchestration timeout            | elapsed time after preflight                                        | `Orchestrator.execute`                                                     |
| cleanup grace                    | cooperative cancellation and thread-cleanup window                  | `Orchestrator._cancel_remaining`                                           |

A provider capacity or open QPS of `0` disables the corresponding limit. Both caches are process-global and safe across event loops. A whole-attempt retry releases and reacquires its provider capacity permit. Task concurrency cannot replace provider capacity because multiple orchestrations may share a provider in the same process.

Network policy belongs to the `ExecutionPlan`, while enforcement belongs to the Environment provider:

| Phase      | Intended work                                                     | Runtime transition                                                              |
| ---------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| baseline   | trusted Environment setup, Benchmark preparation, Harness startup | provider opens the Environment with the plan's baseline policy                  |
| run        | agent or Model execution and artifact collection                  | switch to `plan.run_network_policy` when needed                                 |
| evaluation | Benchmark verification                                            | switch a reused or fresh evaluation session to `plan.evaluation_network_policy` |

`BaseEnvironment.build_config` validates that the provider supports the requested transition. A provider that cannot switch dynamically must reject a plan whose phase policy differs from baseline; it must not silently use broader access. After evaluation, the runtime shields baseline-policy restoration from cancellation before entering the normal cleanup path.

## Clean up and finalize

Resources are released from the innermost owner outward:

```text theme={"system"}
Harness.start_session
  -> Harness.close_session in the run-stage finally block

Environment.open for the task
  -> Environment.close in the attempt finally block

Environment.open for fresh evaluation
  -> Environment.close in the fresh-evaluation finally block

request progress and log resources
  -> UnifiedEvaluationRuntime.close and RunLogRegistry.close
```

`Harness.close_session` runs before `Benchmark.collect_artifacts`, and the task Environment remains available for artifact collection. Driver-side or fresh-Environment evaluation closes the task Environment first; `reuse` retains it through evaluation. Harness or Environment cleanup errors are logged as warnings so they do not erase the original task failure. Cancellation, `KeyboardInterrupt`, and `SystemExit` are re-raised.

`ExecutionSpec.keep_environment=True` is only for explicit retention or debugging; it cannot mask cleanup failures. A new provider's `close` method must still be idempotent and handle partial initialization, failures, and cancellation. See [Environment Validation and Alignment](/en/developer_guide/extensions/environment/validation_and_alignment) for provider-level tests.

`Orchestrator.execute` completes component preflight before starting the orchestration deadline. On timeout or outer `asyncio` cancellation, it freezes the terminal status, cancels preparation and workers, waits within `cleanup_grace_seconds` for owned async tasks and tracked threads, persists the terminal state of unfinished requests, and finally closes logs and progress rendering. The first CLI interrupt requests this cooperative path; a second forces process exit, so cleanup code cannot depend on unlimited time.

Cancellation can arrive during any `await`. After acquiring a session, process, proxy, client, background task, or temporary endpoint, install cleanup immediately and re-raise `asyncio.CancelledError`.

After one request's pending queue and active tasks drain, `UnifiedEvaluationRuntime.finalize` orders new and reused records by the selected task list, delegates aggregation to `Benchmark.aggregate_metrics`, and writes summary and optional analysis artifacts. Its return value represents the complete request, not the per-execution `RunResult`.

Continue with [Results and Reuse](/en/developer_guide/architecture/results_and_reuse) for partial results, aggregate artifacts, and terminal-state shapes.