Execution, Scheduling, and Cleanup

The shared runtime executes tasks within one request, while the Orchestrator schedules work across named requests. Before changing lifecycle behavior, identify whether its scope is one attempt, one task, one request, or the entire orchestration.

The main implementation lives in these files:

src/agentcompass/runtime/runner.py
src/agentcompass/runtime/orchestration.py
src/agentcompass/runtime/limits.py
src/agentcompass/runtime/base.py

Prepare the request

UnifiedEvaluationRuntime.prepare performs request-level work before task workers run:

1. reserve a new output directory and initialize progress reporting;
2. write the initial run_info.json;
3. run dependency and component compatibility preflight if it has not already completed;
4. call Benchmark.load_tasks, validate task IDs, and call Benchmark.select_tasks;
5. materialize reusable normal detail files into the new run;
6. load those details and remove their task IDs from the pending queue.

Task loading and selection happen once per request. Planner.plan and Benchmark.prepare_task belong to the attempt scope and run again later when needed.

Run one attempt

For each selected task, _run_attempts reads RunRequest.execution.attempts, resolves the registered strategy, and asks AttemptScheduler which attempt indices still need work. One attempt follows this logical sequence:

The runtime builds and records the ExecutionPlan before it opens an Environment. It builds a new plan for the next semantic attempt but reuses the current plan for runtime retries within that attempt.

Benchmark.prepare_task receives the original TaskSpec, the open task EnvironmentSession, the complete request, and plan.benchmark_plan, and returns a PreparedTask. The normal Harness path then adjusts the input, starts the Harness session, switches to the run network policy, executes the task, and closes the Harness session in a nested finally block.

For a HarnessFreeBenchmark, the request uses the placeholder Harness ID none. The runtime skips Harness construction and calls HarnessFreeBenchmark.run_task under the same network-policy boundary. Both paths return a raw RunResult at this stage; neither scores the task during inference.

After receiving a RunResult, the runtime calls Benchmark.collect_artifacts while the task Environment is still alive under the run policy. This is the last shared hook that can read agent-created files before the task Environment may be released. It can attach or normalize artifacts, but it does not aggregate request metrics.

ExecutionPlan.evaluation_environment_mode determines the Environment passed to Benchmark.evaluate:

Mode	Task Environment	env passed to evaluate()	Network and cleanup behavior
reuse	remains open through evaluation	task EnvironmentSession	switch to evaluation policy, evaluate, restore baseline, then close or retain
none	closes before evaluation unless retained	None	evaluate in the driver process after the task Environment close/retain decision
fresh	closes before evaluation unless retained	newly opened evaluation session	open from the plan, switch to evaluation policy, evaluate, restore baseline, then close or retain

Benchmark.evaluate returns the authoritative score for each attempt. After evaluation, analyze_task selects eligible implementations by dataset, data requirements, configuration, and analyzer family. An analysis failure is recorded separately and does not overwrite the Benchmark score or the original failure.

Here, “retain” means ExecutionSpec.keep_environment=True. It only skips Environment close calls; it does not change which session reaches the evaluator.

Separate semantic attempts from runtime retries

The two loops solve different problems:

Mechanism	Configuration owner	Plan behavior	Result behavior
Semantic attempt	ExecutionSpec.attempts.k and registered strategy	call Planner.plan again	checkpoint one evaluated payload under its stable attempt index
Runtime retry	ExecutionSpec.max_retries and retry_pattern_list	reuse the current attempt’s plan	write failed tries to retry_details/; only the terminal try enters the task detail

The retry budget resets for each semantic attempt and is shared by every retryable stage in that attempt. With retry_pattern_list=None, any non-empty error can match. A list retries only errors matching at least one regular expression; an empty list matches none.

The failure location determines retry scope:

• preparation, Environment, inference, artifact, or general attempt failures rerun the whole attempt with a new task Environment and the same plan;
• an error result from inference also reruns the whole attempt when it matches;
• a reuse evaluation failure reruns the whole attempt because the evaluator shared task Environment state;
• a none evaluation failure retries only Benchmark.evaluate with the same prepared task and run result;
• a fresh evaluation failure reopens the evaluation Environment and retries with the same prepared task and run result.

RunStore.save_retry_detail writes the discarded diagnostic payload, retry number, stage, scope, matched pattern, and redacted plan outside details/, so retries do not change the countable task set. If retries are exhausted, the runtime preserves the exception trace or returned error in the result.

Each evaluated result is checkpointed under its stable attempt index. The avg strategy completes all configured attempts; the pass strategy may stop after the first valid successful primary observation. The scheduler restores existing compatible checkpoints before dispatch, so an interrupted k>1 task resumes only its missing attempts. When the task is terminal, the runtime calls RunStore.save_partial_result and then removes its internal attempt checkpoints.

Orchestrate requests and apply limits

The stages above all belong to one request. Cross-request scheduling belongs to the Orchestrator. Orchestrator._run_orchestration starts three cooperating groups:

• _prepare_in_order prepares requests in declaration order;
• task_concurrency copies of _worker execute tasks through one global pool;
• _finalize_ready starts aggregation when a request becomes eligible to finalize.

Preparation is pipelined. A later request starts preparation only after earlier non-terminal requests are prepared and their total pending work falls below the worker count. _select_state chooses the first prepared request with pending tasks in declaration order:

dispatch pending tasks from the earliest request
  -> when its pending queue is empty, dispatch from the next request
  -> active tasks may overlap tasks from a later request

This is ordered priority, not a barrier between complete requests. Round-robin or eager interleaving would change user-visible scheduling semantics. A request becomes eligible to finalize after its own pending and active work drains and every earlier request has cleared its pending queue; finalization may overlap other active tasks. For direct runtime compatibility, UnifiedEvaluationRuntime.execute retains a bounded TaskExecutor path. Public single-request helpers wrap the request in an Orchestration, so they use the central worker pool.

AgentCompass applies independent limits at different ownership layers:

Limit	Scope	Enforcement point
Orchestration.task_concurrency	total active Benchmark tasks in one orchestration	number of Orchestrator._worker tasks
provider capacity	concurrent attempts for an Environment ID in this process	process-global permit for that Environment ID around _run_single_attempt
Environment open QPS	rate of BaseEnvironment.open calls for a provider in this process	wrapper installed by BaseEnvironment.__init_subclass__
orchestration timeout	elapsed time after preflight	Orchestrator.execute
cleanup grace	cooperative cancellation and thread-cleanup window	Orchestrator._cancel_remaining

A provider capacity or open QPS of 0 disables the corresponding limit. Both caches are process-global and safe across event loops. A whole-attempt retry releases and reacquires its provider capacity permit. Task concurrency cannot replace provider capacity because multiple orchestrations may share a provider in the same process.

Network policy belongs to the ExecutionPlan, while enforcement belongs to the Environment provider:

Phase	Intended work	Runtime transition
baseline	trusted Environment setup, Benchmark preparation, Harness startup	provider opens the Environment with the plan’s baseline policy
run	agent or Model execution and artifact collection	switch to plan.run_network_policy when needed
evaluation	Benchmark verification	switch a reused or fresh evaluation session to plan.evaluation_network_policy

BaseEnvironment.build_config validates that the provider supports the requested transition. A provider that cannot switch dynamically must reject a plan whose phase policy differs from baseline; it must not silently use broader access. After evaluation, the runtime shields baseline-policy restoration from cancellation before entering the normal cleanup path.

Clean up and finalize

Resources are released from the innermost owner outward:

Harness.start_session
  -> Harness.close_session in the run-stage finally block

Environment.open for the task
  -> Environment.close in the attempt finally block

Environment.open for fresh evaluation
  -> Environment.close in the fresh-evaluation finally block

request progress and log resources
  -> UnifiedEvaluationRuntime.close and RunLogRegistry.close

Harness.close_session runs before Benchmark.collect_artifacts, and the task Environment remains available for artifact collection. Driver-side or fresh-Environment evaluation closes the task Environment first; reuse retains it through evaluation. Harness or Environment cleanup errors are logged as warnings so they do not erase the original task failure. Cancellation, KeyboardInterrupt, and SystemExit are re-raised.

ExecutionSpec.keep_environment=True is only for explicit retention or debugging; it cannot mask cleanup failures. A new provider’s close method must still be idempotent and handle partial initialization, failures, and cancellation. See Environment Validation and Alignment for provider-level tests.

Orchestrator.execute completes component preflight before starting the orchestration deadline. On timeout or outer asyncio cancellation, it freezes the terminal status, cancels preparation and workers, waits within cleanup_grace_seconds for owned async tasks and tracked threads, persists the terminal state of unfinished requests, and finally closes logs and progress rendering. The first CLI interrupt requests this cooperative path; a second forces process exit, so cleanup code cannot depend on unlimited time.

Cancellation can arrive during any await. After acquiring a session, process, proxy, client, background task, or temporary endpoint, install cleanup immediately and re-raise asyncio.CancelledError.

After one request’s pending queue and active tasks drain, UnifiedEvaluationRuntime.finalize orders new and reused records by the selected task list, delegates aggregation to Benchmark.aggregate_metrics, and writes summary and optional analysis artifacts. Its return value represents the complete request, not the per-execution RunResult.

Continue with Results and Reuse for partial results, aggregate artifacts, and terminal-state shapes.