agentcompass launch

Use a YAML or JSON orchestration file to coordinate multiple explicit evaluation requests with one global scheduler.

agentcompass launch <orchestration.yaml> [OPTIONS]

Each request in the orchestration still selects one benchmark, harness, model, and environment, using the same structure as an evaluation request created by agentcompass run. Use run for one request. Use launch to compare models, evaluate several benchmarks, or mix harnesses and environments across multiple requests.

AgentCompass does not infer a matrix. Every request is named and declared explicitly, which keeps its parameters, results, failures, and reuse source auditable.

Define an Orchestration

The following orchestration defines two evaluation requests. They share one global pool of 16 Benchmark task slots, while common model settings are defined once under defaults:

# terminal-evaluations.yaml

# Maximum Benchmark tasks running concurrently across all requests.
task_concurrency: 16

# Values inherited by every request unless that request overrides them.
defaults:
  model:
    id: ${MODEL_NAME}
    base_url: ${MODEL_BASE_URL}
    api_key: ${MODEL_API_KEY}
    api_protocol: openai-chat
    params:
      temperature: 1
      top_p: 0.95

requests:
  # A user-defined request name used in orchestration progress and outcomes.
  - name: tb21
    benchmark:
      id: terminal_bench_2_1
    harness:
      id: terminus2
      max_turns: 300
    environment:
      id: daytona

  - name: tb2vrf
    benchmark:
      id: terminal_bench_2_verified
    harness:
      id: terminus2
      max_turns: 300
    environment:
      id: docker

Export every referenced environment variable before resolving the file:

export MODEL_NAME=""
export MODEL_BASE_URL=""
export MODEL_API_KEY=""

Environment references must occupy the complete field, as in ${MODEL_API_KEY}. AgentCompass rejects partial string interpolation so unresolved or accidentally concatenated secrets do not silently enter a request.

What the fields mean

Field	Meaning
task_concurrency	One global Benchmark-task concurrency limit shared by every request. It is not applied independently to each request.
defaults	Values inherited by all requests. A request may override only the fields that differ.
defaults.model.id	The actual model ID sent to the endpoint and recorded in result paths.
base_url / api_key / api_protocol	Connection settings for the shared model endpoint. Environment references keep credentials out of the YAML file.
model.params	Model inference parameters forwarded through the selected harness/protocol, such as temperature and top_p.
requests	Evaluation requests to schedule; declaration order determines scheduling priority.
requests[].name	A unique, user-defined request label used for progress display and request-level outcomes. It does not select a benchmark.
benchmark.id	The real registered benchmark ID, such as terminal_bench_2_1.
harness.id	The real registered harness ID, such as terminus2.
harness.max_turns	A harness-specific parameter. Component fields are written beside id, without a params wrapper.
environment.id	The real registered Environment provider ID, such as daytona or docker.

The request name therefore remains stable even if its benchmark or environment configuration changes. Use agentcompass list benchmark, agentcompass list harness, and agentcompass list env to inspect valid component IDs.

Mapping rules

Section	Shape	Meaning
model	id, endpoint fields, and optional params	Generation and provider request options remain under model.params.
benchmark, harness, environment	id plus component fields at the same level	id selects the component; every other field becomes one of its parameters. Do not add a params wrapper.
execution	Partial execution mapping	Controls analysis, retries, recipes, and environment retention for the request. The orchestration still owns global task concurrency.
Request runtime	reuse and reuse_run_id	Selects whether that request can reuse an earlier run.
output	run_name and run_id	Organizes the new request result directory.

The optional version defaults to the latest supported orchestration format. Request names must be non-empty and unique.

Validate Before Running

Resolve the complete orchestration before starting an evaluation:

agentcompass launch terminal-evaluations.yaml --dry-run

--dry-run loads configuration layers, expands environment references, resolves component defaults, validates every request, and prints a redacted orchestration. It does not load benchmark tasks or create result directories. Review the selected component IDs, task filters, environments, endpoint hostnames, concurrency, and reuse settings in this output.

Start the same orchestration after validation:

agentcompass launch terminal-evaluations.yaml

For a one-off change, CLI options can override the shared run controls in the orchestration file:

agentcompass launch terminal-evaluations.yaml \
  --task-concurrency 8 \
  --provider-limit docker=8 \
  --progress plain

Use agentcompass launch --help for the complete option list. Common orchestration-level options include:

Option	Purpose
--task-concurrency <n>	Sets the Benchmark-task concurrency limit shared by every request.
--timeout-seconds <n>	Sets one wall-clock deadline for the complete orchestration. The default is 360000 seconds (100 hours); explicitly set 0 to disable it.
--provider-limit <provider>=<n>	Caps simultaneous attempts using a provider; repeat for multiple providers.
--env-open-qps <provider>=<qps>	Limits environment startup rate; repeat for multiple providers.
--progress auto|plain|none	Selects the multi-request terminal renderer.
--auto-install-dependencies	Allows trusted missing host-side extras to be installed; disabled by default.
--reuse	Enables latest matching-run reuse by default for every request.
--run-id <id>	Applies one explicit run ID to every new request output.
--dry-run	Resolves, validates, redacts, and prints without executing.

Understand Scheduling and Failure Isolation

All requests share one task worker pool. Declaration order defines admission priority: tasks from an earlier request are admitted first, and later requests use idle slots after all pending tasks from earlier requests have been admitted. This ordering is deterministic, but it does not force one complete evaluation to finish before the next begins.

In the task_concurrency: 16 example under Define an Orchestration:

1. AgentCompass fills available slots with tasks from tb21 first.
2. As tb21 tasks finish, its remaining unstarted tasks continue to receive priority.
3. Once all tb21 tasks have been admitted, any free slots immediately begin tb2vrf tasks, even if the final tb21 tasks are still running.
4. If tb21 contains fewer than 16 tasks, the unused slots begin tb2vrf immediately.

This is ordered admission with overlap, not a strict barrier between requests. Request order controls which pending tasks get capacity first; task_concurrency controls the total number of Benchmark tasks running across the orchestration.

Each request keeps its own run directory, progress files, logs, summary, and terminal outcome. A request-level failure is recorded as failed and does not prevent later requests from running. The orchestration returns completed when all requests complete, partial_failure when only some fail, and a terminal timeout or cancellation status when the shared operation is stopped.

See Logs and Progress for the terminal behavior of all three progress modes and how they interact with progress files.

Use the capacity guidance in Run Controls before raising global concurrency.

Reuse Existing Runs

--reuse enables latest-run reuse by default for every request in the orchestration:

agentcompass launch terminal-evaluations.yaml --reuse

An individual request can opt out with runtime.reuse: false. To select an exact source, set runtime.reuse_run_id in that request or in defaults; output.run_id names the new result and is not a reuse source.

Multiple requests may intentionally share the same benchmark and model. AgentCompass emits a warning because their result hierarchy overlaps. Implicit latest-run reuse is ambiguous for such requests and is rejected; assign an explicit runtime.reuse_run_id to each duplicate benchmark/model request. Explicit output-directory collisions are also rejected before task execution.

See Resume an Interrupted Run for reuse matching and usage constraints.

Related Pages

• Run Controls
• agentcompass run
• Python SDK
• Results