agentcompass launch

Use an explicit orchestration to run multiple models, benchmarks, harnesses, or environments with one global scheduler.

agentcompass launch executes an ordered multi-request orchestration from a YAML or JSON file:

agentcompass launch <orchestration.yaml> [OPTIONS]

agentcompass run executes one evaluation request. A request selects one model, benchmark, harness, and environment, and the selected benchmark may itself contain many tasks. Use agentcompass launch when one operation must coordinate multiple requests, such as comparing models, evaluating several benchmarks, or mixing execution environments.

AgentCompass does not infer a matrix. Every request is named and declared explicitly, which keeps its parameters, results, failures, and reuse source auditable.

Define an Orchestration

The following orchestration evaluates Terminal-Bench 2.1 first and Terminal-Bench 2 Verified next, using one global pool of 16 task slots. Shared model settings are defined once under defaults:

# terminal-evaluations.yaml

# Maximum benchmark samples running concurrently across all requests.
task_concurrency: 16

# Values inherited by every request unless that request overrides them.
defaults:
  model:
    id: ${MODEL_NAME}
    base_url: ${MODEL_BASE_URL}
    api_key: ${MODEL_API_KEY}
    api_protocol: openai-chat
    # Enable only for a gateway that understands the AgentCompass session envelope.
    wrap_api_key: false
    params:
      temperature: 1
      top_p: 0.95

requests:
  # A user-defined request name used in orchestration progress and outcomes.
  - name: tb21
    benchmark:
      id: terminal_bench_2_1
    harness:
      id: terminus2
      max_turns: 300
    environment:
      id: daytona

  - name: tb2vrf
    benchmark:
      id: terminal_bench_2_verified
    harness:
      id: terminus2
      max_turns: 300
    environment:
      id: docker

Export every referenced environment variable before resolving the file:

export MODEL_NAME=""
export MODEL_BASE_URL=""
export MODEL_API_KEY=""

Environment references must occupy the complete field, as in ${MODEL_API_KEY}. AgentCompass rejects partial string interpolation so unresolved or accidentally concatenated secrets do not silently enter a request.

What the fields mean

Field	Meaning
task_concurrency	One global sample-concurrency limit shared by every request. It is not applied independently to each request.
defaults	Values inherited by all requests. A request may override only the fields that differ.
defaults.model.id	The actual model id sent to the endpoint and recorded in result paths.
base_url / api_key / api_protocol	Connection settings for the shared model endpoint. Environment references keep credentials out of the YAML file.
wrap_api_key	When true, wraps the key in the AgentCompass session envelope for a compatible gateway. Keep false for normal model endpoints.
model.params	Model inference parameters forwarded through the selected harness/protocol, such as temperature and top_p.
requests	Ordered evaluation requests admitted by the orchestration scheduler.
requests[].name	A unique, user-defined request label used for progress display and request-level outcomes. It does not select a benchmark.
benchmark.id	The real registered benchmark id, such as terminal_bench_2_1.
harness.id	The real registered harness id, such as terminus2.
harness.max_turns	A harness-specific parameter. Component fields are written beside id, without a params wrapper.
environment.id	The real registered environment-provider id, such as daytona or docker.

The request name therefore remains stable even if its benchmark or environment configuration changes. Use agentcompass list benchmark, agentcompass list harness, and agentcompass list env to inspect valid component ids.

Mapping rules

Section	Shape	Meaning
benchmark, harness, environment	id plus component fields at the same level	id selects the component; every other field becomes one of its parameters. Do not add a params wrapper.
model	id, endpoint fields, optional wrap_api_key, and optional params	Generation and provider request options remain under model.params; leave wrap_api_key disabled unless the endpoint is a compatible internal AgentCompass gateway.
execution	Partial execution mapping	Controls analysis, retries, recipes, and environment retention for the request. The orchestration still owns global task concurrency.
Request runtime	reuse and reuse_run_id	Selects whether that request can reuse an earlier run.
output	run_name and run_id	Organizes the new request result directory.

The optional version defaults to the latest supported orchestration format. Request names must be non-empty and unique.

Validate Before Running

Resolve the complete orchestration before starting an evaluation:

agentcompass launch terminal-evaluations.yaml --dry-run

--dry-run loads configuration layers, expands environment references, resolves component defaults, validates every request, and prints a redacted orchestration. It does not load benchmark tasks or create result directories. Review the selected component ids, task filters, environments, endpoint hostnames, concurrency, and reuse settings in this output.

Start the same orchestration after validation:

agentcompass launch terminal-evaluations.yaml

CLI flags override orchestration-level runtime values when a one-off change is needed:

agentcompass launch terminal-evaluations.yaml \
  --task-concurrency 8 \
  --provider-limit docker=8 \
  --progress plain

Use agentcompass launch --help for the complete option list. The most important launch-only controls are:

Option	Purpose
--task-concurrency <n>	Sets the single global limit shared by tasks from every request.
--timeout-seconds <n>	Sets one wall-clock deadline for the complete orchestration.
--provider-limit <provider>=<n>	Caps simultaneous attempts using a provider; repeat for multiple providers.
--env-open-qps <provider>=<qps>	Limits environment startup rate; repeat for multiple providers.
--progress auto|plain|none	Selects the multi-request terminal renderer.
--auto-install-dependencies	Allows trusted missing host-side extras to be installed; disabled by default.
--reuse	Enables latest matching-run reuse by default for every request.
--run-id <id>	Applies one explicit run id to every new request output.
--dry-run	Resolves, validates, redacts, and prints without executing.

Understand Scheduling and Failure Isolation

All requests share one task worker pool. Declaration order defines admission priority: tasks from an earlier request are admitted first, and later requests use idle slots after all pending tasks from earlier requests have been admitted. This ordering is deterministic, but it does not force one complete evaluation to finish before the next begins.

With task_concurrency: 16, the example behaves as follows:

1. AgentCompass fills available slots with samples from tb21 first.
2. As tb21 samples finish, its remaining unstarted samples continue to receive priority.
3. Once all tb21 samples have been admitted, any free slots immediately begin tb2vrf samples, even if the final tb21 samples are still running.
4. If tb21 contains fewer than 16 samples, the unused slots begin tb2vrf immediately.

This is ordered admission with overlap, not a strict barrier between requests. Request order controls which pending samples get capacity first; task_concurrency controls the total number of samples running across the orchestration.

Each request keeps its own run directory, progress files, logs, summary, and terminal outcome. A request-level failure is recorded as failed and does not prevent later requests from running. The orchestration returns completed when all requests complete, partial_failure when only some fail, and a terminal timeout or cancellation status when the shared operation is stopped.

Progress modes behave as follows:

• auto renders a live multi-request view in a TTY and refreshes it periodically;
• plain prints state changes and periodic snapshots for CI or redirected logs; and
• none disables the terminal progress renderer without removing request log and result files.

Use the capacity guidance in agentcompass run before raising global concurrency.

Reuse Requests Deliberately

This command enables latest-run reuse through defaults.runtime.reuse:

agentcompass launch terminal-evaluations.yaml --reuse

An individual request can opt out with runtime.reuse: false. To select an exact source, set runtime.reuse_run_id in that request or in defaults; output.run_id names the new result and is not a reuse source.

Multiple requests may intentionally share the same benchmark and model. AgentCompass emits a warning because their result hierarchy overlaps. Implicit latest-run reuse is ambiguous for such requests and is rejected; assign an explicit runtime.reuse_run_id to each duplicate benchmark/model request. Explicit output-directory collisions are also rejected before task execution.

Reuse does not prove that two requests have equivalent harness, environment, model endpoint, task selection, or scoring settings. Confirm those settings yourself before treating reused details as part of one evaluation.

Earlier batch helpers and repeatable model flags are no longer the multi-evaluation interface. Keep agentcompass run for one request and use launch() or agentcompass launch for multiple explicit requests.

Related Pages

• agentcompass run
• Python SDK
• Results