Skip to main content
Runtime is the center of AgentCompass. It takes CLI flags or Python arguments, normalizes them into a RunRequest, loads registered components, executes tasks with bounded concurrency, and writes durable artifacts. Use this page when you need to understand what happens after agentcompass run starts.

​
What Runtime Owns

Request construction

Merge config/defaults.yaml, component defaults, CLI overrides, and Python kwargs into RunRequest.

Execution planning

Build an ExecutionPlan for each task and let recipes rewrite it before the environment starts.

Bounded concurrency

Enforce per-run task_concurrency and process-global provider limits.

Artifact persistence

Write run info, task details, summary, progress events, logs, and optional analysis output.
Runtime does not implement benchmark scoring, agent behavior, provider SDK calls, or badcase diagnosis. Those belong to benchmarks, harnesses, environments, and analyzers.

​
Core Objects

ObjectPurpose
RunRequestThe full request: benchmark, harness, environment, model, execution controls, output controls.
ExecutionSpecTask concurrency, enabled recipes, retry count, and analysis settings.
ExecutionPlanPer-task plan after benchmark and harness planning, before and after recipe rewrites.
UnifiedEvaluationRuntimeThe in-process executor used by CLI and Python API.
ProgressEventStructured event emitted during task loading, setup, execution, summary, and release.

​
Execution Flow

run_evaluation(...) / agentcompass run
  -> build RunRequest
  -> load built-in registries
  -> benchmark.load_tasks()
  -> benchmark.select_tasks()
  -> load reusable details
  -> run pending tasks with task_concurrency
  -> process and save summary
  -> optionally save analysis summary
For each pending task: 
planner builds ExecutionPlan
  -> matching recipes apply
  -> environment.open()
  -> benchmark.prepare_task()
  -> harness.start_session()
  -> harness.run_task()
  -> benchmark.evaluate()
  -> analyzers run if enabled
  -> environment.close()

​
CLI Usage

agentcompass run \
  screenspot \
  qwen3vl_gui \
  qwen3-vl \
  --env <env-provider> \
  --benchmark-params '{"category":"desktop"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 8 \
  --results-dir results \
  --data-dir data
The three positional arguments select benchmark, harness, and model. The environment, model endpoint, concurrency, and output directories become fields on the same RunRequest.

​
Python Usage

from agentcompass import run_evaluation

result = run_evaluation(
    benchmark="screenspot",
    harness="qwen3vl_gui",
    model="qwen3-vl",
    environment="host_process",
    benchmark_params={"category": "desktop"},
    model_base_url="https://your-endpoint/v1",
    model_api_key="sk-...",
    model_api_protocol="openai-chat",
    task_concurrency=8,
    results_dir="results",
    data_dir="data",
)
The return value contains aggregate metadata, metrics, summary fields, output paths, and applied recipes. Per-task details remain on disk so large evaluations do not need to keep every artifact in memory.

​
Configuration Precedence

SourceRole
config/defaults.yamlProject defaults for runtime, execution, components, providers, and analysis.
Component defaultsbenchmarks.<id>, harnesses.<id>, and environments.<id> blocks.
CLI flags / Python kwargsHighest-priority run-specific overrides.
RecipesFinal provider-aware plan rewrites before environment startup.

​
Common Patterns

PatternHow to express it
Run one sample--benchmark-params '{"sample_ids":["task-id"]}'
Run with a provider--env <env-provider>; recipes may infer image and workspace for supported benchmark/provider pairs.
Increase throughput--task-concurrency 32, then confirm model endpoint and provider limits.
Reuse existing resultsUse output reuse options from the CLI or config; runtime skips already materialized details.
Post-analyze resultsAdd --enable-analysis during a run or use agentcompass analysis later.

​
Developer Notes

Most extensions do not need to modify runtime. Add or change a benchmark, harness, environment, recipe, or analyzer first. Touch runtime only when you need a new cross-cutting execution concern such as a new progress event, new output persistence behavior, or a new request-level control.