Quick Start

Use the interactive SWE-bench example to quick start and experience a minimal complete workflow of AgentCompass.

The fastest complete AgentCompass workflow is the interactive SWE-bench Verified example that we offer for beginners. It collects model and environment settings, runs one real repository-repair task, summarizes the result, and can open the local trajectory viewer.

Before You Start

Complete Installation, then confirm with the following command:

agentcompass --version

You also need to prepare:

• Access for an OpenAI-compatible model calling.
• Docker, or credentials for remote sandboxes like Daytona or Modal. (Check the detailed support information for your operating system here.)
• Network access for the first time dataset, image, and optional dependency download.

Start the Guided Run

From the repository root:

python examples/run_swebench_verified.py

The script asks for:

1. MODEL_BASE_URL, MODEL_API_KEY, and MODEL_NAME.
2. An execution environment: Docker, Daytona, or Modal.
3. Provider credentials when a remote environment is selected.
4. A final review after it prints the exact redacted command and a parameter table.
5. Confirmation before the evaluation starts.

Entered credentials are passed only to the child process. The script does not modify your shell profile.

Docker

Use a local Docker daemon. This is the shortest path when the machine can pull and run the SWE-bench task image.

docker version

Daytona

Provide a Daytona API key when prompted. DAYTONA_API_URL and DAYTONA_TARGET are optional.

export DAYTONA_API_KEY="..."

Modal

The example can reuse ~/.modal.toml or collect service-token credentials.

export MODAL_TOKEN_ID="..."
export MODAL_TOKEN_SECRET="..."

What the Example Runs

The workflow evaluates astropy__astropy-12907 with mini_swe_agent on swebench_verified. Before starting the subprocess, the script pauses and displays two things:

• A copyable command preview with the model API key redacted.
• A terminal table that maps every important positional argument and flag to its selected value and runtime purpose.

The preview explains the following run configuration:

Parameter	Example value	Purpose
benchmark	swebench_verified	Loads the task and owns patch evaluation.
harness	mini_swe_agent	Runs the coding-agent loop against the prepared repository.
model	$MODEL_NAME	Selects the model under test and its result path.
--env	docker, daytona, or modal	Selects where repository commands and verification execute.
--env-params	Provider-specific JSON when needed	Configures the selected remote environment without changing benchmark settings.
--benchmark-params	One sample_ids value	Restricts the guided run to a single real task.
--model-*	Endpoint, hidden key, openai-chat, temperature 0	Configures model connectivity, protocol, and deterministic sampling.
--task-concurrency	1	Prevents multiple tasks from running during the tutorial.
--results-dir and --run-id	Generated local paths	Isolates the details, summary, logs, and analysis for this run.
--enable-analysis	Enabled	Adds post-run trajectory analysis to the saved result.
--progress and --log-level	auto, ERROR	Keeps lifecycle progress visible while suppressing noisy library logs.

Nothing runs until you approve this preview. The equivalent core CLI shape for Docker is:

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --model-params '{"temperature":0}' \
  --task-concurrency 1 \
  --enable-analysis \
  --progress auto \
  --log-level ERROR

AgentCompass then:

loads one benchmark task
  → resolves the task image and workspace through a recipe
  → opens the selected environment
  → runs Mini-SWE-agent against the repository
  → executes the SWE-bench evaluator
  → saves details, summaries, logs, and analysis

Preview Without Actual Running

Complete the prompts and print the redacted command plus the parameter table without starting a task:

python examples/run_swebench_verified.py --dry-run

Skip the browser visualization:

python examples/run_swebench_verified.py --no-visualization

Read the Result

The script prints the outcome, trajectory step count, tool-call count, duration, analyzer findings, and run directory. Results follow this layout:

results/
└── swebench_verified/
    └── <model>/
        └── <run-id>/
            ├── details/
            ├── logs/
            ├── summary.md
            └── analysis_summary.md

When Node.js and npm are available, the example can install the local result-browser dependencies, start the viewer, and open the task trajectory in your browser.

Continue From Here

Run a complete evaluation

Build a full-benchmark command and choose an appropriate concurrency level.

Learn the CLI

Understand run, summary, analysis, component discovery, and configuration commands.

Choose an environment

Compare Docker, Daytona, Modal, and provider-specific setup.

Inspect result artifacts

Learn the run directory, per-task detail, and aggregate result formats.