Installation

AgentCompass currently needs to be installed from source. This page explains how to prepare the host environment, install AgentCompass, configure optional dependencies, and verify local or remote execution environments.

Prerequisites

Before installation, prepare Git, CA certificates, download and archive-extraction tools, and native build tools for your operating system:

Linux / WSL

On Ubuntu, Debian, and Ubuntu-based WSL distributions:

sudo apt-get update
sudo apt-get install -y \
  build-essential \
  ca-certificates \
  curl \
  git \
  unzip \
  wget

For other Linux distributions, use the appropriate package manager to install these tools. See the official Git installation page for Git-specific instructions.

macOS

macOS includes curl, unzip, and CA certificates. Use the Xcode Command Line Tools for native builds and Homebrew for Git and wget:

xcode-select --install
brew install git wget

Windows

Windows includes curl.exe, and PowerShell provides archive extraction through Expand-Archive. Install Git with WinGet:

winget install --id Git.Git --exact --source winget

Alternatively, install Scoop using its official instructions, then manage Git and wget with:

scoop install git wget

Use WSL 2 for workloads that require /bin/sh, POSIX paths, or Unix build tools.

Before running an evaluation, you also need model endpoint credentials and a supported execution environment. See Supported Operating Systems for details.

After completing these steps, verify that Git and curl are available:

git --version
curl --version

Install AgentCompass

Install AgentCompass in an isolated virtual environment. Do not mix uv, pip, and conda in the same environment unless you understand how each tool resolves dependencies.

First, clone the repository and enter the project directory:

git clone https://github.com/open-compass/AgentCompass.git
cd AgentCompass

Then choose one installation method:

uv (recommended)

Install uv using the official installation guide.

Linux, WSL, or macOS:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.12
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e .

Windows PowerShell:

winget install --id=astral-sh.uv -e
uv python install 3.12
uv venv --python 3.12
.venv\Scripts\Activate.ps1
uv pip install -e .

If the host does not already have Python 3.12, the commands above install and manage a Python 3.12 runtime through uv. See the uv Python installation guide.

pip + venv

First install Python 3.12 from the official Python downloads.

Linux, WSL, or macOS:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Windows PowerShell:

py -3.12 -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e .

Conda

After installing and initializing Conda, run:

conda create -n agentcompass python=3.12
conda activate agentcompass
python -m pip install --upgrade pip
python -m pip install -e .

Verify the installation: With the environment activated, confirm that the Python version is correct and the AgentCompass CLI is available:

python --version
agentcompass --version

Install Optional Dependencies as Needed

The base installation includes only AgentCompass’s core dependencies; you do not need to preinstall every component dependency. When an evaluation starts, AgentCompass checks dependencies as needed for the selected benchmark and harness.

If you plan to run SWE-bench or use mini-swe-agent locally on the host, you can preinstall both sets of optional dependencies:

uv pip install -e ".[swebench,mini-swe-agent]"

If a dependency is missing, AgentCompass stops and displays the appropriate installation command. Install the dependency, then rerun the original command. You can also enable automatic installation for trusted built-in components:

agentcompass run <benchmark> <harness> <model> --auto-install-dependencies

--auto-install-dependencies installs dependencies only in the host Python environment that runs AgentCompass. It does not modify Docker, Daytona, or Modal environments; their dependencies come from the task image or environment configuration.

To prepare an offline environment or review all optional dependencies, see Dependency Management.

Execution Environments

Prepare the execution environment required by each selected benchmark. Running multiple benchmarks may require different environments.

See Environments in the User Guide for selection guidance, parameters, resources, and network configuration.

host_process

host_process runs commands as normal subprocesses and uses the real host filesystem, installed tools, permissions, and network access. It starts quickly but provides no isolation, and results can vary with host state.

Linux and WSL 2 are fully supported. Use macOS only for lightweight or service-backed workloads explicitly supported by the benchmark documentation, because packages, utilities, paths, and evaluation scripts can still depend on Linux. Native Windows is unsupported because the current implementation and common workflows rely on /bin/sh, POSIX paths, permissions, and signals.

Do not use host_process for an untrusted agent or one that can execute commands. It can read, modify, or delete files available to your user account and start processes directly on the host.

See the host_process guide for parameters and safety limits.

Docker

Docker creates an isolated container for each task; its dependencies and filesystem layout come from the task image. It is more consistent than host_process, but the first run may need to pull a large image.

AgentCompass supports local Docker on Linux and WSL 2. Follow the Docker Engine installation guide, then verify:

docker version
docker info
docker run --rm hello-world

If Docker works only with sudo, follow the Linux post-installation guide to configure access:

sudo groupadd docker
sudo usermod -aG docker "$USER"
newgrp docker
docker run --rm hello-world

If the docker group was created during installation, skip the first command.

Membership in the docker group grants root-level privileges on the host.

On WSL 2, choose one topology: install Docker Engine inside the WSL distribution, or enable Docker Desktop WSL integration. Do not maintain both daemons. Keep the checkout in the WSL Linux filesystem, such as ~/code/AgentCompass, rather than under /mnt/c/.

See the Docker guide for registry credentials, smoke tests, and parameters.

Daytona

Daytona runs tasks in cloud sandboxes and supports Linux, WSL, Windows, and macOS. Create an account and an API key with sandbox access, then set the credential:

export DAYTONA_API_KEY="..."

Windows PowerShell:

$env:DAYTONA_API_KEY = "..."

API endpoints, target, and organization settings are optional. Never commit credentials to the repository. See the Daytona guide for complete setup.

Modal

Modal runs tasks in cloud sandboxes and supports Linux, WSL, Windows, and macOS. Create an account, then follow the user account setup guide or service user guide to create a token and set the credentials:

export MODAL_TOKEN_ID="..."
export MODAL_TOKEN_SECRET="..."

Windows PowerShell:

$env:MODAL_TOKEN_ID = "..."
$env:MODAL_TOKEN_SECRET = "..."

The Modal CLI can also store credentials in ~/.modal.toml. Never commit tokens to the repository. See the Modal guide for complete setup.

Supported Operating Systems

AgentCompass is installed on your host machine. Evaluation tasks can run directly on that host, in a local Docker container, or in a cloud sandbox:

Operating system	Install and use AgentCompass	host_process	Local Docker	Daytona / Modal
Linux	Yes	Yes	Yes	Yes
WSL 2 (see installation)	Yes	Yes	Yes	Yes
Windows	Yes	No	No	Yes
macOS	Yes	Limited	No	Yes

Even if Docker Desktop can start Linux containers on native Windows or macOS, AgentCompass does not currently support Docker Desktop as a local benchmark environment. Use WSL 2, Daytona, or Modal for coding, terminal, and other Linux-specific workloads.

Troubleshooting

Symptom	Check
Python version mismatch	Run python --version; recreate the environment with Python >=3.12.
agentcompass: command not found	Activate the environment, reinstall editable mode, or use uv run agentcompass.
uv, pip, or dataset downloads fail	Check DNS, proxy, CA certificates, and HTTPS access to the package index.
Cannot connect to the Docker daemon	Start Docker and run docker info from the same Linux or WSL shell.
Docker works in Windows but not WSL	Enable Docker Desktop integration for that WSL 2 distribution.
Repository operations are slow in WSL	Move the checkout from /mnt/c/ to the WSL Linux filesystem.
Daytona startup fails	Verify the API key and optional API endpoint or target.
Modal authentication fails	Run modal token info and verify the active workspace credentials.
Optional dependency installation fails in a restricted sandbox	Preinstall it in the task image or prepare it before network access is disabled.