AgentCompass is developed from a source checkout. Install it in editable mode from the repository root, then configure model access on the Setup page.
Requirements
- Python 3.10 or newer (3.12+ recommended).
git for cloning and updating the source checkout.wget and unzip for automatic dataset preparation.- Optional runtime dependencies for selected environments, such as Docker, Modal, Daytona, or HBox credentials.
Clone Repository
git clone https://github.com/open-compass/AgentCompass.git
cd AgentCompass
sudo apt-get update && sudo apt-get install -y wget unzip
Install Python Dependencies
Choose one setup path. Replace <compatible_python_version> with a supported Python version, such as 3.12 or 3.13.
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .
uv venv --python <compatible_python_version>
source .venv/bin/activate
uv pip install -r requirements.txt
uv pip install -e .
conda create -n <env_name> python=<compatible_python_version>
conda activate <env_name>
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .
uv pip install -r requirements/swe.txt
uv pip install -r requirements/mini-swe-agent.txt
Verify Installation
If the environment is activated:
agentcompass --version
agentcompass --help
agentcompass list benchmark
uv and prefer not to activate .venv:
uv run agentcompass --version
uv run agentcompass --help
uv run agentcompass list benchmark
Update From Source
Pull the latest source with rebase to keep local commits on top of upstream changes, then reinstall editable dependencies when requirements or package metadata change. Commit or stash local edits before rebasing.
git pull --rebase
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pip install -e .
git pull --rebase
uv pip install -r requirements.txt
uv pip install -e .
git pull --rebase
conda activate <env_name>
python -m pip install -r requirements.txt
python -m pip install -e .
Data, Cache, And Output Directories
AgentCompass uses predictable local directories by default:
| Path | Purpose |
|---|
data/ | Downloaded or prepared benchmark datasets. Existing datasets are reused as the local dataset cache. |
results/ | Evaluation outputs under results/<benchmark>/<model>/<timestamp>/. |
logs/ inside a run directory | Runtime logs for debugging a specific evaluation. |
agentcompass run <benchmark> <harness> <model> --data-dir <data_dir> --results-dir <results_dir>
Troubleshooting
| Symptom | Check |
|---|
agentcompass: command not found | Activate the environment, reinstall with python -m pip install -e ., or use uv run agentcompass .... |
| Python version mismatch | Run python --version and recreate the environment with Python >=3.10. |
| Dataset preparation fails | Confirm wget, unzip, network access, and write permission for data/ or --data-dir. |
| Remote environment startup fails | Check provider credentials and environment-specific setup for Docker, Modal, Daytona, or HBox. |
| Model provider errors during a run | Configure model endpoint, key, protocol, and provider naming on the Setup page. |
Run Without Editable Install
Editable install is preferred. If you intentionally avoid it, run through the module path:
PYTHONPATH=src python -m agentcompass.cli run \
swebench_verified \
mini_swe_agent \
glm-5.2 \
--env docker \
--benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
--model-base-url "$MODEL_BASE_URL" \
--model-api-key "$MODEL_API_KEY" \
--model-api-protocol openai-chat
