Start From Here
Install AgentCompass
Install the CLI from source with Python 3.12 or newer.
Run the guided example
Enter model and environment settings, review the generated command, and run one real task.
Open the Command Builder
Search the benchmark catalog and generate a ready-to-run complete evaluation command.
Explore GitHub
Browse the source code, follow releases, report issues, and join the community.
What is AgentCompass?
AgentCompass is a unified open-source evaluation framework for next-generation agents. Its highly decoupled modular design makes it easy to extend and integrate. To address the fragmentation, tight component coupling, and limited reproducibility of existing agent evaluation workflows, AgentCompass decouples four core modules—Model, Benchmark, Harness, and Environment—allowing you to switch among tasks, agent workflows, and execution environments under a unified standard without having to reimplement complex execution logic. AgentCompass natively supports widely recognized benchmarks and mainstream agent harnesses and provides stable, secure sandboxed execution environments. Its integrations will continue to expand. It also provides an end-to-end evaluation workflow covering task scheduling, environment interaction, metric aggregation, and trajectory analysis, along with mechanisms for preventing reward hacking. Together, these capabilities provide unified, easy-to-use, and reproducible open-source infrastructure for agent research and multidimensional capability evaluation. For more information, read our technical report on arXiv or visit Hugging Face Daily Papers.AgentCompass runs evaluations directly through the CLI or Python SDK. Its primary runtime path does not require a long-running API server, queue, worker pool, or global LLM gateway.
Key Features
- Composable modular architecture: Decouples Model, Benchmark, Harness, and Environment through stable protocols, enabling existing components to be reused and freely combined across tasks, agents, and execution backends.
- Unified agent execution interface: Supports everything from single-turn model calls to mainstream agent harnesses such as Claude Code, Codex, OpenHands, OpenClaw, Mini-SWE-agent, and Terminus2, all running through a unified session interface on local, Docker, or remote sandboxes.
- Asynchronous, fault-tolerant evaluation runtime: Provides concurrent task scheduling, incremental result persistence, failure retries, and resumable evaluation for long-running, I/O-intensive agent trajectories, reducing the cost of large-scale evaluation.
- Comprehensive trajectory recording and behavioral analysis: Uniformly records reasoning processes, tool calls, environment feedback, token usage, and inference latency, while pluggable analyzers identify failures across models, environments, and frameworks as well as abnormal behaviors such as truncation, repetitive generation, and reward hacking.
- Broad capability and benchmark coverage: Integrates widely recognized benchmarks across multiple capability dimensions, using consistent metric and trajectory protocols to evaluate heterogeneous models and agents.
- Extensible and reproducible: Uses a lightweight registration mechanism to integrate new Benchmarks, Harnesses, Environments, Recipes, and Analyzers, while preserving complete configurations, task logs, and aggregated results so evaluations are auditable, resumable, and reproducible.
Learn More
Explore the User Guide
Learn about advanced runtime options and find reference information for benchmarks, harnesses, environments, and result handling.
Read the Architecture
Understand the runtime contracts, lifecycle, and component design boundaries.