AgentCompass

The framework paper introduces AgentCompass as shared infrastructure for evaluating agents across tasks, execution harnesses, and environments.

Field	Details
Paper	AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
First submitted	2026-07-15
Acceptance	Accepted to EMNLP 2026 Demo Track
Topics	Evaluation infrastructure, reproducibility, trajectory analysis

Overview

The paper separates Benchmark, Harness, and Environment responsibilities so researchers can compose evaluation pipelines without rebuilding execution logic. It describes an asynchronous runtime with fault tolerance and trajectory analysis for investigating agent failures, including reward hacking.

Relation to AgentCompass

This is the paper introducing the project and its evaluation design. For the current implementation and component responsibilities, see the Architecture Overview.

Resources

• Paper and citation export on arXiv
• AgentCompass code

All AgentCompass research papers