Skip to main content
This paper studies the reliability of software engineering agent evaluation and introduces a verified version of SWE-Bench Pro.

​
Overview

The study identifies two problems that distort evaluation: agents can access leaked solutions or hidden evaluation information, and some tasks contain misleading instructions or unsuitable tests. SWE-Bench Pro Verified combines controls against information leakage with targeted corrections to task descriptions and tests. The paper evaluates the effects of these changes and examines agent trajectories to understand changes in scores.

​
Relation to AgentCompass

The paper releases its code in AgentCompass. You can run the benchmark through the swebench_pro_verified integration; the benchmark guide documents its controls, supported environments, parameters, and evaluation outputs.

​
Resources

All AgentCompass research papers