Skip to main content
AgentCompass is organized around a direct evaluation runtime. The architecture follows one main rule: components depend on runtime contracts, not on each other’s concrete implementations. The runtime treats benchmark, harness, model, and environment as independent specs on RunRequest. Benchmarks can prepare and score tasks without knowing which sandbox provider is used. Harnesses can run an agent loop without owning benchmark scoring. Environments can provide execution and file primitives without knowing whether the task came from SWE-bench, Terminal-Bench, GUI grounding, or a research benchmark. 
CLI / Python SDK
  -> RunRequest
  -> UnifiedEvaluationRuntime
  -> Benchmark / Harness / Environment / Recipe
  -> FileManager / ResultProcessor / ProgressReporter

​
Components

  • agentcompass.cli: command-line entrypoint.
  • agentcompass.launcher: Python SDK entrypoints and request construction.
  • agentcompass.runtime.models: request, plan, task, result, and progress dataclasses.
  • agentcompass.runtime.runner: orchestration loop.
  • agentcompass.runtime.registry: benchmark, harness, environment, recipe, and analyzer registries.
  • agentcompass.benchmarks: dataset loading and scoring.
  • agentcompass.harnesses: agent/model execution.
  • agentcompass.environments: command and file primitives.
  • agentcompass.recipes: provider-aware plan rewrites.
  • agentcompass.foundation: persistence, locking, limiters, metrics, and utility code.

​
Composition Model

RunRequest
  benchmark: BenchmarkSpec
  harness: HarnessSpec
  environment: EnvironmentSpec
  model: ModelSpec
This shape is the main extension point. Adding a benchmark should not require a new harness. Adding an environment provider should not require benchmark rewrites. Adding a model endpoint should not change task loading or scoring. Compatibility is checked at the boundary through BaseHarness.supports(...) and recipe matching.

​
Execution Flow

  1. CLI or SDK builds a RunRequest.
  2. Runtime loads component registries and config defaults.
  3. The benchmark loads and filters tasks.
  4. The planner builds benchmark, harness, and environment plans.
  5. Recipes may rewrite compatible plans.
  6. The environment session starts.
  7. The benchmark prepares task material.
  8. The harness runs the agent.
  9. The benchmark evaluates the output.
  10. Details, progress, logs, and summary artifacts are persisted.

​
Design Boundaries

  • Benchmarks own task data, preparation, scoring, and aggregation.
  • Harnesses own agent execution and return normalized RunResult objects.
  • Environments expose EnvironmentSession primitives such as exec, upload/download, text I/O, and optional endpoints.
  • Models are carried as ModelSpec; harnesses decide how to call the endpoint or forward credentials to an agent CLI.
  • Recipes adapt benchmark/provider combinations before sandbox startup, especially images, workspace roots, snapshots, and resource hints.
  • Results and analyzers are downstream of execution, so post-analysis can be rerun without rerunning the benchmark.