Configure an Evaluation

Understand the parts of an evaluation request and find the authoritative documentation for each setting.

Use this page after completing the Quick Start to decide whether to adjust the model, benchmark, harness, environment, or run-level controls. See agentcompass run for complete CLI parameters and defaults, and agentcompass config for configuration files and precedence.

This page describes one agentcompass run request. To coordinate multiple models, benchmarks, harnesses, or environments with one scheduler, use agentcompass launch.

Evaluation Structure

Every run selects four core components:

agentcompass run <benchmark> <harness> <model> --env <environment>

Each part owns a different set of decisions:

Part	What it controls	Where to configure it
Model	Model ID, API endpoint, credentials, protocol, and inference parameters	Third positional argument and --model-* flags
Benchmark	Dataset, task selection, task preparation, scoring, and aggregate metrics	First positional argument and --benchmark-params
Harness	Agent loop, framework integration, tool behavior, and harness execution limits	Second positional argument and --harness-params
Environment	The environment provider—the backend that creates and manages task environments, such as Docker, Daytona, or Modal—plus workspace behavior, network policy, images, and resources	--env and --env-params

Keep parameters with the component that owns the behavior. For example, task filters belong in --benchmark-params, agent step limits belong in --harness-params, and container resources belong in --env-params.

Run-Level Controls

An evaluation request also contains controls that do not belong to any one of the four core components, including concurrency, timeouts, retries, result naming, historical run reuse, environment retention, and logging. See Run Controls for how these settings work together, and the agentcompass run parameter reference for complete signatures and defaults.

Next Actions

Adjust Run Controls

Configure concurrency, timeouts, retries, output reuse, debugging state, and logging.

Run Multiple Evaluations

Coordinate requests that use different models, benchmarks, harnesses, or environments with one global scheduler.

Manage Configuration

Inspect merged file values and component fields, and understand configuration-file loading order.

Inspect Results

Read per-task details and summaries, preserve reusable evidence, and analyze failures.

Troubleshoot a Run

Reduce a failure to one task, locate its lifecycle phase, and apply the correct fix.