Evaluation Modes and Artifacts

Choose none, reuse, or fresh from the state the verifier must access, and extract any artifacts needed later before the task Environment closes.

Three Modes

Mode	env passed to evaluate()	Lifecycle	Use it when
none	None	Close the task Environment after execution and artifact collection, then evaluate in the AgentCompass process	Exact match, structured answers, or controller-side evaluators
reuse	The current task EnvironmentSession	Close the task Environment only after execution, artifact collection, and evaluation finish	The verifier must inspect the same filesystem or service state left by execution
fresh	A new evaluation EnvironmentSession	Extract artifacts and close the task Environment, then open a separate evaluation Environment	Evaluation needs isolated tests, a clean image, or a different network policy

In all three modes, the runtime calls collect_artifacts() while the task Environment is still alive. This hook extracts submissions; it does not decide the score.

Set Defaults and Per-Task Overrides

Set a class-level default when every sample uses the same mode:

class ExampleBenchmark(BaseBenchmark):
    evaluation_environment_mode = "reuse"

Set the mode on individual TaskSpec values when samples require different verifier paths:

TaskSpec(
    task_id=task_id,
    question=question,
    category=category,
    ground_truth=ground_truth,
    evaluation_environment_mode="fresh",
)

The runtime Planner initially resolves the mode in this order:

RunRequest.environment.evaluation_environment_mode
  > TaskSpec.evaluation_environment_mode
  > Benchmark.resolve_evaluation_environment_mode(req)

This is the selection order while building the initial ExecutionPlan. Matching Recipes can then adjust the plan according to their own contracts. If a Recipe changes the evaluation Environment, its implementation and tests must state which fields it may override and which it must preserve. When debugging the effective mode, inspect resolved_execution_plans in run_info.json; it records the plan after Recipe adjustments.

none: Evaluate in the AgentCompass Process

Use none when evaluation depends only on TaskSpec, PreparedTask, and RunResult. Do not read the task workspace in this mode:

async def evaluate(
    self,
    task: TaskSpec,
    prepared: PreparedTask,
    result: RunResult,
    req: RunRequest,
    plan: ExecutionPlan,
    env: EnvironmentSession | None = None,
) -> RunResult:
    _ = task, prepared, req, plan
    if env is not None:
        raise RuntimeError("none evaluation must not receive an Environment")
    return score_answer(result)

If a controller-side evaluator calls another Model, configure that judge Model explicitly and record its version and inference parameters. Do not silently let the Model under test score its own result.

reuse: Inspect the Task Environment

reuse evaluates before the task Environment closes, so the verifier can see the workspace left by the agent:

class WorkspaceBenchmark(BaseBenchmark):
    evaluation_environment_mode = "reuse"

    async def evaluate(
        self,
        task: TaskSpec,
        prepared: PreparedTask,
        result: RunResult,
        req: RunRequest,
        plan: ExecutionPlan,
        env: EnvironmentSession | None = None,
    ) -> RunResult:
        _ = task, prepared, req, plan
        if env is None:
            raise RuntimeError("workspace verification requires reuse mode")
        verifier = await env.exec(
            ["python3", "/opt/verifier/check.py"],
            timeout=300,
        )
        return apply_verifier_result(result, verifier)

The verifier path and its dependencies must exist in the task Environment. Evaluation uses the resolved evaluation_network_policy; do not assume the execution-stage network permissions remain active.

For a complete production reuse implementation, inspect terminalbench2.py.

fresh: Extract First, Then Verify in Isolation

fresh does not automatically copy the task workspace into the evaluation Environment. First, use collect_artifacts() to convert the submission into RunResult.artifacts, which can cross the Environment boundary:

from dataclasses import replace


async def collect_artifacts(
    self,
    task: TaskSpec,
    prepared: PreparedTask,
    result: RunResult,
    env: EnvironmentSession,
    req: RunRequest,
    plan: ExecutionPlan,
) -> RunResult:
    _ = task, prepared, req
    benchmark_plan = self._require_plan(plan.benchmark_plan)
    patch = await env.read_text(benchmark_plan.submission_path)

    artifacts = dict(result.artifacts)
    files = dict(artifacts.get("file") or {})
    files["submission.patch"] = patch
    artifacts["file"] = files
    return replace(result, artifacts=artifacts)

After the task Environment closes, the runtime opens a new evaluation Environment and passes it to evaluate(). The evaluator writes the captured artifact into the new Environment, then runs the official verifier:

async def evaluate(
    self,
    task: TaskSpec,
    prepared: PreparedTask,
    result: RunResult,
    req: RunRequest,
    plan: ExecutionPlan,
    env: EnvironmentSession | None = None,
) -> RunResult:
    _ = task, prepared, req, plan
    if env is None:
        raise RuntimeError("isolated verification requires fresh mode")

    patch = str(result.artifacts["file"]["submission.patch"])
    await env.write_text("/tmp/submission.patch", patch)
    verifier = await env.exec(
        ["bash", "/opt/verifier/run.sh", "/tmp/submission.patch"],
        timeout=600,
    )
    return apply_verifier_result(result, verifier)

A production implementation should also limit artifact size, validate file types, handle empty submissions, and record the verifier return code, timeout state, and truncated output as result evidence. For a complete fresh + collect_artifacts() implementation, inspect deepswe.py.

Check Before Choosing a Mode

• Choose none when scoring depends only on an answer or in-memory objects; do not create an extra sandbox for simple scoring.
• Choose reuse when the verifier must see the original filesystem modified by the agent, and ensure the verifier does not corrupt results that must be retained.
• Choose fresh when the verifier must not trust dependencies or processes left by the agent, and transfer only the minimum required submission.
• Set explicit timeouts for commands in prepare_task(), collect_artifacts(), and the verifier, and make every step safe to repeat during retries.
• Use different statuses for evaluation failure and a valid zero score; see Results and Aggregation for the mapping.