> ## Documentation Index > Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Evaluation Modes and Artifacts Choose `none`, `reuse`, or `fresh` from the state the verifier must access, and extract any artifacts needed later before the task Environment closes. ## Three Modes | Mode | env passed to evaluate() | Lifecycle | Use it when | | ------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `none` | `None` | Close the task Environment after execution and artifact collection, then evaluate in the AgentCompass process | Exact match, structured answers, or controller-side evaluators | | `reuse` | The current task `EnvironmentSession` | Close the task Environment only after execution, artifact collection, and evaluation finish | The verifier must inspect the same filesystem or service state left by execution | | `fresh` | A new evaluation `EnvironmentSession` | Extract artifacts and close the task Environment, then open a separate evaluation Environment | Evaluation needs isolated tests, a clean image, or a different network policy | In all three modes, the runtime calls `collect_artifacts()` while the task Environment is still alive. This hook extracts submissions; it does not decide the score. ## Set Defaults and Per-Task Overrides Set a class-level default when every sample uses the same mode: ```python theme={"system"} class ExampleBenchmark(BaseBenchmark): evaluation_environment_mode = "reuse" ``` Set the mode on individual `TaskSpec` values when samples require different verifier paths: ```python theme={"system"} TaskSpec( task_id=task_id, question=question, category=category, ground_truth=ground_truth, evaluation_environment_mode="fresh", ) ``` The runtime Planner initially resolves the mode in this order: ```text theme={"system"} RunRequest.environment.evaluation_environment_mode > TaskSpec.evaluation_environment_mode > Benchmark.resolve_evaluation_environment_mode(req) ``` This is the selection order while building the initial `ExecutionPlan`. Matching Recipes can then adjust the plan according to their own contracts. If a Recipe changes the evaluation Environment, its implementation and tests must state which fields it may override and which it must preserve. When debugging the effective mode, inspect `resolved_execution_plans` in `run_info.json`; it records the plan after Recipe adjustments. ## `none`: Evaluate in the AgentCompass Process Use `none` when evaluation depends only on `TaskSpec`, `PreparedTask`, and `RunResult`. Do not read the task workspace in this mode: ```python theme={"system"} async def evaluate( self, task: TaskSpec, prepared: PreparedTask, result: RunResult, req: RunRequest, plan: ExecutionPlan, env: EnvironmentSession | None = None, ) -> RunResult: _ = task, prepared, req, plan if env is not None: raise RuntimeError("none evaluation must not receive an Environment") return score_answer(result) ``` If a controller-side evaluator calls another Model, configure that judge Model explicitly and record its version and inference parameters. Do not silently let the Model under test score its own result. ## `reuse`: Inspect the Task Environment `reuse` evaluates before the task Environment closes, so the verifier can see the workspace left by the agent: ```python theme={"system"} class WorkspaceBenchmark(BaseBenchmark): evaluation_environment_mode = "reuse" async def evaluate( self, task: TaskSpec, prepared: PreparedTask, result: RunResult, req: RunRequest, plan: ExecutionPlan, env: EnvironmentSession | None = None, ) -> RunResult: _ = task, prepared, req, plan if env is None: raise RuntimeError("workspace verification requires reuse mode") verifier = await env.exec( ["python3", "/opt/verifier/check.py"], timeout=300, ) return apply_verifier_result(result, verifier) ``` The verifier path and its dependencies must exist in the task Environment. Evaluation uses the resolved `evaluation_network_policy`; do not assume the execution-stage network permissions remain active. For a complete production `reuse` implementation, inspect [`terminalbench2.py`](https://github.com/open-compass/AgentCompass/blob/main/src/agentcompass/benchmarks/terminalbench2/terminalbench2.py). ## `fresh`: Extract First, Then Verify in Isolation `fresh` does not automatically copy the task workspace into the evaluation Environment. First, use `collect_artifacts()` to convert the submission into `RunResult.artifacts`, which can cross the Environment boundary: ```python theme={"system"} from dataclasses import replace async def collect_artifacts( self, task: TaskSpec, prepared: PreparedTask, result: RunResult, env: EnvironmentSession, req: RunRequest, plan: ExecutionPlan, ) -> RunResult: _ = task, prepared, req benchmark_plan = self._require_plan(plan.benchmark_plan) patch = await env.read_text(benchmark_plan.submission_path) artifacts = dict(result.artifacts) files = dict(artifacts.get("file") or {}) files["submission.patch"] = patch artifacts["file"] = files return replace(result, artifacts=artifacts) ``` After the task Environment closes, the runtime opens a new evaluation Environment and passes it to `evaluate()`. The evaluator writes the captured artifact into the new Environment, then runs the official verifier: ```python theme={"system"} async def evaluate( self, task: TaskSpec, prepared: PreparedTask, result: RunResult, req: RunRequest, plan: ExecutionPlan, env: EnvironmentSession | None = None, ) -> RunResult: _ = task, prepared, req, plan if env is None: raise RuntimeError("isolated verification requires fresh mode") patch = str(result.artifacts["file"]["submission.patch"]) await env.write_text("/tmp/submission.patch", patch) verifier = await env.exec( ["bash", "/opt/verifier/run.sh", "/tmp/submission.patch"], timeout=600, ) return apply_verifier_result(result, verifier) ``` A production implementation should also limit artifact size, validate file types, handle empty submissions, and record the verifier return code, timeout state, and truncated output as result evidence. For a complete `fresh + collect_artifacts()` implementation, inspect [`deepswe.py`](https://github.com/open-compass/AgentCompass/blob/main/src/agentcompass/benchmarks/deepswe.py). ## Check Before Choosing a Mode * Choose `none` when scoring depends only on an answer or in-memory objects; do not create an extra sandbox for simple scoring. * Choose `reuse` when the verifier must see the original filesystem modified by the agent, and ensure the verifier does not corrupt results that must be retained. * Choose `fresh` when the verifier must not trust dependencies or processes left by the agent, and transfer only the minimum required submission. * Set explicit timeouts for commands in `prepare_task()`, `collect_artifacts()`, and the verifier, and make every step safe to repeat during retries. * Use different statuses for evaluation failure and a valid zero score; see [Results and Aggregation](/en/developer_guide/extensions/benchmark/code_implementation/results_and_aggregation) for the mapping.