Results and Reuse

AgentCompass persists task details incrementally, then aggregates them into request-level metrics and summaries. Reuse copies non-error task details—whether correct or incorrect—into a new run; it never resumes by writing into the source directory.

The main implementation is the RunStore class in src/agentcompass/runtime/results/store.py, with shaping in detail.py, aggregation in summary.py, and validated metric types under src/agentcompass/runtime/metrics/.

Result directory and persistence boundaries

RunStore reserves this directory shape:

<results_dir>/<optional output.run_name components>/<benchmark id>/<model id>/<run id>/

The user-controlled output.run_name, Model ID, and run ID portions are normalized before use; the Benchmark directory uses the selected registered Benchmark ID. An explicit run_id must not already exist. Without one, the store generates a timestamp-like ID and advances it when necessary to reserve a unique directory atomically.

The core artifacts are:

Artifact	Purpose	Writer
run_info.json	schema version, start/finish state, sanitized complete request, per-task/attempt resolved plans, reuse source	write_run_info, record_resolved_execution_plan, write_terminal_status
params.json	compact sanitized Benchmark, Model, and output identity used by result tooling	_write_params_record
details/<task>[_<category>].json	one countable completed task record	save_partial_result
details/_error_<task>[_<category>].json	task record containing a run/evaluation error	save_partial_result
retry_details/*.json	discarded runtime retry diagnostics; never part of the countable task set	save_retry_detail
summary.md	human-readable request metrics	save_results
.summary_counts.json	machine-readable count backing for summary rendering	save_results
analysis_summary.md and analysis_summary.json	optional Analyzer aggregation	save_analysis_summary

Progress and log paths are added to the returned paths mapping by the progress reporter and runtime log registry.

RunRequest.to_task_payload serializes the complete execution request, including metadata when populated; run_info.json stores its sanitized form. RunRequest.to_persistence_params contains only the Benchmark, Model, output, reuse, and metadata values needed for storage and reuse, and params.json compacts that data again to sanitized Benchmark, Model, and output fields. Inspect run_info.json.request for Harness, Environment, and execution settings.

Fields and values that look sensitive are replaced at persistence boundaries. Resolved plans use an explicit allowlisted view instead of dumping the complete ExecutionPlan. Before adding a persisted field, verify that it is necessary for reproducibility and cannot expose nested credentials.

Result layers

Do not treat every result-shaped object as a RunResult. Each layer has a distinct producer, responsibility, and persistence boundary:

RunResult is defined in result.py under src/agentcompass/runtime/models/.

Layer	Producer	Contract and destination
Raw execution result	BaseHarness.run_task or HarnessFreeBenchmark.run_task	A RunResult records status, final_answer, trajectory, artifacts, usage metrics, and run errors; it is not yet authoritative Benchmark evaluation
Artifact-enriched result	Benchmark.collect_artifacts	Adds downloaded or normalized artifacts before the task Environment is released; does not aggregate request metrics
Evaluated attempt	Benchmark.evaluate	Fills authoritative correct, score, evaluation errors, and Benchmark-specific fields; analysis_result and a redacted meta.resolved_execution_plan may be attached before persistence
Task detail	_run_attempts and build_detail_record	Stores attempts under keys such as "1" and "2", plus task identity, category, k, attempts_tried, first correct attempt, task correctness, and retry counts; build_detail_record restricts fields to the supported persisted shape before writing under details/
Request result	summarize_results and UnifiedEvaluationRuntime.finalize	Aggregates new and reused details into metadata, validated metrics, in-memory summary, paths, and applied_recipes; the readable form is written to summary.md
Orchestration result	Orchestrator	One RequestOutcome per request carries terminal status, error, and available paths; OrchestrationResult adds orchestration identity, status, timestamps, and an ordered outcome mapping

save_partial_result writes each task detail immediately and atomically. It recursively redacts secrets, uses a staging file and fsync, prevents a concurrent writer from replacing an existing normal result, and removes a stale _error_ file after a later success.

Request metrics returned by Benchmark.aggregate_metrics must validate as MetricResult: they include schema_version, at least one finite numeric value in metrics, counts.total, counts.evaluated, counts.error, and optional details and extra. None of these request-level objects is a task RunResult.

Result reuse

Select a source. RunRuntimeSpec contains reuse and reuse_run_id. A non-empty reuse_run_id automatically enables reuse.

Before any orchestration request reserves its output, Orchestrator._preflight calls UnifiedEvaluationRuntime.freeze_reuse_source. This prevents an implicit lookup from selecting a run directory created earlier by the same orchestration.

Source lookup follows these rules:

• with reuse_run_id, use that directory under the current run-name/Benchmark/Model root and fail if it does not exist;
• with reuse=True and no ID, select the newest directory under that root that contains run_info.json;
• if implicit reuse finds no source, disable reuse for the request and continue as a new run;
• if multiple launch requests share the same Benchmark and Model, implicit reuse is rejected as ambiguous; assign an explicit source ID per request.

“Under that root” is important: implicit source selection matches the result namespace, Benchmark ID, and Model ID. It does not compare the complete Benchmark, Harness, Environment, Model parameter, or evaluator configuration. Inspect run_info.json before reusing across configuration changes, or choose an explicit known source.

Materialize task details. RunStore.materialize_reused_details always targets a newly reserved run directory:

1. find a normal detail file for each currently selected task_id, preferring its category-qualified filename;
2. hard-link the source file into the new details/ directory, falling back to shutil.copy2 when linking is unavailable;
3. ignore missing tasks and all _error_ files;
4. load materialized normal details with load_partial_results;
5. schedule only task IDs without a normal detail in the new run.

Ignoring error files is deliberate: failed tasks run again. Missing tasks also run normally. Reused non-error details and newly produced details are ordered against the current selected task list and aggregated together during finalization.

run_info.json.reused_from records the source run ID and path when reuse succeeds. The source run remains unchanged.

Compatibility rules

Persisted detail files are an input to reuse, summary recomputation, analysis, and external tooling. When changing them:

• keep task_id stable across dataset loads;
• prefer additive attempt fields and compatible defaults;
• preserve normal versus _error_ filename meaning;
• keep retry diagnostics outside details/;
• test loading an existing run directory, not only writing a new one;
• keep MetricResult.counts aligned with the actual denominator;
• preserve atomic writes and recursive redaction.

Continue with Runtime Contracts and Planning for the in-memory types, or Execution, Scheduling, and Cleanup for terminal-state behavior.