Analyzer Integration

Add an Analyzer when you need post-execution diagnosis or statistics without changing the Benchmark’s authoritative evaluation result.

Analyzers run after a task attempt has produced a RunResult. They may inspect the normalized answer, trajectory, metrics, status, error, and artifacts, then return an AnalysisResult. They must not rerun the agent, change Benchmark correctness, or replace scoring logic that belongs in BaseBenchmark.evaluate().

Implement a Public Analyzer

The following example is scoped to the public example_exact_match Benchmark from the Harness-driven Benchmark tutorial and flags empty or unusually long final answers. It does not depend on any Environment provider implementation:

from agentcompass.runtime import (
    ANALYZERS,
    AnalysisResult,
    AnalyzerCategory,
    BaseAnalyzer,
    RunResult,
)


@ANALYZERS.register()
class ExampleAnswerLengthAnalyzer(BaseAnalyzer):
    id = "ExampleAnswerLengthAnalyzer"
    description = "Flag empty or unusually long example final answers."
    category = AnalyzerCategory.BEHAVIOR
    datasets = ["example_exact_match"]
    data_requirements = ["$.final_answer"]
    conf = {
        "only_incorrect": False,
        "max_characters": 1000,
    }
    distribution_fields = {
        "answer_characters": "numeric_stats",
    }

    async def analysis(self, task, prepared, result: RunResult, req, plan) -> AnalysisResult:
        _ = prepared, req, plan
        answer = str(result.final_answer or "").strip()
        answer_characters = len(answer)
        max_characters = max(1, int(self.conf.get("max_characters", 1000)))

        return AnalysisResult(
            task_id=task.task_id,
            is_badcase=not answer or answer_characters > max_characters,
            details={
                "answer_characters": answer_characters,
                "max_characters": max_characters,
                "empty_answer": not answer,
            },
        )

Keep rule-based analyzers deterministic. If an Analyzer calls a model, expose and validate its configuration, keep the evaluated result immutable, and report analysis failures as Analyzer output rather than changing the task status or score.

BaseAnalyzer Contract

Attribute or method	Current runtime meaning
id	Unique registry ID and default output family key
description	Human-readable text shown by agentcompass list analyzer
category	Classification metadata declared with AnalyzerCategory; current persistence and run-level aggregation do not group by this field
datasets	Eligible Benchmark IDs; an empty list matches every Benchmark
data_requirements	JSONPath expressions checked against RunResult.json; a missing requirement skips the Analyzer for that attempt
conf	Default Analyzer configuration; the matching <AnalyzerId> object under execution.analysis_params overlays it for the instance
distribution_fields	details fields to aggregate with numeric_stats or value_counts
base_analyzer	Optional family ID for a specialized implementation
priority	Within one eligible family, a strictly higher value wins
analysis()	Async method that returns an AnalysisResult for one attempt

The base class also provides matches_dataset(), check_requirements(), should_skip(), and is_threshold_badcase(). The shared should_skip() recognizes conf.only_incorrect; custom configuration remains the Analyzer’s responsibility.

Use exact registered Benchmark IDs in datasets. Use JSONPath only for data genuinely required before calling analysis(); optional fields should be handled inside the method so their absence does not silently remove the Analyzer output.

Register and Export

Place the implementation under src/agentcompass/analyzers/, commonly in analyzers/basic/ for deterministic rules. Decorate the concrete class with @ANALYZERS.register() and import its module through package __init__.py files so importing agentcompass.analyzers executes the decorator.

The registry rejects duplicate IDs. A top-level re-export is optional for Python name convenience, but transitive module import is required for registration. Confirm discovery with:

uv run agentcompass list analyzer

Analyzer-specific values are passed through execution.analysis_params; agentcompass config docs currently documents only Benchmark, Harness, and Environment config dataclasses, not Analyzer conf dictionaries. Document every supported Analyzer key and default explicitly.

Selection and Family Resolution

For each attempt, the runtime follows this sequence when analysis is enabled:

1. Apply the analyzers whitelist when present; otherwise apply exclude_analyzers.
2. Construct each remaining registered Analyzer and overlay its per-ID configuration object.
3. Check datasets, data_requirements, and only_incorrect eligibility.
4. Group eligible implementations by base_analyzer or, when absent, their own id.
5. Select the strictly highest-priority implementation in each family and call its analysis() method.

When eligible family members have equal priority, the earlier registered implementation remains selected. Use a specialized Analyzer only when it extends the same diagnostic contract and output family. Give it base_analyzer = "<generic-id>", a higher priority, and a narrower datasets list. Unrelated Analyzers should keep base_analyzer = None so they can run independently.

This priority behavior is specific to Analyzers. Recipe priority does not currently affect Recipe application order.

Return AnalysisResult

analysis() returns:

Field	Meaning
task_id	The current task ID; the persisted task record already carries this identity
is_badcase	True or False for a detector; None for statistics-only output or an indeterminate result
details	JSON-serializable diagnostic data retained per attempt
score	Optional numeric Analyzer score aggregated independently from the Benchmark score
error	Optional explanation when analysis could not be completed
extra	Optional additional JSON-serializable metadata

The runtime persists the selected family’s payload at:

attempts.<attempt-index>.analysis_result.<analyzer-family>

The payload always includes is_badcase and details for a returned result, and includes score, error, and extra only when populated. AnalysisResult.task_id is not duplicated inside that family payload. If analysis() raises, the runtime records an indeterminate family error and continues; it does not alter the Benchmark’s existing result.

Declare only supported aggregation methods:

distribution_fields = {
    "answer_characters": "numeric_stats",  # int or float
    "error_types": "value_counts",         # str or list[str]
}

When at least one persisted attempt contains aggregatable analysis output, run-level aggregation can write analysis_summary.json and analysis_summary.md. A detector that explicitly returns is_badcase but produces zero bad cases and zero analysis errors remains in per-attempt details but is omitted from the run-level summary. See Summary and Analysis Results for the persisted format.

Validate on One Public Task

After adding example_exact_match and example_answer from the Benchmark and Harness implementation tutorials, run their deterministic task with the example Analyzer. This path needs no model endpoint or credentials:

uv run agentcompass run example_exact_match example_answer unused-model \
  --env host_process \
  --benchmark-params '{"sample_ids":["capital-france"]}' \
  --harness-params '{"answer":"Paris"}' \
  --task-concurrency 1 \
  --max-retries 0 \
  --enable-analysis \
  --analysis-params '{
    "analyzers": ["ExampleAnswerLengthAnalyzer"],
    "ExampleAnswerLengthAnalyzer": {"max_characters": 4}
  }' \
  --results-dir results-dev \
  --run-name analyzer-smoke

Verify:

• exactly the requested family appears under the attempt’s analysis_result;
• details.answer_characters matches the saved final_answer;
• the configured limit overrides the class default without mutating later Analyzer instances, and the five-character answer is marked as a bad case by the four-character limit;
• a different Benchmark is skipped because of datasets;
• missing required data skips the Analyzer, while an exception inside analysis() produces a family error;
• analysis_summary.json and analysis_summary.md contain the declared numeric distribution when output is aggregatable.

Also rerun the Analyzer on a copied existing result with agentcompass analysis to verify compatibility with persisted RunResult reconstruction. Follow Testing and Validation for complete repository checks and pull request evidence.

Completion Checklist

• The new logic diagnoses an existing result and does not replace Benchmark scoring.
• The ID, description, category, dataset scope, required fields, and configuration defaults are explicit.
• Registration is reachable from agentcompass.analyzers, and agentcompass list analyzer shows the ID.
• Family and priority settings cannot suppress an unrelated Analyzer.
• AnalysisResult and details are JSON-serializable, and declared distributions use supported value types.
• Inline analysis, copied-result re-analysis, skips, errors, and aggregate output have been checked on public data.
• User-facing configuration and output fields are documented in both languages.