> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Analyzer Integration

Add an Analyzer when you need post-execution diagnosis or statistics without changing the Benchmark's authoritative evaluation result.

Analyzers run after a task attempt has produced a `RunResult`. They may inspect the normalized answer, trajectory, metrics, status, error, and artifacts, then return an `AnalysisResult`. They must not rerun the agent, change Benchmark correctness, or replace scoring logic that belongs in `BaseBenchmark.evaluate()`.

## Implement a Public Analyzer

The following example is scoped to the public `example_exact_match` Benchmark from the [Harness-driven Benchmark tutorial](/en/developer_guide/extensions/benchmark/code_implementation/harness_driven) and flags empty or unusually long final answers. It does not depend on any Environment provider implementation:

```python theme={"system"}
from agentcompass.runtime import (
    ANALYZERS,
    AnalysisResult,
    AnalyzerCategory,
    BaseAnalyzer,
    RunResult,
)


@ANALYZERS.register()
class ExampleAnswerLengthAnalyzer(BaseAnalyzer):
    id = "ExampleAnswerLengthAnalyzer"
    description = "Flag empty or unusually long example final answers."
    category = AnalyzerCategory.BEHAVIOR
    datasets = ["example_exact_match"]
    data_requirements = ["$.final_answer"]
    conf = {
        "only_incorrect": False,
        "max_characters": 1000,
    }
    distribution_fields = {
        "answer_characters": "numeric_stats",
    }

    async def analysis(self, task, prepared, result: RunResult, req, plan) -> AnalysisResult:
        _ = prepared, req, plan
        answer = str(result.final_answer or "").strip()
        answer_characters = len(answer)
        max_characters = max(1, int(self.conf.get("max_characters", 1000)))

        return AnalysisResult(
            task_id=task.task_id,
            is_badcase=not answer or answer_characters > max_characters,
            details={
                "answer_characters": answer_characters,
                "max_characters": max_characters,
                "empty_answer": not answer,
            },
        )
```

Keep rule-based analyzers deterministic. If an Analyzer calls a model, expose and validate its configuration, keep the evaluated result immutable, and report analysis failures as Analyzer output rather than changing the task status or score.

## BaseAnalyzer Contract

| Attribute or method   | Current runtime meaning                                                                                                            |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `id`                  | Unique registry ID and default output family key                                                                                   |
| `description`         | Human-readable text shown by `agentcompass list analyzer`                                                                          |
| `category`            | Classification metadata declared with `AnalyzerCategory`; current persistence and run-level aggregation do not group by this field |
| `datasets`            | Eligible Benchmark IDs; an empty list matches every Benchmark                                                                      |
| `data_requirements`   | JSONPath expressions checked against `RunResult.json`; a missing requirement skips the Analyzer for that attempt                   |
| `conf`                | Default Analyzer configuration; the matching `<AnalyzerId>` object under `execution.analysis_params` overlays it for the instance  |
| `distribution_fields` | `details` fields to aggregate with `numeric_stats` or `value_counts`                                                               |
| `base_analyzer`       | Optional family ID for a specialized implementation                                                                                |
| `priority`            | Within one eligible family, a strictly higher value wins                                                                           |
| `analysis()`          | Async method that returns an `AnalysisResult` for one attempt                                                                      |

The base class also provides `matches_dataset()`, `check_requirements()`, `should_skip()`, and `is_threshold_badcase()`. The shared `should_skip()` recognizes `conf.only_incorrect`; custom configuration remains the Analyzer's responsibility.

Use exact registered Benchmark IDs in `datasets`. Use JSONPath only for data genuinely required before calling `analysis()`; optional fields should be handled inside the method so their absence does not silently remove the Analyzer output.

## Register and Export

Place the implementation under `src/agentcompass/analyzers/`, commonly in `analyzers/basic/` for deterministic rules. Decorate the concrete class with `@ANALYZERS.register()` and import its module through package `__init__.py` files so importing `agentcompass.analyzers` executes the decorator.

The registry rejects duplicate IDs. A top-level re-export is optional for Python name convenience, but transitive module import is required for registration. Confirm discovery with:

```bash theme={"system"}
uv run agentcompass list analyzer
```

Analyzer-specific values are passed through `execution.analysis_params`; `agentcompass config docs` currently documents only Benchmark, Harness, and Environment config dataclasses, not Analyzer `conf` dictionaries. Document every supported Analyzer key and default explicitly.

## Selection and Family Resolution

For each attempt, the runtime follows this sequence when analysis is enabled:

1. Apply the `analyzers` whitelist when present; otherwise apply `exclude_analyzers`.
2. Construct each remaining registered Analyzer and overlay its per-ID configuration object.
3. Check `datasets`, `data_requirements`, and `only_incorrect` eligibility.
4. Group eligible implementations by `base_analyzer` or, when absent, their own `id`.
5. Select the strictly highest-`priority` implementation in each family and call its `analysis()` method.

When eligible family members have equal priority, the earlier registered implementation remains selected. Use a specialized Analyzer only when it extends the same diagnostic contract and output family. Give it `base_analyzer = "<generic-id>"`, a higher `priority`, and a narrower `datasets` list. Unrelated Analyzers should keep `base_analyzer = None` so they can run independently.

This priority behavior is specific to Analyzers. Recipe priority does not currently affect Recipe application order.

## Return AnalysisResult

`analysis()` returns:

| Field        | Meaning                                                                                        |
| ------------ | ---------------------------------------------------------------------------------------------- |
| `task_id`    | The current task ID; the persisted task record already carries this identity                   |
| `is_badcase` | `True` or `False` for a detector; `None` for statistics-only output or an indeterminate result |
| `details`    | JSON-serializable diagnostic data retained per attempt                                         |
| `score`      | Optional numeric Analyzer score aggregated independently from the Benchmark score              |
| `error`      | Optional explanation when analysis could not be completed                                      |
| `extra`      | Optional additional JSON-serializable metadata                                                 |

The runtime persists the selected family's payload at:

```text theme={"system"}
attempts.<attempt-index>.analysis_result.<analyzer-family>
```

The payload always includes `is_badcase` and `details` for a returned result, and includes `score`, `error`, and `extra` only when populated. `AnalysisResult.task_id` is not duplicated inside that family payload. If `analysis()` raises, the runtime records an indeterminate family error and continues; it does not alter the Benchmark's existing result.

Declare only supported aggregation methods:

```python theme={"system"}
distribution_fields = {
    "answer_characters": "numeric_stats",  # int or float
    "error_types": "value_counts",         # str or list[str]
}
```

When at least one persisted attempt contains aggregatable analysis output, run-level aggregation can write `analysis_summary.json` and `analysis_summary.md`. A detector that explicitly returns `is_badcase` but produces zero bad cases and zero analysis errors remains in per-attempt details but is omitted from the run-level summary. See [Summary and Analysis Results](/en/user_guide/other_features/results/summary_analysis) for the persisted format.

## Validate on One Public Task

After adding `example_exact_match` and `example_answer` from the Benchmark and [Harness implementation tutorials](/en/developer_guide/extensions/harness/code_implementation), run their deterministic task with the example Analyzer. This path needs no model endpoint or credentials:

```bash theme={"system"}
uv run agentcompass run example_exact_match example_answer unused-model \
  --env host_process \
  --benchmark-params '{"sample_ids":["capital-france"]}' \
  --harness-params '{"answer":"Paris"}' \
  --task-concurrency 1 \
  --max-retries 0 \
  --enable-analysis \
  --analysis-params '{
    "analyzers": ["ExampleAnswerLengthAnalyzer"],
    "ExampleAnswerLengthAnalyzer": {"max_characters": 4}
  }' \
  --results-dir results-dev \
  --run-name analyzer-smoke
```

Verify:

* exactly the requested family appears under the attempt's `analysis_result`;
* `details.answer_characters` matches the saved `final_answer`;
* the configured limit overrides the class default without mutating later Analyzer instances, and the five-character answer is marked as a bad case by the four-character limit;
* a different Benchmark is skipped because of `datasets`;
* missing required data skips the Analyzer, while an exception inside `analysis()` produces a family error;
* `analysis_summary.json` and `analysis_summary.md` contain the declared numeric distribution when output is aggregatable.

Also rerun the Analyzer on a copied existing result with [`agentcompass analysis`](/en/user_guide/using_agentcompass/cli/analysis#re-run-on-existing-results) to verify compatibility with persisted `RunResult` reconstruction. Follow [Testing and Validation](/en/developer_guide/contributing/testing) for complete repository checks and pull request evidence.

## Completion Checklist

* The new logic diagnoses an existing result and does not replace Benchmark scoring.
* The ID, description, category, dataset scope, required fields, and configuration defaults are explicit.
* Registration is reachable from `agentcompass.analyzers`, and `agentcompass list analyzer` shows the ID.
* Family and priority settings cannot suppress an unrelated Analyzer.
* `AnalysisResult` and `details` are JSON-serializable, and declared distributions use supported value types.
* Inline analysis, copied-result re-analysis, skips, errors, and aggregate output have been checked on public data.
* User-facing configuration and output fields are documented in both languages.