Skip to main content
Analyzers are the post-execution analysis layer of AgentCompass. After a task produces a RunResult, analyzers inspect trajectories, metrics, errors, latency, model output, and tool calls. Their output is attached to each attempt under analysis_result.<AnalyzerId> and aggregated into analysis_summary.json / analysis_summary.md.

Badcase detection

Flag failures such as runtime exceptions, truncation, JSON errors, repetition, latency spikes, or terminal misuse.

Statistics

Compute distributions such as step counts, tool-call counts, duration, token length, and value-count summaries.

Qualitative analysis

Use LLM-backed analyzers to annotate trajectory phases, summarize behavior, and render segment-graph reports.

Component registry

agentcompass list analyzer prints registered analyzers, and agentcompass list dump exports them to agentcompass_components.md.

​
Run with Evaluation

agentcompass run \
  terminal_bench_2 \
  terminus2 \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["git-leak-recovery"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --enable-analysis \
  --analysis-params '{"analyzers":["ExceptionAnalyzer","TruncationAnalyzer"]}'

​
Re-run on Existing Results

agentcompass analysis \
  --input results/terminal_bench_2/your-model/20260703_120000 \
  --analysis-params '{
    "analyzers": ["ExceptionAnalyzer", "QualitativeAnalyzer"],
    "QualitativeAnalyzer": {"render_mode": "file"}
  }' \
  --task_concurrency 8
By default, analysis copies the input run into a new timestamped sibling. Use --output to choose the copy destination, or --override to mutate the input run in place.

​
Selection Rules

FieldMeaning
analyzersWhitelist. If set, only these analyzer ids are considered.
exclude_analyzersBlacklist. Excluded analyzers are skipped even if otherwise compatible.
<AnalyzerId>Per-analyzer config, such as thresholds or qualitative model settings.

​
Supported Families

Current agentcompass list analyzer output reports 25 registered analyzers:
  • Basic statistics: BasicMetricAnalyzer, TrajectoryTimeCostAnalyzer, CompletionLengthAnalyzer.
  • Error detection: ExceptionAnalyzer, TerminalBench2ExceptionAnalyzer, TruncationAnalyzer, JSONErrorAnalyzer, EmptyContentAnalyzer.
  • Efficiency: LLMInferLatencyAnalyzer, ToolExecutionLatencyAnalyzer.
  • Behavior patterns: repetition, keyword, mixed-language, network-operation, command-running, and reward-hacking analyzers.
  • Qualitative: QualitativeAnalyzer, MultiQualitativeAnalyzer.

​
Output Shape

Per-task details keep every analyzer result under analysis_result. Aggregated summaries group analyzers by category and render badcase ratios, average scores, numeric stats, and value counts declared by each analyzer.