> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# SGI Deep Research

SGI Deep Research ([arxiv](https://arxiv.org/abs/2512.16969)) — "Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows"; AgentCompass uses the SGI Deep Research subset. It evaluates a deep-research agent on scientific-research questions that require multi-step web retrieval and evidence gathering to produce a detailed research answer, which an **LLM judge** then grades as correct or not against the ground truth.

Like BrowseComp, SGI Deep Research uses single-sided judging. The judge only compares the agent-under-test's answer against the ground truth, without comparing to any baseline. Both inference and judging run in the local process (`host_process`) — the harness first drives the model under test through the search loop to produce a final answer, then the judge model grades it.

## How it works

An SGI Deep Research run has two stages — inference and judging.

### Inference and judging

* **Inference.** The model under test acts as a research agent and, driven by the harness (default [`naive_search_agent`](https://agent-compass.mintlify.site/en/user_guide/modules/harnesses/naive_search_agent)), completes multi-turn tool loops such as search / visit per task, gathering evidence across sources to produce a detailed natural-language research answer.
* **Judging.** The judge model (`judge_model`) receives "question + ground truth + answer under test" and grades it with the built-in A/B/C protocol. The judge compares only the final answer, ignoring reasoning and formatting differences; equivalent expressions are accepted. The judge and the model under test are two separate endpoints; `judge_model` must be specified explicitly.

### The A/B/C verdict

The judge returns exactly one verdict, and only **A** counts as correct:

* **A — CORRECT:** the answer semantically matches the ground truth (equivalent expressions and formatting allowed).
* **B — INCORRECT:** any deviation from the ground truth.
* **C — INCOMPLETE / REPETITIVE / REFUSAL:** an invalid answer (cut off mid-sentence, looping repetition, or an explicit refusal).

## Parameters

Pass a JSON object via `--benchmark-params '{...}'`, or a `benchmark.params` block in the YAML given to `--config`; the CLI wins on shared keys. See the [Benchmark overview](https://agent-compass.mintlify.site/en/user_guide/modules/benchmarks/overview.md) for merge precedence.

### Parameter reference

<table>
  <colgroup>
    <col width="18%" />

    <col width="16%" />

    <col width="15%" />

    <col width="20%" />

    <col width="31%" />
  </colgroup>

  <thead>
    <tr><th>Parameter</th><th>Type</th><th>Default</th><th>Choices / values</th><th>Description</th></tr>
  </thead>

  <tbody>
    <tr><td>`judge_model`</td><td>dict</td><td>`null`</td><td>`id`, `base_url`, `api_key`, `api_protocol`, `params`</td><td>Judge model spec, **required** (see [Judge model spec](#judge-model-spec)). It decides grading, and is not the CLI `--model-*`.</td></tr>
    <tr><td>`category`</td><td>string / list</td><td>`"all"`</td><td>`"all"`, `life`, `earth`, `material`, `physics`, `mathematics`, `neuroscience`, `information`, `astronomy`, `chemistry`, `energy`</td><td>Filter tasks by category; `"all"` = no filter, a list takes the union. Task counts by category — life (87), earth (54), material (38), physics (32), mathematics (25), neuroscience (24), information (20), astronomy (17), chemistry (11), energy (10); 318 in total.</td></tr>
  </tbody>
</table>

Shared Benchmark fields such as `sample_ids` follow [Benchmark Parameters](https://agent-compass.mintlify.site/en/user_guide/modules/benchmarks/overview.md). Configure repeated attempts with `--k` and `--attempt-strategy`; see [Metrics and Aggregation](https://agent-compass.mintlify.site/en/user_guide/other_features/results/metrics_aggregation.md).

### Judge model spec

`judge_model` is passed as a dict with the fields `id`, `base_url`, `api_key`, `api_protocol`, and `params`, pointing to the judge model's own endpoint, with inference parameters under `params`.

We recommend **fixing a single judge** across all models under test. Grading directly decides the scores, so switching judges makes scores no longer comparable across models; likewise, the model under test should not serve as its own judge, as that is neither fair nor comparable. The judge need not be especially strong — the A/B/C criterion (semantic match) is relatively objective, so a mid-sized model suffices. AgentCompass recommends `Qwen3.6-35B-A3B`.

## Run examples

The three positional arguments to `agentcompass run` are Benchmark, Harness, and Model. These examples use `sgi_deep_research`, [`naive_search_agent`](https://agent-compass.mintlify.site/en/user_guide/modules/harnesses/naive_search_agent), and `$MODEL_NAME`, with [`host_process`](https://agent-compass.mintlify.site/en/user_guide/modules/environments/providers/host_process) as the Environment.

Set the following environment variables in your terminal before running the examples:

* Model under test: `MODEL_NAME`, `MODEL_BASE_URL`, and `MODEL_API_KEY`; see [Model connection details](https://agent-compass.mintlify.site/en/user_guide/modules/models/overview.md#configure-connection-details).
* Judge Model: `JUDGE_MODEL_NAME`, `JUDGE_MODEL_BASE_URL`, and `JUDGE_MODEL_API_KEY`. Use a fixed, independent judge configuration.
* Search tools: `SERPER_API_KEY` for `search` and `JINA_API_KEY` for `visit`.

See the [run command](https://agent-compass.mintlify.site/en/user_guide/using_agentcompass/cli/run.md) for configuration ownership and CLI overrides.

> [tab] Smoke test (single task end-to-end)
>
> Use `sample_ids` to evaluate a single task, verifying that the end-to-end inference and judging flow works; defaults for the rest.
>
> ```bash wrap theme={"system"}
> agentcompass run \
>   sgi_deep_research \
>   naive_search_agent \
>   "$MODEL_NAME" \
>   --env host_process \
>   --benchmark-params '{
>     "judge_model": {
>       "id": "'"$JUDGE_MODEL_NAME"'",
>       "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
>       "api_key": "'"$JUDGE_MODEL_API_KEY"'"
>     },
>     "sample_ids": ["SGI_DeepResearch_0000"]
>   }' \
>   --harness-params '{
>     "serper_api_key": "${SERPER_API_KEY}",
>     "jina_api_key": "${JINA_API_KEY}"
>   }' \
>   --model-base-url "$MODEL_BASE_URL" \
>   --model-api-key "$MODEL_API_KEY" \
>   --model-api-protocol openai-chat
> ```

> [tab] Custom parameters
>
> Evaluate only a subset of categories to focus analysis on a specific domain; also demonstrates lowering the iteration limit in `--harness-params`.
>
> ```bash wrap theme={"system"}
> agentcompass run \
>   sgi_deep_research \
>   naive_search_agent \
>   "$MODEL_NAME" \
>   --env host_process \
>   --benchmark-params '{
>     "judge_model": {
>       "id": "'"$JUDGE_MODEL_NAME"'",
>       "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
>       "api_key": "'"$JUDGE_MODEL_API_KEY"'"
>     },
>     "category": ["physics", "life"]
>   }' \
>   --harness-params '{
>     "max_iterations": 40,
>     "serper_api_key": "${SERPER_API_KEY}",
>     "jina_api_key": "${JINA_API_KEY}"
>   }' \
>   --model-base-url "$MODEL_BASE_URL" \
>   --model-api-key "$MODEL_API_KEY" \
>   --model-api-protocol openai-chat \
>   --task-concurrency 16
> ```

> [tab] AgentCompass recommended config
>
> Evaluate the full task set. `--benchmark-params` only needs the judge model `judge_model`; use `--task-concurrency` to raise cross-task concurrency.
>
> ```bash wrap theme={"system"}
> agentcompass run \
>   sgi_deep_research \
>   naive_search_agent \
>   "$MODEL_NAME" \
>   --env host_process \
>   --benchmark-params '{
>     "judge_model": {
>       "id": "'"$JUDGE_MODEL_NAME"'",
>       "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
>       "api_key": "'"$JUDGE_MODEL_API_KEY"'"
>     }
>   }' \
>   --harness-params '{
>     "serper_api_key": "${SERPER_API_KEY}",
>     "jina_api_key": "${JINA_API_KEY}"
>   }' \
>   --model-base-url "$MODEL_BASE_URL" \
>   --model-api-key "$MODEL_API_KEY" \
>   --model-api-protocol openai-chat \
>   --task-concurrency 16
> ```

## Evaluation Results

For shared result conventions, see [Run Directory](https://agent-compass.mintlify.site/en/user_guide/other_features/results/overview.md#directory-layout), [Aggregate Scores](https://agent-compass.mintlify.site/en/user_guide/other_features/results/summary_analysis.md), and [Task Files and Shared Fields](https://agent-compass.mintlify.site/en/user_guide/other_features/results/task_results.md).

### Scoring Metrics

SGI Deep Research's primary metric is binary `correct`: under the [A/B/C verdict](#the-abc-verdict) described above, A maps to `true` and B/C map to `false`. There is no partial credit.

With the default configuration, each task has one attempt and the overall score is accuracy over tasks with valid scores, ranging from 0 to 1; higher is better. For example, if all 100 tasks receive valid judgments and 63 are correct, `0.63` in the report means 63%. When you filter tasks with `category` or `sample_ids`, scores cover only the selected tasks.

See [Metrics and Aggregation](https://agent-compass.mintlify.site/en/user_guide/other_features/results/metrics_aggregation.md) for repeated attempts, category aggregation, and scoring failure rules.

### Task Results and Scoring Evidence

After an attempt is scored successfully, `scoring` under `meta.benchmark` contains:

| Field | Contents |
| - | - |
| `evaluation_type` | Fixed as `llm_judge`, indicating that a judge model evaluated the answer. |
| `correct` | The parsed boolean verdict, matching `metrics.correct`. |
| `model_answer` | The final answer sent to the judge. |
| `ground_truth` | The reference answer sent to the judge. |

This scoring record preserves the verdict and compared answers, but not the judge's raw response or reasoning.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.