Skip to main content
Model configuration in AgentCompass is run-local. A model spec tells the selected harness which model id to call, where the endpoint is, which protocol to use, and which generation parameters to pass. AgentCompass does not require a global model registry for normal runs.

​
Model Spec Fields

FieldMeaning
idModel name sent to the provider and used in result paths.
base_urlAPI endpoint base URL.
api_keyAPI key or token for the endpoint.
api_protocolProtocol adapter, such as openai-chat, openai-responses, or anthropic.
paramsSampling and provider-specific request options such as temperature or max_tokens.
The CLI maps these fields from --model-base-url, --model-api-key, --model-api-protocol, and --model-params.

​
Basic Usage

export MODEL_BASE_URL="https://your-endpoint/v1"
export MODEL_API_KEY="sk-..."

agentcompass run \
  screenspot \
  qwen3vl_gui \
  qwen3-vl \
  --env <env-provider> \
  --benchmark-params '{"category":"desktop"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --model-params '{"temperature":0}'
Use shell variables for secrets. Shell variable names must use underscores, for example MODEL_BASE_URL, not model-base-url.

​
Protocol Selection

ProtocolTypical use
openai-chatOpenAI-compatible chat completions style endpoints.
openai-responsesOpenAI Responses API style endpoints.
anthropicAnthropic Messages API style endpoints.
JSON listOrdered fallback list when a harness can choose among supported protocols.
If api_protocol is empty, the harness may choose its own default. For public docs and reproducible commands, prefer passing the protocol explicitly.

​
Python Usage

from agentcompass import run_evaluation

run_evaluation(
    benchmark="screenspot",
    harness="qwen3vl_gui",
    model="qwen3-vl",
    environment="host_process",
    benchmark_params={"category": "desktop"},
    model_base_url="https://your-endpoint/v1",
    model_api_key="sk-...",
    model_api_protocol="openai-chat",
    model_params={"temperature": 0},
)

​
Judge Models

Some benchmarks and analyzers need a second model for judging or qualitative analysis. They use the same conceptual fields: model id, base URL, API key, protocol, and params. Examples:
  • benchmark judge models under benchmark_params.judge_model;
  • qualitative analyzer models under analysis_params.QualitativeAnalyzer;
  • comparison analyzers under analysis_params.MultiQualitativeAnalyzer.

​
Common Failures

SymptomLikely cause
Provider not detectedThe model id needs a provider prefix for the underlying client, or the harness protocol is wrong.
401 or authentication errors--model-api-key is missing or points to the wrong endpoint.
404 or model not foundThe model id does not exist on --model-base-url.
Unsupported protocol--model-api-protocol is not supported by the selected harness.