Tool Calling

Evaluate model protocol compatibility and tool-use style workflows.

Tool-calling evaluations are useful when the main risk is model I/O discipline: protocol support, structured outputs, tool-call formatting, retries, and whether the harness can turn task context into tool-aware model messages.

Current Runtime Path

Use harnesses that explicitly support tool-aware model protocols:

Workflow	Benchmark	Harness	Environment
Scientific coding with optional code execution	scicode	scicode_tool_use	host_process
Terminal tasks	terminal_bench_2, terminal_bench_2_1, terminal_bench_2_verified	terminus2	remote sandbox
Deep research with web/tool actions	gaia, deepsearchqa, frontierscience	naive_search_agent	host_process

Confirm available benchmark and harness ids with Supported Components before documenting or launching a run.

uv run agentcompass list benchmark

uv run agentcompass list harness

Tool-Use Run

agentcompass run \
  scicode \
  scicode_tool_use \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --model-params '{"temperature":0}'

What To Check

Signal	Where to inspect
Final answer format	details/*.json
Protocol mismatch	run log under logs/
Truncation or empty output	agentcompass analysis with output-quality analyzers
Aggregate pass rate	summary.md

Related Pages

• Model protocols
• OpenAI-compatible setup
• Post-analysis