Agentic Coding

Evaluate repository repair and scientific coding agents.

Agentic coding benchmarks combine repository state, issue text, generated patches or output files, and benchmark-specific scoring. AgentCompass keeps the benchmark, harness, model endpoint, and environment separate so the same task set can be tested with different agent implementations.

Recommended Stack

Need	Benchmark	Harness	Environment
SWE-bench Verified	swebench_verified	mini_swe_agent, codex, claude_code, or openhands	modal, daytona, docker, or cluster provider
SWE-bench Multilingual / Pro	swebench_multilingual, swebench_pro	coding harness	recipe-backed provider
Scientific coding	scicode	scicode_tool_use	local or remote

SWE-bench With Recipe Inference

export MODAL_TOKEN_ID="..."
export MODAL_TOKEN_SECRET="..."

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

For supported benchmark/provider combinations, the recipe infers the image and workspace root from task metadata. For SWE-bench Verified, the remote workspace is usually /testbed.

Harness Choice

Harness	Best Use when
mini_swe_agent	You want a compact SWE-style repair agent.
codex	You want to evaluate the OpenAI Codex CLI as the agent loop.
claude_code	You want to evaluate Claude Code in a prepared workspace.
openhands	You want to evaluate OpenHands on repository repair tasks.
scicode_tool_use	You are running SciCode step-by-step code generation.

Related Pages

• Benchmarks
• Harnesses
• Productivity
• SWE-bench with Modal