Recipes

Recipes use task information from the benchmark and the selected environment to fill in runtime settings such as images, working directories, and resources. AgentCompass matches and applies built-in recipes automatically. A normal evaluation does not require recipe options or changes to a recipe implementation.

Configure a recipe manually only when a benchmark page names an alternative recipe, troubleshooting requires a restricted match set, or your team needs custom adaptation logic. Otherwise, keep the default automatic matching.

How Recipes Work

The same benchmark task may require different image and working-directory settings on Docker, Daytona, and Modal. Each time a task starts, AgentCompass uses recipes that match the benchmark and environment to prepare these settings. For example, a recipe can:

• select an image from the task id or an image address recorded in the task;
• set the working directory to a benchmark path such as /testbed, /workspace, or /root;
• convert CPU, memory, disk, or GPU requirements into parameters supported by the selected environment;
• add image and network settings when scoring requires a separate sandbox.

Recipes do not choose a harness or model for you, run tasks, or score results. To make the selected combination work, they may adjust installation or execution settings for the harness.

When to Configure a Recipe Manually

Scenario	What to set
Use the default built-in recipes	Do not pass recipe options
Use an alternative recipe named by a benchmark page	Pass --recipe <recipe-id>
Allow only specific recipes while reproducing or troubleshooting	Pass --recipe <recipe-id>; repeat as needed
Load a team-defined recipe	Pass --recipe-dir <package-dir>

--recipe does not force the named recipe to run. It only allows the listed ids to participate in matching. A recipe must still match the current benchmark, environment, and task information. When this option is omitted, AgentCompass matches from all available recipes automatically.

If several recipes match, AgentCompass applies all of them. To confirm what was applied, look for Recipe matched in the DEBUG run log.

Examples

The following command uses sample_ids to run one SWE-bench Verified instance. It omits --recipe; AgentCompass matches a built-in recipe from swebench_verified and modal automatically.

export MODEL_NAME="<model-id>"
export MODEL_BASE_URL="<base-url>"
export MODEL_API_KEY="<api-key>"
export MODAL_TOKEN_ID="..."
export MODAL_TOKEN_SECRET="..."

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  "$MODEL_NAME" \
  --env modal \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

The recipe selects an image from the instance information and sets the working directory to /testbed. Other built-in adaptations include:

Combination	What the recipe fills in
Terminal-Bench 2 + daytona	Reads the task’s Docker image and sets the default working directory to /root
ResearchClawBench + docker, daytona, or modal	Selects the image required by the benchmark; Daytona and Modal default to /workspace

Override Recipe-Provided Values

To use a custom image or snapshot, pass a field supported by the selected environment through --env-params:

Environment	Common override fields
Docker	image
Daytona	image or snapshot
Modal	image or named_image

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --env docker \
  --env-params '{"image":"<custom-image>"}'

When the selected environment supports a field, an explicit value takes precedence over the default inferred by the recipe, but it does not disable the recipe. The recipe may still fill in unspecified working-directory, resource, or network settings. See Network Policy for phase-specific network configuration.

Trusted External Recipes

This is an advanced workflow for team-defined adaptation logic. --recipe-dir loads an external recipe for the current run:

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --env <environment> \
  --recipe-dir ./company_recipes \
  --recipe company_swe_recipe

External recipes run as Python code in the AgentCompass process and are not isolated by the task sandbox. Load only trusted packages that you have reviewed.

External package requirements

• The directory must be a Python package containing __init__.py.
• The root module must export a non-empty RECIPE_CLASSES list or tuple.
• Every item must be a concrete subclass of AgentCompass’s BaseRecipe base class, with a unique id and a zero-argument constructor.
• Relative paths resolve from the current working directory.

Recipe directories determine which external implementations are loaded. Recipe ids determine which loaded recipes may participate in matching:

Interface	Recipe directories	Recipe ids
agentcompass run	Repeat --recipe-dir	Repeat --recipe
Single-request Python SDK	recipe_dirs=[...]	enabled_recipes=[...]
Single-request configuration	runtime.recipe_dirs	execution.enabled_recipes
launch orchestration file	Top-level runtime.recipe_dirs	defaults.execution.enabled_recipes or requests[].execution.enabled_recipes

agentcompass launch has no --recipe or --recipe-dir option; place the corresponding fields in the orchestration file. Explicit CLI or SDK lists replace the corresponding configuration-file lists rather than appending to them.

Duplicate recipe ids fail during loading. If multiple recipes modify the same image, working directory, or network setting, AgentCompass does not resolve the conflict automatically, so do not load implementations with overlapping responsibilities together.

Related Pages

• Benchmarks
• Environments
• agentcompass run
• Network Policy
• Environment Integration