benchmark

# Benchmark — SWE-bench Lite

You are a benchmark guide. Help the user run, configure, and evaluate SWE-bench Lite benchmarks against agtx agent workflows.

**Answer questions, surface the right commands, and walk through setup interactively.** All commands assume the user is in the `benchmark/` directory.

---

## Prerequisites

| Tool | Install |
|------|---------|
| **Docker** | Required for sandbox mode. macOS: Docker Desktop. Ubuntu: `apt install docker.io` |
| **agtx binary** | `cargo build --release` from repo root |
| **uv** | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |
| **tmux** | macOS: `brew install tmux`. Ubuntu: `apt install tmux` |
| **tokscale** (optional) | `npm install -g tokscale` — enables cost/token tracking in results |
| **Coding agent** | At least one: Claude Code, Gemini CLI, or Codex CLI |

## One-Time Setup

```bash
cd benchmark/swebench

# Initialize Python environment (once, or after pyproject.toml changes)
uv sync

# [Sandbox only] Build the tools image (tmux + Node.js + Claude Code)
python prebake_images.py --verbose

# [Sandbox only] Build the Linux agtx binary (Ubuntu 22.04 / glibc 2.35)
bash build_linux_binary.sh
```

The tools image populates the shared Docker volume `agtx-swebench-tools` on the first benchmark run. To force a refresh after updating Claude Code:

```bash
docker volume rm agtx-swebench-tools
python prebake_images.py --force --verbose
```

---

## Configuration

Config files live in `swebench/configs/`. Each is a standard agtx `ProjectConfig` TOML written to `.agtx/config.toml` in every cloned repo.

**Minimal (no workflow):**
```toml
default_agent = "claude"
workflow_plugin = "void"
```

**Standard agtx workflow:**
```toml
default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"
```

**Sandbox-optimised** (agent works directly in `/testbed`, no worktree):
```toml
default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"
skip_worktree = true
```

**Mixed agents** (different agent per phase):
```toml
default_agent = "claude"
workflow_plugin = "agtx"

[agents]
planning = "gemini"
running  = "claude"
review   = "codex"
```

**With `sandbox_init`** (install extra tooling inside the container before the agent starts):
```toml
default_agent = "claude"
workflow_plugin = "agtx"
skip_worktree = true

sandbox_init = [
    "curl -fsSL https://raw.githubusercontent.com/rtk-ai/rtk/refs/heads/master/install.sh | sh",
    "export PATH=$HOME/.local/bin:$PATH && rtk init -g",
]
```

`sandbox_init` commands run with `HOME=/home/bench` and `PATH` including `/home/bench/.local/bin`. Each config activates only the tools it explicitly installs — other configs are unaffected.

Available plugins: `void`, `agtx`, `agtx-terse`, `gsd`, `spec-kit`, `bmad`, `openspec`, `superpowers`, `agent-skills`

Pre-built configs for common combinations are in `swebench/configs/`.

---

## Running

> **Sandbox mode requires a Linux x86_64 binary.** Use `--agtx ../target/agtx-linux-x86_64` (not `../target/release/agtx`).
>
> **Always recommend sandbox mode.** SWE-bench repos require specific Python versions and C extensions that aren't available on the host — agents typically fail to run `pytest` outside the containers. In sandbox mode each task runs inside its official SWE-bench Docker image with the repo pre-installed in a working conda env (`testbed`).

**Single random task:**
```bash
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-void.toml \
  --instances 1 --verbose \
  --agtx ../target/release/agtx
```

**Specific instance IDs:**
```bash
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-void.toml \
  --instance-ids sympy__sympy-20590 django__django-11099 \
  --agtx ../target/release/agtx
```

**Sandbox mode (recommended):**
```bash
python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --instance-ids astropy__astropy-12907 \
  --sandbox --verbose \
  --agtx ../target/agtx-linux-x86_64
```

**Parallel tasks:**
```bash
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --concurrency 4 \
  --agtx ../target/release/agtx
```

**Full 300-task run:**
```bash
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --agtx ../target/release/agtx
```

**Resume an interrupted run** (pass the same `--output-dir`):
```bash
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --output-dir swebench_output/agtx_claude_20260427_120000 \
  --agtx ../target/release/agtx
```

**Hard mode** (prose only — no code blocks or stack traces in the problem statement):
```bash
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --hard \
  --agtx ../target/release/agtx
```

### All CLI Options

| Flag | Default | Description |
|------|---------|-------------|
| `--config PATH` | *(required)* | agtx config.toml for this run |
| `--instances N` | all 300 | Run first N tasks |
| `--instance-ids ID...` | — | Run specific instance IDs |
| `--concurrency N` | 1 | Parallel tasks |
| `--sandbox` | off | Run inside SWE-bench Docker images (recommended) |
| `--output-dir PATH` | `./swebench_output/{config-name}_{ts}/` | Output directory |
| `--workdir PATH` | `/tmp/swebench_repos` | Repo clone directory (non-sandbox only) |
| `--agtx PATH` | `./target/release/agtx` | agtx binary (must be Linux x86_64 for sandbox) |
| `--phase-timeout SECS` | 1200 | Per-phase max seconds (20 min) |
| `--model-name STRING` | `{config-stem}` | Label in predictions.jsonl |
| `--split STRING` | `test` | HuggingFace dataset split |
| `--verbose` / `-v` | off | Step-by-step progress to stderr |
| `--hard` | off | Strip code blocks and stack traces from problem statement |

---

## Observing a Running Benchmark

**Attach to a running cont