skill-eval-toolkit

# Skill Eval Toolkit

An eval-driven workflow for testing, benchmarking, and optimizing existing skills. For how to *write* a skill from scratch, see the **write-a-skill** skill first — this toolkit picks up where that guide leaves off.

## When to Use This Toolkit

- You have a drafted skill and want to verify it actually works
- You're iterating on a skill and need to measure whether changes improve it
- You want to compare "with skill" vs "without skill" (or old vs new) performance
- The description needs trigger optimization
- You want rigorous blind A/B comparison between two skill versions

## Overview

The eval loop:

1. Create test prompts and run them (with-skill AND baseline, in parallel)
2. While runs execute, draft quantitative assertions
3. Grade outputs, aggregate into benchmarks, launch interactive viewer
4. User reviews outputs and leaves feedback
5. Improve skill based on feedback, repeat

Your job is to figure out where the user is in this process and help them progress.

## Communicating with the user

Pay attention to context cues to calibrate your language. "Evaluation" and "benchmark" are borderline but OK. For "JSON" and "assertion", check for cues that the user knows what those mean before using them unexplained. Briefly explain terms if in doubt.

---

## Test Cases

Come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user for confirmation, then run them.

Save test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions while runs are in progress.

```json
{
  "skill_name": "example-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "User's task prompt",
      "expected_output": "Description of expected result",
      "files": []
    }
  ]
}
```

See `reference/schemas.md` for the full schema (including the `assertions` field, which you'll add later).

## Running and evaluating test cases

This section is one continuous sequence — don't stop partway through.

Put results in `<skill-name>-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Create directories as you go.

### Step 1: Spawn all runs (with-skill AND baseline) in the same turn

For each test case, spawn two subagents in the same turn — one with the skill, one without. Launch everything at once so it all finishes around the same time.

**With-skill run:**

```
Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the user cares about>
```

**Baseline run** (same prompt, but the baseline depends on context):
- **New skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.
- **Improving existing skill**: the old version. Before editing, snapshot the skill (`cp -r <skill-path> <workspace>/skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.

Write an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing.

```json
{
  "eval_id": 0,
  "eval_name": "descriptive-name-here",
  "prompt": "The user's task prompt",
  "assertions": []
}
```

### Step 2: While runs are in progress, draft assertions

Don't just wait — draft quantitative assertions for each test case and explain them to the user. Good assertions are objectively verifiable and have descriptive names that read clearly in the benchmark viewer.

Sanity-check coverage: outcome / process / style / efficiency. An empty axis is usually a blind spot, not a deliberate skip.

Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.

Update the `eval_metadata.json` files and `evals/evals.json` with assertions once drafted.

### Step 3: As runs complete, capture timing data

When each subagent completes, you receive a notification containing `total_tokens` and `duration_ms`. Save immediately to `timing.json` in the run directory:

```json
{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}
```

This is the only opportunity to capture this data — process each notification as it arrives.

### Step 4: Grade, aggregate, and launch the viewer

Once all runs are done:

1. **Grade each run** — spawn a grader subagent that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The `expectations` array must use fields `text`, `passed`, and `evidence` — the viewer depends on these exact field names. For programmatically checkable assertions, write and run a script.

2. **Aggregate into benchmark** — run from the skill-eval-toolkit directory:
   ```bash
   python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
   ```
   This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean +/- stddev and the delta. See `reference/schemas.md` for the exact schema.

3. **Analyst pass** — spawn a benchmark-analyzer subagent that reads `agents/benchmark-analyzer.md` and surfaces patterns: non-discriminating assertions, high-variance evals, time/token tradeoffs. The benchmark-analyzer analyzes benchmark.json only — do not pass skill contents or transcripts.

4. **Launch the viewer**:
   ```bash
   nohup python <skill-eval-toolkit-path>/eval-viewer/generate_review.py \
     <workspace>/iteration-N \
     --skill-name "my-skill" \
     --benchmark <workspace>/iteration-N/benchmark.json \
     > /dev/null 2>&1 &
   VIEWER_PID=$!
   ```
   For iteration 2+, pass `--previous-workspace <workspace>/iteration-<N-1>`.

   Use `gene