graphjin-eval

# GraphJin Eval

Use this skill when a user wants to create or run a GraphJin agent benchmark,
add a real business question to the suite, establish a baseline, compare a
candidate, or understand an evaluation failure.

## Rules

- Always use `graphjin eval` commands with `--json` for machine-readable state.
- Never edit `eval/suite.yml`, hidden oracle definitions, tolerances, reward
  weights, reports, or baseline files by hand.
- Never hand-edit `website/data/benchmarks/<benchmark>.yaml` or
  `website/content/benchmarks/<benchmark>/runs/`. `graphjin eval publish` is the
  only supported writer; it writes one row and one run page and never runs Git.
- Treat `eval publish --label` as presentation only. Supersession uses the
  normalized provider and model identity, not the display label.
- Use `graphjin eval rm <task-id>` as the supported task-removal path; never
  delete a task from the suite file manually.
- Never invent an oracle, field, threshold, or business interpretation.
- Treat exit code 2 as a broken suite, not a model regression.
- Treat the suite generator version as part of the scoring contract. Bump
  `eval.GeneratorVersion` whenever generated task semantics change, including
  method-rule dialect support, then regenerate every committed/frozen suite.
  A binary must refuse suites from any other generator version.
- Treat exit code 3 as an environment problem, not a model regression.
- Treat exit code 130 as an interrupted checkpoint. Resume it; do not score it.
- Provider-backed commands can incur cost. Explain the expected call count and
  use `--yes` only after the user approves provider traffic.
- Read both usage views in the report: finalized tokens measure agent
  efficiency, while provider tokens include failed attempts and retries. On a
  compatible baseline, report the total-token and tokens-per-episode direction
  and percentage; treat cross-model or differently shaped comparisons as
  advisory.
- Check `provider_usage.complete`. If false, `unknown_attempts` counts provider
  calls that returned no usage and all recorded token totals are lower bounds.
  Never compare token percentages across accounting versions, providers,
  models, configured `max_steps`, or incomplete provider usage.
- Before calling two runs a same-build comparison, require matching
  `provenance.binary_fingerprint`. It is the SHA-256 of the exact CLI
  executable and catches runtime changes that do not alter prompt hashes.
- Full prompts, answers, rows, and executed queries stay in local episode files.
  Share reports, not episode files, unless the user explicitly asks for the
  private trajectory.
- Failed/interrupted provider attempts stay under `.graphjin-evals/attempts/`.
  They are private, and no persisted file may contain a credential.
- Use `GOOGLE_API_KEY` as the canonical Google credential name.
- Publishing does not refuse a low score. Never rerun a completed benchmark to
  make the public board look better; publish the observed result with its
  `accepted` state.
- Do not publish a report marked `scoring_suspect` until the scorer/runtime
  mismatch is understood. `--allow-suspect-scoring` is an explicit audited
  override, not a routine publishing flag.
- Publish with the exact binary that ran the benchmark. A missing
  `graphjin_commit` or mismatched `binary_fingerprint` is a broken provenance
  chain and must be rerun, not waived.
- Never publish an off-suite run unless the user explicitly asks for it. When
  asked, use `--allow-off-suite` and verify it appears as unranked with the
  mismatch reason.

## Workflow

1. Inspect current state:

   ```sh
   graphjin eval --json
   ```

2. If no suite exists, create the deterministic 24-task suite:

   ```sh
   graphjin eval create --json
   ```

   Add `--demo` for the bundled demo or `--remote` for the server configured by
   `graphjin cli setup`.

3. Add an important business question through the model-assisted path:

   ```sh
   graphjin eval add "Which customers are at churn risk?" --json
   ```

   Report the CLI's plain-language interpretation and executed oracle result.
   If it asks for clarification, pass the question back to the user. Do not
   resolve ambiguity yourself.

4. Run the suite after approval:

   ```sh
   graphjin eval run --yes --json
   ```

   The first safe, valid run is promoted automatically at its observed recall.
   Recall below 0.90 is a quality warning, not a gate. Existing baselines compare
   only intersecting task IDs; new tasks remain advisory until a deliberate
   promotion.

   The command automatically resumes the newest strictly compatible incomplete
   run. Use `--resume <run-id>` to select one checkpoint. Use `--restart` only
   when the user intentionally wants fresh traffic; never combine the flags.
   The preview includes reused episodes and one possible transient retry for
   every pending initial/confirmation slot.

5. Remove a bad-but-executable task only through the validated CLI path:

   ```sh
   graphjin eval rm <task-id> --yes --json
   ```

6. Deliberately replace the baseline only when the user requests it and the run
   has no confirmed regression or safety failure:

   ```sh
   graphjin eval baseline --yes --json
   ```

7. Run the extended benchmark when the user wants frontier distribution
   coverage:

   ```sh
   graphjin eval bench --scale 100 --seed 23 --yes --json
   ```

8. Collect several attempts per task, for a training corpus rather than a
   verdict:

   ```sh
   graphjin eval sample --repeats 8 --temperature 0.8 --split <split> --side train --yes --json
   ```

   Sampling reaches no verdict and promotes nothing. Without a temperature the
   attempts come back identical.

9. Export a completed run as trajectories:

   ```sh
   graphjin eval export <run-id> --stage executor --out run.jsonl
   ```

   It refuses eval-side episodes unless `--allow-eval-side` is given.

10. Re-grade a completed run with no provider traffic, after a scoring change
    or to compare pro