polly-e2e-dev

# polly orchestrator: end-to-end CUJ dev & testing

`polly` (`examples/polly/`) is a multi-agent **coding orchestrator**: a
`claude-sdk` "brain" that writes no code itself and delegates everything to three
coding sub-agents — `claude_code` (claude-native), `codex` (codex-native), and
`pi` (headless, multi-model). Its critical user journeys are orchestration
behaviors, not single-turn answers:

- **roster preflight** — first turn runs `command -v claude codex pi`, routes
  only to workers whose CLI resolved.
- **investigate** — read-only work fanned to `explore`/`search` sub-agents;
  synthesize from their reports.
- **fanout** — independent tasks, each in its own git worktree + sub-agent, each
  opening its own PR.
- **cross-review** — an implementer's diff is verified by a **different-vendor**
  sub-agent (diff + contract only); blocking issues become fix-tasks.
- **plan gate / inbox** — pull the human in at the plan gate; supervise via the
  inbox + autowake, never busy-poll.
- **guardrails** (`omnigent.inner.nessie.policies`) — `blast_radius` (deny
  force-push / `rm -rf /`), `spawn_bounds` (cap dispatches per turn),
  `headless_subagent_purpose_guard` (every dispatch needs `args.purpose`).

This skill tests those CUJs two ways. Use **both** — they cover different things:

| Half | What it proves | Needs |
|------|----------------|-------|
| **Mock loop** (`polly_cuj.py`) | The **substrate/mechanics** — the brain is *scripted*, so this proves bundle load, server-side policy resolution, bridged `sys_*` tool dispatch, the guardrail DENYs, and fan-out — deterministically, with no creds | nothing (mock LLM) |
| **Live recipe** | polly's **judgment** — does the real brain preflight, decompose, delegate, cross-review, and pull in the human correctly | real `claude`/`codex`/`pi` + model creds + network |

> Like the sibling harness skills, turns run from your **current checkout**
> (`omni run <bundle> --server <url>` = local runner + remote server), so testing
> exercises exactly the code you're on.

## Interpreter

The driver and CLI need the repo's Python ≥3.12 env. If `.venv/` is missing,
create it once from the checkout:

```bash
uv run --frozen python -c "import omnigent; print('ok')"   # builds .venv
```

Then use `.venv/bin/python` / `.venv/bin/omni` below.

---

## Part A — the deterministic mock loop (`polly_cuj.py`)

The driver boots a throwaway local Omnigent server (which carries
`omnigent.inner.nessie.policies` — the module polly's guardrails resolve) plus
the repo's mock-LLM server, rewrites the polly bundle to the `openai-agents`
harness wired to the mock, then runs `omnigent run` turns where the brain is
*scripted* (text or tool calls). It prints one `SUMMARY {json}` per scenario and
exits non-zero if any check failed.

```bash
.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --list-scenarios
.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --scenario all
.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --scenario guardrail_purpose --keep
```

Read the result with `… | grep '^SUMMARY' | python -m json.tool`. Each run takes
~45–55s for all five scenarios; no credentials or egress are required.

### Scenario catalog

| Scenario | Scripts the brain to… | Hard check |
|---|---|---|
| `boot` | reply with text | exit 0 + non-trivial reply (bundle load, server-side policy resolve, turn completes) |
| `tool_dispatch` | call `sys_os_shell` to write a sentinel | the file appears on disk (bridged `sys_*` dispatch works; `blast_radius` ALLOWs benign shell) |
| `guardrail_purpose` | `sys_session_send` with **no** `args.purpose` | tool output carries `Denied by policy: … must declare what kind of work it is` (`headless_subagent_purpose_guard`) |
| `guardrail_blast_radius` | `sys_os_shell("git push --force …")` | tool output carries `Denied by policy: … blast-radius policy` |
| `fanout_dispatch` | emit 6 `sys_session_send` in one turn | ≥2 sub-agent dispatch handles created (fan-out substrate). **Finding:** reports whether the `spawn_bounds` cap fired (see Known sharp edges) |

### The verifiable before→after loop

The driver exists for a *loop*, not a one-shot. To prove a fix:

1. On the **unfixed** code, run the scenario → a check is `false` (baseline).
2. Make the change.
3. Run the **same** scenario → the check **flips** to `true`.

A fix is "verifiable" only if a check flips. If it doesn't flip, you can't prove
the change did anything — keep working. To cover a new mechanism, add a
`scenario_*` function + a row in `_SCENARIOS` (each builds a bundle, scripts the
mock, runs a turn, and asserts an **observable effect** — a session item, a deny
sentinel, a file on disk).

### What the mock loop can and can't prove

It tests **mechanics** because the brain is scripted: tool dispatch, the
guardrail gate, session persistence, fan-out plumbing. It does **not** test
polly's judgment (whether the *real* brain preflights, decomposes, picks the
right vendor, cross-reviews). That is the live recipe.

---

## Part B — the live recipe (real claude/codex/pi)

### Prereqs (check first)

1. **You're on the branch you want to test.**
2. **A Claude provider for the brain** (`omni setup`, or `ANTHROPIC_API_KEY`, or
   a Databricks default). Verify booleans only — never print keys.
3. **Worker CLIs on PATH** — this *is* the roster preflight:
   ```bash
   command -v claude codex pi || true
   ```
   A worker is launchable only if its binary resolved. Cross-review needs **two
   different vendors** available.
4. **Network egress** to the model backends; **`gh`** authed if you want real PRs.

### Run a live turn

```bash
.venv/bin/omni server --background && .venv/bin/omni server status   # prints $SERVER, e.g. http://127.0.0.1:6767
SERVER=http://127.0.0.1:6767
timeout 280 .venv/bin/omni run examples/polly \
  -p "Investigate how the runner enforces tool-call policies and report file:line evidence." \
  --server "$SERVER" 2>&1
```

Always pass `--server "$SERVER"`; omitting it r