copilot-sdk-e2e-dev

# Copilot SDK harness: end-to-end dev & testing

The `copilot` harness drives the **GitHub Copilot SDK** (`github-copilot-sdk`,
imported as `copilot`) — a persistent `CopilotClient` + `CopilotSession` per
Omnigent conversation — and bridges Omnigent's `sys_*` tools into Copilot as SDK
`Tool`s. The Python SDK **bundles the Copilot CLI binary it drives** as a backing
server, so there is no separate `@github/copilot` install. This skill is the
proven recipe for running it **for real** against a live local server — not just
the unit tests.

> The harness runs as a **local runner** from your current checkout, so
> `omni run <bundle> --server <url>` exercises exactly the code you're on.

## Prerequisites (check these first)

1. **You're on the branch you want to test.** The copilot harness is an
   optional extra — install it (without disturbing other extras) with
   `uv sync --frozen --extra dev --extra copilot`. NB: a bare
   `uv run --frozen --extra dev` re-syncs the venv and **prunes** the copilot
   SDK; for live testing call `.venv/bin/omni` / `.venv/bin/python` directly and
   avoid `uv run` mid-session.
2. **The SDK is installed:**
   `.venv/bin/python -c "import copilot; print(copilot.__file__)"`.
3. **A GitHub token with Copilot access is configured.** Copilot needs a
   fine-grained PAT with the "Copilot Requests" permission, or an OAuth token
   from the GitHub CLI / Copilot CLI app (classic `ghp_` PATs are rejected).
   Verify (booleans only — never print the token):
   ```bash
   .venv/bin/python -c "from omnigent.onboarding.copilot_auth import copilot_github_token_configured; import os; print('config:', copilot_github_token_configured(), 'env:', bool(os.environ.get('GH_TOKEN') or os.environ.get('COPILOT_GITHUB_TOKEN')))"
   ```
   If both are `False`, run `omni setup` and register a Copilot token, or
   `export GH_TOKEN=$(gh auth token)` (when `gh` is logged into an account with
   Copilot). Check the account's entitlement with
   `gh api /copilot_internal/user` (look for `chat_enabled`/`cli_enabled`).
4. **Network egress to GitHub's Copilot backend.** A turn that hangs or fails to
   connect on a locked-down host is usually egress, not a harness bug.

## Step 1 — start a local server

```bash
cd /path/to/omnigent
.venv/bin/omni server --port 7788 --no-open    # foreground; or `omni server start` for detached
curl -s http://127.0.0.1:7788/health           # {"status":"ok"}
```

Use the URL below as `$SERVER`.

## Step 2 — build a copilot agent bundle

A spec with `spec_version` **must be a directory containing `config.yaml`** —
not a single `.yaml` file. Minimal copilot agent:

```bash
mkdir -p /tmp/copilot-dev
cat > /tmp/copilot-dev/config.yaml <<'YAML'
spec_version: 1
name: copilot-dev
description: Copilot SDK dev/test agent.
executor:
  type: omnigent
  config:
    harness: copilot
    # model: gpt-5-mini      # optional; omit for Copilot auto-select
prompt: |
  You are a terse test agent. Answer in as few words as possible.
YAML
```

For sub-agents, tools, guardrails/policies, copy the field shapes from
`examples/polly/config.yaml` and `examples/debby/config.yaml`. (Declare policies
under `guardrails.policies:` — a top-level `policies:` key is silently dropped on
the `spec_version` + `config.yaml` path.)

## Step 3 — run a turn (and smoke-test)

```bash
SERVER=http://127.0.0.1:7788
timeout 280 .venv/bin/omni run /tmp/copilot-dev \
  -p "Reply with exactly the single word: PONG" \
  --server "$SERVER" 2>&1
```

A healthy run prints connection lines then the reply (`PONG`). If that works,
the full stack is good: token, egress, bundled CLI, harness.

- **Shell / file tools:** add `--tools coding`.
- **Specific model:** add `--model gpt-5-mini` (or `claude-haiku-4.5`, `auto`).

## Targeted scenarios

| Goal | How |
|------|-----|
| Native tools (shell/edit/read) | `--tools coding`, prompt to create→read→edit a file; confirm it actually touches disk |
| Bridged `sys_*` / sub-agent dispatch | declare a sub-agent (harness `copilot` so auth is satisfied), prompt the parent to delegate — exercises the SDK `Tool` async-handler bridge into `_tool_executor` |
| Model routing | run the same bundle with several `--model` values; an unknown id fails **loud**, a `databricks-*` id is dropped to auto with a warning |
| LLM-phase policy | add a guardrail that denies a keyword; confirm `PHASE_LLM_REQUEST`/`PHASE_LLM_RESPONSE` blocks it |
| Concurrency / leaks | fire several `omni run … &` at once; then `pgrep -af "copilot/bin/copilot"` to check for orphaned bundled-CLI subprocesses |

## Running polly (or any orchestrator) on a copilot brain

The copilot harness can serve as an **async orchestrator** brain (polly / debby),
not just a standalone agent — it dispatches to sub-agents via the bridged
`sys_*` tools and synthesizes their results. Two ways to exercise it:

**1. Committed regression guard (brain smoke).**
`tests/e2e/test_polly_copilot_e2e.py` boots a local server from your checkout and
runs `examples/polly` with `--harness copilot --model auto`, asserting the brain
boots and replies. It is **skipped** unless a Copilot token is configured (so CI
without one skips it). Run it with:

```bash
.venv/bin/python -m pytest -o addopts="" tests/e2e/test_polly_copilot_e2e.py -v
```

**2. Full orchestration (dispatch → collect → synthesize).** Use the
`polly-e2e-dev` driver (in the internal `agent-framework` clone) — it boots a
local server, polls the AP API, auto-answers elicitations, and asserts the
fan-out. Drive the brain on copilot with `--brain-harness copilot`, and **always
pass a Copilot-catalog `--brain-model`** (`auto`, `claude-haiku-4.5`,
`gpt-5-mini`): the driver's default `--brain-model` is a Claude id that Copilot
(no Databricks gateway) can't route. From the agent-framework clone:

```bash
.venv/bin/python .claude/skills/polly-e2e-dev/polly_driver.py \
  --local --code-dir <this-worktree> \
  --cuj smoke --brain-harness copilot --brain-model auto      # brain only
# --cuj fanout  …