cli-setup-verify

# Verifying the Omnigent CLI setup & UX in a closed loop

The Omnigent CLI's first impression is: `curl | sh` → run `omnigent` → pick a
model credential → start a session. This skill lets an agent **enter that flow,
examine the UI/UX, and prove whether a change is verifiable** — without a
browser, without real credentials, and **without ever touching the developer's
real `~/.omnigent`**.

The engine is `verify_cli.py` (next to this file). It drives the real
`omnigent` binary through a pseudo-terminal (`pexpect`) inside a throwaway
sandbox, captures what renders, runs assertions, and prints one machine-readable
`SUMMARY {json}` line.

> **The whole point is a verifiable loop**, not a one-shot check:
> 1. Run a scenario on the **unfixed** code → baseline (`--label before`).
> 2. Make the change.
> 3. Run the **same** scenario → `--label after`.
> 4. Diff the two `SUMMARY` lines. A fix is "verifiable" only if a concrete
>    check or note **flips** between the two runs. If it doesn't flip, you
>    can't prove the fix did anything — go back to step 2.

## Why this is safe (read first)

The real `~/.omnigent` here can be **many GB** (chat DB, runner logs, native
harness state). The sandbox isolates every write three ways:

- **`HOME` is redirected into the sandbox by default.** This is the load-bearing
  one. `OMNIGENT_CONFIG_HOME` / `OMNIGENT_DATA_DIR` (`omnigent/cli.py`
  `_CONFIG_HOME_ENV_VAR` / `_DATA_DIR_ENV_VAR`) redirect config + data — but the
  CLI's **diagnostics logger ignores them**: it writes a per-invocation
  `cli-*.log` under `state_dir()`, hardcoded to `Path.home()/.omnigent/logs`
  (`omnigent_ui_sdk/terminal/_config.py`). So only redirecting `HOME` keeps a
  non-help command (`config list`, the setup PTY spawns, `server stop`) from
  writing into the real home. The driver does this for you.
- `--strip-path` reduces `PATH` so `node`/`tmux`/`claude`/`codex` read as "not
  installed" → the true fresh-machine cold start.
- Ambient model keys (`ANTHROPIC_API_KEY`, …) are stripped from the child env
  unless you pass `--keep-env-creds`.

**`--inherit-home` opts out** of `HOME` isolation — use it only to reach a real
credentialed REPL via ambient `~/.claude` / `~/.databrickscfg` auth. It is
**less safe**: a non-help command then writes `cli-*.log` into the real
`~/.omnigent/logs`.

Every run **fingerprints the real `~/.omnigent` before and after** (stat-only,
no content reads): the top-level config files **and** the set of
`logs/cli-*.log` diagnostic files. A new config file/mtime *or* a new `cli-*.log`
basename trips `real_config_untouched: false`. With the default isolation that
never happens; under `--inherit-home` it correctly does — which is exactly the
violation the guard is meant to catch. If that check is ever `false`, stop and
investigate. Run `check-isolation` first to confirm the loop is safe on your
machine.

## Prerequisites

- You're in the **worktree whose code you want to test** (each parallel agent
  on its own worktree). The driver runs `omnigent` from `--repo`'s checkout.
- A Python with `pexpect` — the project's `.venv/bin/python` bundles it
  (`pexpect>=4.9` in `pyproject.toml`). Run the driver with that interpreter.
- An `omnigent` binary: the driver auto-finds `<repo>/.venv/bin/omnigent`, or
  pass `--omnigent <path>`.
- The setup / picker / help / cold-start scenarios need **no credentials and no
  harness**. Only `repl-commands` needs a working harness + credential: pass
  `--inherit-home` (ambient `~/.claude` auth) and/or `--keep-env-creds` (env API
  key) with `--agent`. It reports `skipped`, never a false pass, when the prompt
  isn't reachable.

## Quick start

```bash
REPO=/path/to/your/worktree
PY=$REPO/.venv/bin/python
DRV=$REPO/.claude/skills/cli-setup-verify/verify_cli.py

# 0. Prove the sandbox is safe on this machine (do this once).
#    HOME is isolated by default — no flag needed.
$PY $DRV --scenario check-isolation --repo "$REPO"

# 1. See exactly what a brand-new user sees on a fresh machine.
$PY $DRV --scenario cold-start --strip-path --keep-sandbox --repo "$REPO"
#    → reads the printed `artifacts` path, then `cat <that>/cold_start.txt`

# 2. Lint the top-level help (and any subcommand's).
$PY $DRV --scenario help-snapshot --repo "$REPO"
$PY $DRV --scenario help-snapshot --subcommand server --repo "$REPO"
```

Each run prints `SUMMARY {…}` and exits non-zero if any check failed (a
`skipped` scenario exits 0). Pipe to `… | grep '^SUMMARY' | python -m json.tool`
to read it.

## Scenario catalog

| Scenario | What it drives | Key checks / notes | Maps to findings |
|---|---|---|---|
| `check-isolation` | `omnigent config list` in the sandbox (no PTY) | `config_list_ran`, `sandbox_config_home_used`, `real_config_untouched` | safety gate for everything |
| `cold-start` | `omnigent setup` via PTY on a simulated fresh machine | `onboarding_rendered`, `harness_menu_present`; note `guided_default_affordance` | cold-start dead-end; missing "recommended start here" |
| `setup-snapshot` | `omnigent setup`, optional `--nav-down N` arrow steps | `menu_rendered`; saves a frame per step | picker markers/footer/alignment; narrow-terminal at 80×24 |
| `help-snapshot` | `omnigent [--subcommand] --help` (no PTY) | `help_rendered`, `no_param_leak`, `no_update_dup`; note `top_level_command_count` | `:param` leak, duplicate `update`/`upgrade`, command sprawl |
| `repl-commands` | `omnigent run <agent>` REPL, sends `/help` + `/quit` | `help_lists_commands`; note `quit_advertised` | REPL discoverability (`/help`, `/quit`) |

`--list-scenarios` prints them too. Captured frames land in the printed
`artifacts` dir as both `<name>.txt` (ANSI-stripped, for reading/asserting) and
`<name>.ansi.txt` (raw, to see real colors with `less -R`).

## The verifiable loop — a worked example

Finding: *"`server --help` leaks Sphinx `:param`/`:returns` into user help."*

```bash
# BEFORE the fix (on the unfixed code):
$PY $DRV --scenario help-snapshot --subcommand server --label