flow-next-drive

# flow-next-drive — surface-aware UI automation

Drive any UI surface the way a real user would. Whatever driver the environment has, the work is the same shape: **observe / navigate → snapshot → act on fresh refs → capture evidence → release**. This skill is a *router*: it detects the surface, picks the highest available driver on a ladder, degrades gracefully when a richer driver is absent, and hands off to a per-rung reference for the command detail.

It orchestrates drivers — it does not reimplement them. The default rung (Vercel's `agent-browser` CLI) is the only driver assumed present; every other rung is detected and optional. A pass must succeed with whatever the environment actually has — most cloud VMs, Linux, and CI have no Computer Use, so it is never a hard dependency and never on a headless/no-display path.

> Driver ladder + universal-flow structure adapted from Ray Fernando's `running-bug-review-board` skill (Apache-2.0) — see CHANGELOG.

## Step 1 — Detect the surface, then branch

Classify the target into one of three buckets and take the matching path. The universal flow (Step 2) is shared; only the actuation and the per-surface reference differ.

| # | Surface | What it is | Path |
|---|---------|------------|------|
| A | **Web app** | A URL in a browser (localhost dev server, staging, production) | **Web ladder** (Step 3) |
| B | **Chromium-backed desktop app** | Electron / Windows WebView2 — Chromium under the hood, exposes a CDP debug port | **Web ladder** (Step 3), attaching over CDP to the app's remote-debugging port |
| C | **True-native / non-CDP surface** | macOS AppKit/SwiftUI, Catalyst, or a webview exposing no CDP (macOS WKWebView, which Tauri uses on macOS) | **Native rung** (Step 4) → Computer Use |

How to decide:

- A bare URL, or a dev/staging/prod web app → **A**.
- A desktop app you can launch with `--remote-debugging-port=<n>` (or one already exposing one) → **B**. Electron and Windows WebView2 are Chromium; the web ladder drives them by CDP-attach. Do **not** route these to Computer Use.
- A desktop app with no CDP port — genuinely native (AppKit/SwiftUI), or a macOS WKWebView / Tauri-on-macOS app — → **C**. Per-platform caveat: **Windows WebView2 is CDP-drivable (→ B); macOS WKWebView generally is not (→ C)** — verify per platform.

When unsure whether a desktop app exposes CDP, probe for B first (try to launch/attach with a debug port). If no port is reachable, fall to C.

## Step 2 — The universal flow (all surfaces)

```
observe / list what's open
navigate to the target (URL, or focus the app window)
snapshot              → fresh element refs (REQUIRED before each act)
act                   → click / fill / type / press / scroll toward the next step
verify                → confirm the expected text / state appeared
capture               → screenshot + console/errors at the moment of interest (and on failure)
release               → close the tab / end the session when fully done
```

Refs (`@e1`, `@e2`, …) go **stale** after any navigation, click, or form submit. Always re-snapshot. "Element X has pointer-events: none" or "ref not found" almost always means a stale snapshot, not a real bug — re-snapshot before concluding.

## Step 3 — Web ladder (surfaces A and B)

Probe availability top-down and use the **highest rung that passes**; fail soft to the next; the terminal rung is manual. Never hard-depend on any rung above the default.

| Rung | Driver | Use when | Reference |
|------|--------|----------|-----------|
| 1 (default) | **agent-browser** CLI | Always assumed present. CDP-based, headless-safe, no extra install. Drives web apps; drives Electron / WebView2 over CDP (`--cdp <port>` / `--auto-connect`). | `references/agent-browser.md` |
| 2 | **chrome-devtools-mcp** | You want built-in auto-wait (fewer stale-ref failures), DevTools-grade network/console inspection, Lighthouse, or to **attach to your real signed-in Chrome** (`--browser-url` / `--autoConnect`) so bot defenses don't challenge an automated profile. | `references/chrome-devtools-mcp.md` |
| 3 | **Playwright** (CLI or MCP) | The repo already has Playwright configured, or you need a headless CI-style run / large cross-browser regression suite. | `references/playwright.md` |
| 4 | **cursor-ide-browser** MCP | Running inside Cursor with this MCP installed and you want its snapshot YAML + `browser_cdp` control. | `references/cursor-ide-browser.md` |
| 5 (terminal) | **Manual + screenshot relay** | No browser driver available — drive yourself, paste console errors and screenshots into chat. | — |

**Surface B note:** the SAME ladder drives Electron / WebView2 apps — attach to the app's remote-debugging port (`agent-browser --cdp <port>` / `--auto-connect`; chrome-devtools-mcp `--browser-url=http://127.0.0.1:<port>`). Launch the app with a dedicated debug port and a dedicated user-data-dir; treat the open debug port as a security exposure (any local app can drive that session).

> **agent-browser command detail lives in the rung reference, not here.** The default-rung reference [`references/agent-browser.md`](references/agent-browser.md) is the entry point — setup/version check, the universal flow in agent-browser commands, the Chromium-desktop (Electron / WebView2) CDP driver, the `--headed` daemon-reuse gotcha, and an index into the per-topic references it folds: `commands.md`, `advanced.md` (CDP attach), `auth.md`, `snapshot-refs.md`, `session-management.md`, `proxy.md`, `debugging.md`.

## Step 4 — Native rung (surface C): Computer Use

A genuinely native app (or a non-CDP webview) has no browser tab to attach to. The only way to drive it is **Computer Use** — the model looks at the screen, moves a cursor, clicks, and types. Driver-agnostic across what the host offers: **Codex Computer Use** (macOS/Windows) and/or **Anthropic "Claude" Computer Use** (the API `computer` tool, run via its own harness — a controlled display/sandbox or an MCP wrapper). Detect availability and use wh