cua-driver

# Qwen Cua Driver

Orchestrates cross-platform app automation via `qwen-cua-driver`. Whenever
a user asks to drive a native app, follow the loop in this skill
rather than calling tools ad-hoc — the snapshot-before-action
invariant is not optional and silently breaks if you skip it.

## Platform-specific reading — read this first

This file is the **cross-platform core**: snapshot invariant, CLI vs
MCP choice, tool surface naming, behavior matrix, canonical loop,
pixel-click contract, common failure modes. The platform-specific
material (forbidden-list, accessibility tree implementation, launch
semantics, click dispatch) lives in companion files in this same
directory:

- **macOS** — read `MACOS.md` (no-foreground contract, forbidden
  `open`/`osascript`/`cliclick` invocations, AXMenuBar navigation,
  SkyLight pixel-click dispatch).
- **Windows** — read `WINDOWS.md` (UIA tree vs AX, UWP /
  ApplicationFrameHost hosting, layered UIA+PostMessage click chain,
  Session 0 isolation, Windows-specific focus-steal vectors).
- **Linux** — read `LINUX.md` (X11 background input via AT-SPI +
  XSendEvent and compositor-specific Wayland capabilities).

Cross-cutting topics also have their own files:

- `BROWSER.md` — exact native-window binding, explicit browser preparation,
  typed Chromium/Electron page tools, input trust classes, and native
  fallbacks for browser chrome and unsupported engines.
- `RECORDING.md` — session recording + `replay_trajectory`.

Use whichever combination matches the host. When in doubt, run
`qwen-cua-driver doctor` — it reports the platform and the right entry
point.

## Start with the narrowest semantic route

Before opening or operating an application, name the desired postcondition and
use the first applicable route below. Verify the result in the same domain
before stopping or advancing:

0. **Caller-provided headless/background operation for a non-GUI outcome.**
   Prefer an exact application API/SDK, service or database client, CLI, or
   filesystem operation over imitating a user. This includes batch-safe file
   moves, renames, copies, directory creation, archive extraction, data
   conversion, and process inspection. Read the resulting semantic state back;
   a zero exit status alone is not proof.
1. **Typed Cua operation for an application or window outcome.** Use
   `set_window_frame` for exact geometry, `invoke_menu` for a known native
   application-menu path, typed browser tools for supported page content, and
   clipboard tools for clipboard state. Verify with
   `list_windows`, `get_browser_state`, or `clipboard_read`, respectively.
2. **Background accessibility action.** Use a fresh AX/UIA/AT-SPI target.
3. **Background pixel action.** Use the pixels from the same state snapshot.
4. **Foreground delivery.** Retry only the action that evidence says could not
   land in the background.
5. **Desktop fallback.** Select an exact desktop target for that call only.
   Later calls may return to an exact window target in the same session.

Use Cua Driver when the outcome lives in an application's UI or window state,
or when the user explicitly asks to operate that GUI. Once the task crosses
that boundary, do not replace Cua's targeted and verified actions with shell
scripts that mutate the app UI. A shell is a capability of the calling agent,
not of the Cua Driver MCP server; an MCP-only client must not assume one exists.

### Filesystem outcomes and GUI fallbacks

When the requested outcome is a filesystem change and the caller has a
headless filesystem or command capability, keep it on rung 0. Enumerate the
exact source set, decide the destination-conflict policy before changing
anything, perform one batch-safe operation, then independently read back both
source and destination manifests. Do not open a file manager merely to mimic a
move, copy, or rename that the caller can execute and verify directly.

If the caller has no such capability, use the file manager as a GUI fallback
and keep each claim narrow:

1. After entering an inline rename and setting its value, commit it with the
   platform's confirmation key, then take a fresh snapshot. Value readback from
   the inline editor proves only that the editor changed; it does not prove the
   filesystem rename committed.
2. For a multi-selection, use the platform modifier (`cmd` on macOS, `ctrl` on
   Windows/Linux). On macOS and Windows, issue that modified click with
   `delivery_mode:"foreground"` so the target observes physical modifier state;
   a refused background attempt is an escalation signal, not a failed action to
   trust or repeat. Re-snapshot before the next operation. Continue only when
   every intended item is selected and the prior selection was preserved.
3. After a cross-window drag or paste, verify the destination contains the
   complete expected set and the source reflects copy-versus-move semantics.
   A delivered drag, keypress, or menu action is not file-operation proof.
4. If a destination conflict presents an unrecognized policy or ambiguous
   partial result, stop that GUI path and surface the unresolved state instead
   of retrying blindly.

### Clipboard outcomes and GUI fallbacks

When the requested postcondition is an exact value on the system clipboard,
rather than the literal gesture of selecting and copying it, keep the operation
semantic. Read the value from the narrowest typed source, call
`clipboard_write`, then prove the real clipboard state with `clipboard_read`.
For browser content, this means reading the page with `get_browser_state` and
writing the exact observed text; a passive page-text ref does not need to be
clicked first.

Use visual selection followed by the platform copy hotkey only when the user
explicitly asks for that gesture, the source cannot expose the value
semantically, or direct clipboard tools are unavailable. Treat that as a GUI
fallback: re-snapshot before acting, verify the selected range when the
application exposes it, and escalate only the del