computer_use

# Computer Use

Use Computer Use only for tasks that require a live desktop interface or
visual verification. Prefer a purpose-built integration or command-line tool
when it can complete and verify the task.

Use only the native desktop runtime. It operates on one approved application
and one observed window at a time; it never accepts a free-form screen target.

## Operating Loop

Follow this loop for every task:

1. Discover the canonical application and the correct window.
2. Observe the window and identify the requested state from current evidence.
3. Define the next expected visible or accessible state change.
4. Choose one action channel and perform the smallest useful action.
5. Inspect the replacement observation before deciding the next action.
6. Observe the final state and verify every requested outcome before reporting
   success.

Treat `dispatched: true` or an intermediate acknowledgement only as evidence
that input was sent, not that the application completed the operation. If the
final state is incomplete or uncertain, report that accurately.

## Discover the Target

1. Call `list_apps` and select the canonical App ID.
2. Call `list_windows`, optionally limited by that App ID.
3. Match the target by title, content, and observed state. When several windows
   are plausible, observe them read-only until one matches; never choose only
   because it is first or most recent.
4. Keep using the matched `window_id` until an action explicitly hands off to
   another window.

Use `launch_app` with a canonical App ID. If the application is not listed,
use an explicit absolute executable path on Windows or application-bundle path
on macOS. After launch, list its windows again because launch completion does
not prove that a usable window already exists.

When the runtime reports a missing system permission, stop and ask the user to
grant it. Do not retry until the user confirms the permission was granted.

## Read an Observation

`observe_window` returns a point-in-time window observation with screenshots
and accessibility text. Start with:

- `accessibility.focused_element`: the control that owns keyboard focus.
- `accessibility.document_text`: a capped view of the focused document; never
  assume it contains the complete document when truncated.
- `accessibility.elements`: actionable controls and their current properties.

Each accessibility line begins with an `element_id`, control type, and name.
Use labels, roles, identifiers, actions, and current state together; do not
infer behavior from an opaque identifier alone.

Indentation preserves the native accessibility hierarchy. Use parent and
container context to distinguish controls with duplicate names.

Each attached image has a `screenshots[].id`, image-local dimensions, screen
origin, kind, and z-index. On Windows, one observation may include the selected
window plus related menus, drop-downs, or dialogs. Treat the highest z-index
related image as the frontmost visual surface, while keeping the original
`window_id` as the stable target. Attached images and `screenshots` entries
use the same order.

Common markers:

- `[disabled]`: do not act on this element.
- `[offscreen]`: scroll it into view first.
- `[selected]`: the application selected this exact element.
- `[settable]`: `set_value` is supported.
- `[actions=...]`: invoke only an explicitly listed action.
- `[resource-backed]`: the label represents an application-owned object, not
  an editable text buffer.

When duplicate names exist, discard disabled candidates, then choose by role,
actions, identifier, and surrounding state. Prefer accessibility elements over
coordinates. When `visual.available` is false, continue only with listed
elements, semantic actions, or verified keyboard focus; coordinates are not
valid for that observation.

Every successful desktop mutation invalidates its input observation. The
response normally installs and returns a settled replacement observation.
Inspect it before the next action and derive fresh element IDs from it.
Post-action replacements do not attach images. On Windows,
`visual.related_surface_count` reports related menus, drop-downs, or dialogs
seen during that lightweight refresh. Call `observe_window` before choosing a
visual target or using coordinates.

Interpret result fields conservatively:

- `accessibility_changed: false` means no AX-visible transition was observed;
  it does not rule out a visual-only change.
- `effect: observed` verifies the edited buffer; `effect: unverified` requires
  confirmation from replacement state or a fresh observation.
- Follow an explicit `next_action` before choosing another action. Use a
  returned replacement observation or window when present.
- `requires_observe` invalidates the current observation, not necessarily the
  window. Reobserve the current or returned window for `observe_window`; use
  `list_windows` to rediscover a target only when instructed.
- `confirmation_required` or `pending_action` means the edit is not complete.

When a visual transition is expected to expose an accessibility element, wait
and observe again until it becomes actionable or the operation times out or
stops making progress. For a stable control with no accessibility
representation, use current screenshot coordinates as described below and
observe again after acting.

`wait` only delays execution; it does not observe or verify application state.
Call `observe_window` afterward when current state is needed.

## Choose an Action

Use the safest channel that expresses the requested operation:

1. Use an observed semantic element when available.
2. Use a platform-standard shortcut when focus and target are verified.
3. Use current screenshot coordinates only when accessibility is unavailable
   or unsuitable.

Preserve the semantics and side effects of the requested operation. Do not
approximate an unsupported operation with a broader sequence that adds side
effects. If no available action preserves the reques