unix_computer_use

# Unix Computer Use

This bundled extension exposes a reviewable computer-use substrate for local
macOS/Linux desktops and explicitly configured remote targets. **All actions are
for supervised, low-risk workflows under human or benchmark observation.** Windows
support is deferred to a separate backend/skill.

Tools:

- `capabilities` reports the platform, display-session type, and detected
  screenshot/input backends plus the active connection/backend.
- `list_connections`, `add_connection`, `test_connection`, `activate_connection`,
  and `use_local` manage the local/remote connection registry in skill state.
- `screenshot` captures the desktop, downscales the image to fit WXGA
  (1280x800, configurable) and persists the exact image-to-input coordinate
  transform. Pass coordinates read off the returned image directly to the
  input tools — they are remapped automatically. The result carries the typed
  `auto_attach_image` field, so the host attaches the image to the
  conversation in the same round (v6.81.1) — no `view_image` call is needed
  for a fresh capture; the returned `path` stays `view_image`-readable for
  re-viewing later.
- `click` (left/right/middle, double/triple), the thin aliases
  `double_click`/`triple_click` (models call these names unprompted), `move`,
  `left_click_drag`, `mouse_down`/`mouse_up`, `cursor_position`, `type_text`,
  `key`, `hold_key`, `scroll`, and `wait` execute input through platform tools
  when available. Pointer coordinates pass through one normalizer that accepts
  the malformations models actually emit (the pair packed into `x` with `y`
  absent or duplicated) and fails loudly on anything ambiguous.
- `window_list` lists visible windows/processes where the platform exposes a
  lightweight backend.
- `ax_tree` returns a set-of-marks accessibility snapshot of the frontmost
  window on macOS: numbered interactive elements (role, title, center
  coordinates in input space — click them with `raw=true`). It degrades
  honestly to a process/window list when the AX walk fails or on Linux.
- `remote_exec` runs a shell command on the active remote backend only; it
  refuses on local backend. Use ordinary Ouroboros shell/file tools for local
  work.

Coordinate contract:

- Input tools accept coordinates in the LAST screenshot's image space and
  remap them through the stored transform; `raw=true` bypasses remapping.
- macOS input consumes LOGICAL points while screenshots are physical pixels;
  the stored transform already folds the Retina scale in. On multi-display
  Macs the scale is approximate (`approx` flag) — prefer `ax_tree` marks.

Backend matrix (honest limitations):

- macOS: `screencapture` + `cliclick` + `osascript`; `sips` for downscaling.
  `scroll` is **unsupported** (cliclick has no scroll-wheel command — page via
  `key` with `page-down`/`page-up`); `middle` click and non-left
  `mouse_down`/`mouse_up` are unsupported via cliclick. `hold_key` holds
  PURE-MODIFIER combos only (`cmd`, `cmd+shift`, ...): cliclick `kd:`/`ku:`
  accept only modifiers and `kp:` cannot hold — non-modifier holds report an
  honest error. In `key` combos the base key is tapped while modifiers are
  held (`cmd+s` = hold cmd, tap s). TCC permission state (Screen Recording /
  Accessibility) is NOT verified: tools exit 0 even when denied
  (wallpaper-only capture / dropped input) — ensure grants.
- Linux X11: `xdotool` (input), `gnome-screenshot`/`scrot` (capture),
  `wmctrl` (windows), ImageMagick (downscaling). Function keys and
  case-sensitive keysyms are supported (`f5`→`F5`, `XF86AudioPlay` as-is).
- Linux Wayland: `ydotool` (pointer + typing; requires a running `ydotoold`)
  or `wtype` (typing only), `grim` or `gnome-screenshot` (capture). `xdotool`
  does NOT work on Wayland. `key` and `hold_key` are **unsupported** on
  Wayland (`ydotool key` takes raw keycodes only — combos would silently
  no-op, which this skill refuses to fake); use `type_text` for text.
  Pointer press/release uses ydotool button masks (`0x40`/`0x80`).
- Remote OSWorld HTTP: `screenshot` uses `GET <target>/screenshot`; input and
  `remote_exec` use `POST <target>/execute`, matching OSWorld's in-VM
  `pyautogui` channel. The runner writes an `osworld_http` connection before
  each task. Honesty guarantees on this backend: `scroll` clicks are 1:1 wheel
  detents (not multiplied); non-ASCII `type_text` is pasted via the in-VM
  clipboard (`pyperclip` + `ctrl+v`, ASCII `typewrite` fallback) so Unicode is
  not silently dropped; every action's `ok` reflects the guest process
  `returncode`/`status` (HTTP 200 alone is not success); `key` `backspace`
  presses BackSpace and `fwd-delete` presses forward Delete; screenshots are
  capped at 20 MB and PNG-validated before use.
- Remote SSH macOS: `screenshot` uses remote `screencapture` + `scp`; input uses
  a remote input helper path. The remote Mac must have Screen Recording and
  Accessibility grants configured by the owner. This skill does not install or
  store private SSH keys.

Connection registry:

- Connections live in `data/state/skills/unix_computer_use/connections.json`.
- Default active connection is `local`.
- `add_connection` can add `osworld_http` or `ssh_macos` connection metadata, but
  it never accepts private key material.
- For SSH, configure `~/.ssh/config`, `IdentityFile`, and/or `ssh-agent`
  yourself. If auth fails, `test_connection` tells you what to fix.
- Agents should add/test/activate connections only when the user or a benchmark
  runner provides an explicit target. They should not scan for arbitrary remote
  machines.
- A disabled active connection FAILS CLOSED: action tools return an error rather
  than silently falling back to the local desktop (an action aimed at a remote
  target must never land on the host).

Permissions note: the remote backends require the `net` permission. `net` needs
no owner grant (skill grants only gate `inject_chat`/`subscribe_event`), but it
does remove this skill from the launcher's