oswright

# OSWright

[![PyPI](https://img.shields.io/pypi/v/oswright?color=1f6feb)](https://pypi.org/project/oswright/)
[![Tests](https://github.com/Ask-812/oswright/actions/workflows/test.yml/badge.svg)](https://github.com/Ask-812/oswright/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/oswright)](https://pypi.org/project/oswright/)
[![License](https://img.shields.io/github/license/Ask-812/oswright?color=444)](LICENSE)

**Desktop automation for AI agents, without paying for a screenshot every step.**

mcp-name: io.github.Ask-812/oswright

An MCP server that lets an LLM drive real desktop applications — the desktop
equivalent of [Playwright MCP](https://github.com/microsoft/playwright-mcp). It
keeps a model of the screen between actions and re-reads only the parts that
changed, so the same work costs an order of magnitude fewer tokens.

![OSWright transcribing an invoice into an expense form](docs/demo.gif)

Eight fields read off an invoice and typed into an expense form, verified by the
application itself. Same task, same result, **7.4× less context** than returning
a screenshot after every action. Every number on screen is measured during the
run — regenerate the whole thing with `python benchmarks/record_demo.py`.

### Why this exists

Most GUI agents re-perceive the entire screen on every step: screenshot, OCR,
hand the model an image, repeat. Measured on a live desktop, **the median
observation changes 0.012% of the screen's pixels**. Re-reading everything does
far more work than the change warrants, and charges ~2,800 image tokens whether
anything happened or not.

OSWright asks the compositor what changed, rescans only that, and answers
element lookups from the cheapest source that can. The claims below are measured
on this machine and reproducible from [`benchmarks/`](benchmarks/) — including
the ones that did not come out in its favour.

### Key Features

- **Cross-platform.** Windows (Win32 API), Linux (pynput/X11), macOS (pynput/Quartz).
- **Accessibility tree.** Find elements deterministically by role and name via Windows UI Automation — 100% accurate, instant, no model needed.
- **Fast OCR.** Windows OCR (built-in, instant) with EasyOCR fallback for Linux/macOS. Results are cached automatically.
- **Lightweight on Windows.** No PyTorch download — Windows uses the built-in OCR engine, so a full install is a few MB rather than a few GB.
- **Image matching.** Locates elements by template image via OpenCV.
- **Window management.** List, focus, minimize, close, and screenshot specific windows.
- **Screenshot diffing.** Detect when the screen changes with `wait_for_change`.
- **Clipboard access.** Read and write system clipboard for data transfer.
- **App launcher.** Launch applications and wait for them to load.
- **Auto-snapshot.** Every action returns a screenshot so the agent always sees current state.
- **43 MCP tools.** Screen, OCR, UIA, mouse, keyboard, windows, clipboard, and compound actions.
- **Incremental perception.** Rescans only the parts of the screen that changed, and can return what changed instead of a full screenshot — ~21× fewer tokens per step.
- **Screen memory.** Recognises screens it has read before and reuses them, verified by pixels — 89× cheaper than reading again.
- **Speculative perception.** Learns what actions do and confirms the expected result instead of re-reading — 19–23× cheaper, with a `surprise` report when the interface does something unexpected.
- **Adaptive waiting.** Waits for the screen to actually settle rather than sleeping a fixed 300 ms — 11.9 s saved over a 50-step task.
- **Resolution cascade.** Element lookups stop at the cheapest method that works; repeat lookups cost ~0.05 ms.
- **DPI-correct.** Coordinates are physical pixels everywhere, so clicks land correctly on scaled displays.
- **Test suite.** 237 automated tests; the desktop-driving ones skip themselves when no display is available.

### Requirements

- Python 3.10 or newer
- VS Code, Cursor, Windsurf, Claude Desktop, or any other MCP client

## Getting started

First, install the OSWright MCP server with your client.

**Standard config** works in most tools:

```json
{
  "mcpServers": {
    "oswright": {
      "command": "uvx",
      "args": ["oswright"]
    }
  }
}
```

> **Note:** If you don't have `uvx`, you can use `pip install oswright` and then set `"command": "oswright"` directly.

<details>
<summary>Claude Desktop</summary>

Follow the MCP install [guide](https://modelcontextprotocol.io/quickstart/user), use the standard config above.

</details>

<details>
<summary>Claude Code</summary>

```bash
claude mcp add oswright uvx oswright
```

</details>

<details>
<summary>VS Code</summary>

Add to your user or workspace `settings.json` under `mcp.servers`:

```json
{
  "mcp": {
    "servers": {
      "oswright": {
        "command": "uvx",
        "args": ["oswright"]
      }
    }
  }
}
```

Or use the VS Code CLI:

```bash
code --add-mcp '{"name":"oswright","command":"uvx","args":["oswright"]}'
```

</details>

<details>
<summary>Cursor</summary>

Go to `Cursor Settings` -> `MCP` -> `Add new MCP Server`. Name it `oswright`, use `command` type with the command `uvx oswright`.

</details>

<details>
<summary>Windsurf</summary>

Follow Windsurf MCP [documentation](https://docs.windsurf.com/windsurf/cascade/mcp). Use the standard config above.

</details>

<details>
<summary>Cline</summary>

Add to your `cline_mcp_settings.json`:

```json
{
  "mcpServers": {
    "oswright": {
      "type": "stdio",
      "command": "uvx",
      "args": ["oswright"],
      "disabled": false
    }
  }
}
```

</details>

<details>
<summary>Goose</summary>

Go to `Advanced settings` -> `Extensions` -> `Add custom extension`. Name it `oswright`, use type `STDIO`, and set the `command` to `uvx oswright`.

</details>

<details>
<summary>Using pip instead of uvx</summary>

If you prefer a standard pip install:

```bash
pip install oswright
```

Then use this config:

```json
{
  "mcpServers": {
    "oswright": {
      "command": "oswright"
    }
  }
}
```

Or run directly:

```bash
python -m oswright
```

</details>

## Incremental perception

Most GUI agents re-perceive the entire screen on every step: full screenshot,
full OCR, then hand the model a fresh image. Measured on a live desktop, the
median observation changes **0.012% of pixels** — so a full rescan does roughly
240× more work than the change warrants, and the screenshot it returns costs
~2,800 image tokens whether anything happened or not.

OSWright keeps a model of the screen between observations and rescans only the
regions that actually moved.

```
observe()  ->  {"changed": true,
                "added":   [{"text": "Saved", "x": 812, "y": 447}],
                "removed": ["Unsaved changes"],
                "screen_fraction_scanned": 0.015}
```

Measured on this machine over a 14-step agent loop:

| | v0.4.0 (full OCR + screenshot) | incremental |
|---|---|---|
| Median latency per step | 212 ms | **33 ms** |
| Tokens per observation | ~2,764 | **~49** |
| Tokens over 14 steps | 38,696 | **1,025** |
| Screen re-read | 100% | **16%** |

The busier the screen, the larger the gap: full OCR scales with how much text is
on screen, whereas the incremental path scales with how much *changed*. The same
comparison measures 6.5× on a quiet desktop and **14.3×** with a dense web page
open. Re-measure with [`benchmarks/`](benchmarks/) rather than trusting these.

Cost is a proxy, though, and a cheaper perception path that quietly degraded
accuracy would be worse than none. So it is checked against task completion:
scripted tasks driving the real tool surface across four applications, graded
against each application's own state — UI Automation for Calculator and
Explorer, the window title for Chrome and VS Code — never against OCR.

| configuration | Calculator | File Explorer | Chrome | tokens |
|---|---|---|---|---|
| v0.4-style (full screenshot) | 9/9 | 3/3 | 3/3 | 118,858 |
| delta only | 9/9 | 3/3 | 3/3 | **5,252** |
| delta + memory | 9/9 | 3/3 | 3/3 | 5,099 |
| delta + memory + prediction | 9/9 | 3/3 | 3/3 | 7,981 |

**Accuracy is identical across every configuration while token cost falls 23×.**
Run it with `python benchmarks/bench_tasks.py`.

### Why both pixels and accessibility

The design bets that neither perception path wins everywhere. Turning each half
off measures that rather than asserting it:

| configuration | Calculator | File Explorer | Chrome |
|---|---|---|---|
| full cascade | **9/9** | **3/3** | **3/3** |
| accessibility only | 9/9 | **0/3** | **0/3** |
| pixels only | **6/9** | 3/3 | 3/3 |

Accessibility-only — the posture most Windows GUI agents take — is perfect on
XAML and blind on a Win32 list view and on web content. Probed against VS Code
it sees **18 elements**, the entire IDE being a single node named `Chrome Legacy
Window`, while OCR reads 94 including every filename.

Pixels-only fails Calculator's buttons, because the button a human reads as `7`
is *named* `Seven`, and Windows OCR returns no digits from Calculator at all.

The cascade is the only configuration that passes everywhere.

### The resolution cascade

`find_element` and `click_element` stop at the first method that can answer,
so cost tracks how *novel* the request is rather than how large the screen is:

| Rung | Method | Typical cost |
|---|---|---|
| 0 | Already in the screen model | **~0.05 ms** |
| 1 | Rescan only what changed | ~70 ms |
| 2 | Accessibility tree (knows a Button *is* a button) | ~40 ms |
| 3 | App's own text buffer via UIA TextPattern — exact characters | ~400 ms |
| 4 | Full-screen OCR | ~250 ms |

Looking up text the model already knows is **~5,000× cheaper** than the v0.4.0
path (0.05 ms versus 244 ms). The response reports which rung answered, so you
can see what a task is actually costing.

Rung 3 is worth understanding: UIA's `TextRange.FindText` searches the
application's *own* text buffer and returns exact boun