computer-use-for-all-agents

# 🖥️ Screen Control

<p align="center">
  <img src="docs/images/logo.png" alt="screen-control logo" width="88" align="right">
</p>
<p align="center">
  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
  <img alt="Platform" src="https://img.shields.io/badge/platform-Windows%2010%2F11-blue">
  <img alt="Python" src="https://img.shields.io/badge/python-3.10%2B-informational">
  <a href="https://github.com/features/actions"><img alt="CI: security tests" src="https://img.shields.io/badge/CI-security%20tests-brightgreen"></a>
  <a href="#mcp-support-one-click-cloud-agents"><img alt="MCP" src="https://img.shields.io/badge/MCP-compatible-9370DB"></a>
  <a href="https://mcpservers.org/servers/xeakaes/computer-use-for-all-agents"><img alt="Listed on mcpservers.org" src="https://mcpservers.org/badge.svg"></a>
</p>

> A local remote-control system for AI agents: watch your computer's screen
> **live** and send **mouse/keyboard commands** to it. Everything runs on your
> own machine — no data ever leaves it, no cloud middleman.

<p align="center">
  <img src="docs/images/demo-typing.gif" alt="An AI agent typing into Notepad through the screen-control API" width="820">
</p>

<p align="center">
  <sub><i>Every keystroke and every screenshot in this GIF went through the API —
  the agent never touched a physical keyboard.</i></sub>
</p>

---

## Why Screen Control?

AI agents today can write code and call APIs — but they can't *see* or *touch*
your desktop. Screen Control gives any agent general-purpose computer use over
a clean, safety-gated HTTP/MCP interface:

- **Perceive** — OCR for text, single frames or a live MJPEG stream for
  vision-capable models, and a text-only diff endpoint for models that can't
  consume images at all.
- **Act** — absolute and relative mouse, Unicode-safe keyboard, window
  management, background (focus-free) control, virtual desktops.
- **Stay safe** — token auth, blocked deadly shortcuts, focus guard, a
  stuck-input watchdog and an emergency failsafe are all enforced server-side,
  no matter how confused the agent gets.

One process, zero configuration, works with any language that can speak HTTP —
or natively through MCP in Claude Desktop, Cursor, VS Code and cloud agents.

### Performance Is Agent-Bound

Screen Control is the **perception and actuation layer** — the eyes and hands.
The effective speed and capability of any agent using it are bounded by that
agent itself and by the environment it runs in:

- **Thinking speed** — one action per agent "turn": the perceive → plan →
  act → verify loop lives in the agent, so model inference latency and
  reasoning depth directly set the pace. The API itself adds only
  milliseconds per call.
- **Context capacity** — screen readings (OCR text, frames, diffs) consume
  the agent's context window; a larger window means more situational
  awareness before verification degrades.
- **Runtime environment** — network latency, MCP/HTTP round-trip overhead,
  tool-call limits and hosting constraints all stack on top of the loop.

In practice this means: the same repo makes a fast reasoning model fast and
capable, and makes a slow model slow — the toolchain is not the bottleneck.
Real-time or action-heavy tasks need an agent with fast inference and tight
tool-loop latency; slower agents should prefer deliberate, verification-heavy
tasks.

---

## Table of Contents

- [Why Screen Control?](#why-screen-control)
- [Features](#features)
- [Architecture](#architecture)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
  - [Authentication](#authentication)
  - [Screen Capture](#screen-capture)
  - [Mouse Control](#mouse-control)
  - [Keyboard Control](#keyboard-control)
  - [OCR (Screen Reading)](#ocr-screen-reading)
  - [Vision Access (Image Models)](#vision-access-image-models)
  - [Window Management](#window-management)
  - [Focus-Free (Background) Control](#focus-free-background-control)
  - [Virtual Desktops](#virtual-desktops)
  - [Game Mode](#game-mode)
  - [Safety Endpoints](#safety-endpoints)
- [🤖 For AI Agents](#-for-ai-agents)
- [MCP Support (One-Click Cloud Agents)](#mcp-support-one-click-cloud-agents)
- [Security Model](#security-model)
- [Game Mode Guide](#game-mode-guide)
- [Vision Access Guide](#vision-access-guide)
- [Troubleshooting](#troubleshooting)
- [Testing](#testing)
- [Contributing](#contributing)
- [License](#license)

---

## Features

| Feature | Description |
|---|---|
| 🖼️ **Live screen feed** | Continuously refreshing screenshot in the browser |
| 🖱️ **Mouse control** | Click, right-click, double-click, scroll, drag & drop via live screenshot |
| ⌨️ **Keyboard control** | Text typing (Unicode/Turkish included, layout-independent), keys and shortcuts (Ctrl+C, Alt+Tab…) |
| 👁️ **OCR** | Converts on-screen text to machine-readable format |
| 📷 **Vision access** | Raw-pixel paths for image-capable models: single frames, MJPEG stream, text-based motion detection |
| 🪟 **Window management** | List, focus, safe close (WM_CLOSE), kill (task-manager style) |
| 🖥️ **Focus-free control** | Read/write background windows via PostMessage without stealing focus |
| 🎮 **Game mode** | Camera look via relative mouse movement, hold-to-move keys |
| 🔐 **Token auth** | Every request requires `X-Auth-Token` (CSRF protection) |
| 🦺 **Stuck-input watchdog** | Auto-releases held keys after 30 s of inactivity |
| 🛟 **Failsafe** | Cursor to top-left corner aborts all commands (disabled in game mode) |

---

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    Browser (Web UI)                      │
│  ┌──────────┐  ┌──────────┐  ┌────────────────────────┐ │
│  │  Live     │  │  Control  │  │  Windows / Game Mode   │ │
│  │  View     │  │  Panel    │  │  Panel                 │ │
│  └────┬─────┘  └────┬─────┘  └───────────┬────────────┘ │
│       │              │                     │              │
└───────┼──────────────┼─────────────────────┼──────────────┘
        │              │                     │
        ▼              ▼                     ▼
┌─────────────────────────────────────────────────────────┐
│                  HTTP API (Flask)                        │
│                  127.0.0.1:8745                           │
│                                                         │
│  /api/screenshot    /api/mouse     /api/key              │
│  /api/vision/*      /api/ocr       /api/window           │
│  /api/game          /api/held      /api/release_all      │
│  /api/windows       /api/desktops  /api/desktop          │
│                                                         │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │ Auth Layer  │  │  Watchdog    │  │  OCR Engine  │   │
│  │ (token)     │  │  (30s auto)  │  │  (RapidOCR)  │   │
│  └─────────────┘  └──────────────┘  └──────────────┘   │
└─────────────────────────────────────────────────────────┘
        │              │                     │
        ▼              ▼                     ▼
┌─────────────────────────────────────────────────────────┐
│                  control.py (Core)                        │
│                                                         │
│  Screen:  mss (fast capture), PIL (processing)          │
│  Mouse:   pyautogui (absolute), SendInput (relative)    │
│  Keyboard: pyautogui + SendInput+KEYEVENTF_UNICODE      │
│  Windows: Win32 API (EnumWindows, SetForegroundWindow)   │
│  Background: PrintWindow (capture), PostMessage (input)  │
│  Virtual Desktops: pyvda                                 │
│  Game Mode: ClipCursor + MOUSE_MOVE_RELATIVE             │
└─────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│              backends/ (pluggable)           │
│  WindowsBackend  │ LinuxBackend │ MacOSBackend│
│      (full)      │   (stub)     │   (stub)   │
└──────────────────────────────────────────────┘
```

### Coordinates & Concurrency

**Per-Monitor DPI awareness.** `control.py` calls
`SetProcessDpiAwarenessContext(PER_MONITOR_AWARE_V2)` at import time —
**before** the `pyautogui` import, because pyautogui touches coordinate APIs
during import and would otherwise lock the process to the interpreter
manifest's default (system-aware). With PMv2 active, every coordinate in the
system is a **physical pixel** end to end: mss capture, OCR bounding boxes,
pyautogui/SendInput clicks, ClipCursor. On High-DPI displays (125%/150%
scaling) nothing drifts between what OCR reports and where the mouse clicks.

**Lock architecture.** The server uses two independent locks instead of one
global lock:

| Lock | Protects | Endpoints |
|---|---|---|
| `_input_lock` | mouse, keyboard, game mode, window ops | `/api/mouse`, `/api/key`, `/api/game`, `/api/window/post`, ... |
| `_read_lock` | capture, OCR, vision, enumeration | `/api/screenshot`, `/api/ocr`, `/api/vision/*`, `/api/windows`, ... |

A slow OCR (3–5 s on a busy screen) no longer freezes concurrent screenshot
or vision reads — reads queue behind reads, inputs behind inputs.

### Live-Loop Working Principle

This system is designed for a **live perceive-act loop**, not pre-written
command chains:

1. **READ** — OCR or vision reads the screen before and after every action
2. **ONE ACTION** — each round sends a single command
3. **VERIFY** — acceptance is "it appeared on screen", not "I sent it"
4. **ADAPT** — if verification fails, the next step changes based on what
   is actually seen

This is enforced by the `expect_hwnd` guard: typing is **refused (409)** if
the foreground window doesn't match the target.

---

## Installation

```bash
cd screen-control
pip install -r requirements.txt
```

### Requirements

| Package | Purpose | Required? |
|---|---|---|
| `mss` | Fast screen capture | ✅ Yes |
| `pyautogui` | Mouse/keyboard control | ✅ Yes |
| `pyvda` | Virtual desktop management | ✅ Yes |
| `