ultranix-mcp

# ultranix-mcp

[![Version](https://img.shields.io/badge/version-1.4.0-blue.svg)](Cargo.toml)
[![License: ISC](https://img.shields.io/badge/License-ISC-yellow.svg)](LICENSE)
[![Platform](https://img.shields.io/badge/platform-Linux%20%28Wayland%20%2B%20Hyprland%29-lightgrey.svg)](https://hyprland.org/)
[![Rust](https://img.shields.io/badge/rust-2024-orange.svg)](https://www.rust-lang.org/)
[![MCP SDK](https://img.shields.io/badge/MCP%20SDK-rmcp-purple.svg)](https://github.com/modelcontextprotocol/rust-sdk)
[![Status](https://img.shields.io/badge/status-1.4.0%20implemented-brightgreen.svg)](ROADMAP.md)

**ultranix-mcp is the enterprise-grade, secure Linux desktop-automation layer
for AI agents.**It gives Model Context Protocol (MCP) clients - Claude
Desktop, Claude Code, Cursor, Windsurf, and any MCP-enabled assistant - the
ability to see, click, type, and drive a Wayland desktop: mouse, keyboard,
screenshots, OCR, icon finding, window management, and accessibility-tree
inspection.

ultranix-mcp is **Wayland-native by design**: compositor protocols first,
`uinput`/`evdev` second, XDG Desktop Portals last - behind the same
governance-and-trust surface as its siblings (ultramac on macOS, ultrawin on
Windows): audit logging, rate limiting, input sanitization, and
AES-256-GCM-encrypted action history. It is the first Linux desktop MCP to
combine a cross-compositor fallback ladder, a full governance surface, and a
tri-OS sibling contract - organisations can let agents control a Linux
desktop without giving up control themselves.

> **Status:**v1.4.0 implemented. Phases 0-5 of
> [ROADMAP.md](ROADMAP.md) have shipped, plus the v1.1.0 wave (layer-shell
> overlay, X11-native providers, PipeWire portal capture, opt-in Sentry,
> OCR cache, additional metrics), the v1.2.0 breadth wave (clipboard
> tools, plugin tool-macros, `screen_record`, sway/Wayfire/river/KDE/GNOME
> session detection, per-backend cargo features, framed history v2), the
> v1.3.0 policy wave (runtime access-control policy with per-key
> roles, `--readonly`/`--allow-tools`/`--deny-tools`, per-backend
> invocation metrics, optional HMAC-signed audit lines), and the v1.4.0
> reach wave (Wayfire/river/GNOME window rungs, live `screen_stream`
> capture, plugin-exposed dynamic tools, OCI image + `-bin` package) - see
> [CHANGELOG.md](CHANGELOG.md) for per-release notes. The verified target
> environment is **CachyOS (Arch) + Hyprland on Wayland**, PipeWire,
> `xdg-desktop-portal-hyprland`, and a live AT-SPI2 bus, on Rust 1.98.1.

---

## Features

- **Precision Mouse Control**- click, double-click, drag, scroll,
  button-state control, position queries, and smooth path movement via the
  `zwlr_virtual_pointer_v1` protocol. No root, no helper daemons.
- **Advanced Keyboard Input**- type text and drive key states through
  `virtual-keyboard-unstable-v1`, with full modifier and keymap handling.
- **Intelligent Vision**- in-process `wlr-screencopy-unstable-v1` capture,
  ONNX Runtime OCR (`ort` crate), and OWL-ViT icon finding. Region
  screenshots, color sampling (`color_at`), and session spatial focus
  (`set_spatial_focus` scopes `screenshot`/`find_text_on_screen`/`find_icon`),
  and real `screen_highlight` overlays via `zwlr_layer_shell_v1`
  (translucent, click-through; `-32010 ProviderUnavailable` on
  compositors/sessions without layer-shell - see
  [docs/TOOLS.md](docs/TOOLS.md)).
- **Window Management**- list, focus, move, resize, close, and inspect
  windows through Hyprland's `hyprctl` IPC socket (`hyprctl -j` JSON:
  `clients`, `activewindow`, `dispatch`, `workspaces`), sway's own IPC
  protocol on `$SWAYSOCK`, Wayfire's `ipc`/`ipc-rules` plugins on
  `$WAYFIRE_SOCKET`, the pinned `riverctl` subprocess on river
  (focused-view-only rung - river has no window-list IPC, so
  `get_windows`/`get_active_window` return `isError` results while
  `window_control` drives the focused view), `kdotool` on KDE, the
  "Window Calls" Shell extension on GNOME (when installed), or `wmctrl`
  on X11 sessions.
- **UI Inspection**- full accessibility-tree access over AT-SPI2
  (`atspi` crate): UI-tree dumps, focused-element queries, element search,
  and wait-for-element synchronization.
- **Clipboard & Plugins**- `clipboard_get`/`clipboard_set`/
  `clipboard_clear` over `wl-clipboard` (Wayland) or `xclip`/`xsel` (X11),
  writes consent-gated; and declarative plugin tool-macros -
  `~/.ultranix-mcp/plugins/*.json` manifests of catalog-tool steps run via
  `plugin_list`/`plugin_run`/`plugin_reload`, each step re-entering the
  secured dispatch path. A manifest `tool` section (v1.4.0) registers the
  plugin as a first-class `tools/list` entry with a generated
  `inputSchema`, dispatched through the same secured `plugin_run`
  pipeline.
- **Screen Recording & Live Capture**- `screen_record` captures a frame
  every `interval_ms` for up to `duration_ms` into a fresh `rec-<ulid>`
  dir plus a `manifest.json` (hard caps: 600 frames, 512 MiB);
  `screen_stream` (v1.4.0) runs a continuous `start`/`status`/`latest`/
  `stop` rolling-window capture under `stream-<ulid>` (≤1800 frames,
  ≤512 MiB, oldest evicted) with `latest` polling the newest frame in
  `screenshot`'s image shape.
- **Enterprise Security**- `uxcp_*` API-key auth on HTTP, 10 req/s token
  bucket, input sanitization, command/path whitelists, AES-256-GCM-encrypted
  action history, and JSONL audit logging. See [SECURITY.md](SECURITY.md).

---

## Architecture

ultranix-mcp is a single Rust 2024 binary on the `tokio` runtime, built on
[`rmcp`](https://github.com/modelcontextprotocol/rust-sdk) - the official
Model Context Protocol Rust SDK - with native **stdio**and **streamable-HTTP**
(`:3010`) transports.

The desktop-automation layer is organised as **provider traits behind
dependency injection**(the pattern proven in ultrawin's `src/traits.rs`):
every capability is an `Option<Arc<dyn Trait>>`, so missing compositor
features, absent portals, or headless CI degrade gracefully instead of
failing hard. Mock providers implement the same traits, which keeps the full
tool surface testable without a Wayland session.

| Provider trait | Responsibility | Primary backend |
| --- | --- | --- |
| `CaptureProvider` | Screenshots, region capture, screen info | `wlr-screencopy-unstable-v1` (in-process) |
| `InputProvider` | Pointer, scroll, keyboard events | `zwlr_virtual_pointer_v1` + `virtual-keyboard-unstable-v1` |
| `UIAutomationProvider` | UI tree, focused element, element search | AT-SPI2 via `atspi` |
| `WindowProvider` | Window list/focus/move/close | `hyprctl` IPC socket (new vs. ultrawin) |
| `VisionProvider` | OCR, icon finding | `ort` (ONNX Runtime; CPU, OpenVINO, CUDA, ROCm EPs) |
| `BrowserProvider` | Web queries, DOM access | CDP bridge on `127.0.0.1:9222` |
| `OverlayProvider` | `screen_highlight` overlay | `zwlr_layer_shell_v1` (Wayland-only) |
| `ClipboardProvider` | Clipboard read/write | `wl-copy`/`wl-paste` (Wayland), `xclip`/`xsel` (X11/XWayland) | **Backend priority ladder.**At startup the server detects the session via
`XDG_CURRENT_DESKTOP` plus compositor signatures
(`HYPRLAND_INSTANCE_SIGNATURE`, `SWAYSOCK`, `WAYFIRE_SOCKET`,
`KDE_SESSION_VERSION`, ...) resolving Hyprland, sway, Wayfire, river, KDE,
GNOME, or Other - then binds each provider to the best available backend:

1. **wlroots-native**- in-process Wayland protocols (Hyprland, sway,
   Wayfire, river, and most unknown wlroots compositors; no root)
2. **`grim`/`slurp` (capture) + `uinput`/`evdev` (input)**- whitelisted
   helper binaries and kernel-level input for non-wlroots sessions
3. **XDG Desktop Portal**- `Screenshot` and `RemoteDesktop` over `zbus`
   (universal fallback, subject to portal consent; the *only* route on
   KDE/GNOME Wayland, which implement neither wlr-screencopy nor the
   wlr virtual-input protocols)

Window management rides compositor IPC where it exists: `hyprctl` on
Hyprland, sway's i3-flavoured IPC (`$SWAYSOCK`, shipped at v1.2.0) on
sway, Wayfire's `ipc`/`ipc-rules` plugins (`$WAYFIRE_SOCKET`, v1.4.0) on
Wayfire, `riverctl` on river (v1.4.0, focused-view-only rung - river has
no window-list IPC, so `get_windows`/`get_active_window` report
`ProviderUnavailable` while `window_control` reaches the focused view via
`window:"focused"` with `close` and relative-delta `move`/`resize`),
`kdotool` (KWin
scripting - Wayland and X11
alike) on KDE, the "Window Calls" Shell extension over D-Bus on GNOME
(v1.4.0, extension required - `org.gnome.Shell.Eval` is deliberately
unused), and `wmctrl` on other X11 sessions.

On X11 sessions the X11-native rungs shipped at v1.1.0 resolve instead:
`scrot` capture, `xdotool` input (both still ahead of portal/uinput), and
`wmctrl` window management on non-Hyprland X11.

```mermaid
graph TB
    subgraph "Client Layer"
        AI[AI Assistant / Agent]
        MCP[MCP Client]
    end

    subgraph "Transport Layer"
        STDIO[STDIO<br/>never requires auth]
        HTTP[Streamable HTTP :3010<br/>uxcp_* API key]
    end

    subgraph "Security Layer"
        AUTH[API Key Auth]
        RATE[Rate Limiter<br/>10 req/s token bucket]
        SAN[Input Sanitization<br/>command + path whitelists]
    end

    subgraph "Core (rmcp + tokio)"
        SERVER[ultranix-mcp server]
        TOOLS[40 tools - 6 categories]
    end

    subgraph "Providers - Option&lt;Arc&lt;dyn Trait&gt;&gt;"
        CAP[CaptureProvider]
        INP[InputProvider]
        UIA[UIAutomationProvider]
        WIN[WindowProvider]
        VIS[VisionProvider]
        BRW[BrowserProvider]
        CLIP[ClipboardProvider]
    end

    subgraph "Backends - priority order"
        WLR[wlroots-native<br/>screencopy - virtual-pointer - virtual-keyboard]
        UIN[uinput / evdev]
        PORTAL[XDG Desktop Portal<br/>zbus]
        HYPR[hyprctl IPC]
        SWAY[sway IPC - SWAYSOCK]
        WF[Wayfire IPC - WAYFIRE_SOCKET]
        RIV[riverctl - river focused-view rung]
        GS[gnome-shell - Window Calls ext]
        KDOT[kdotool - KDE]