open-gui-remote-control

# OpenGUI Remote Control

Use this skill when an AI coding agent such as Claude Code, Codex, or OpenCode needs to operate an Android phone through OpenGUI.

The agent should not drive the phone with raw `adb shell input` commands. The supported path is:

```text
Claude Code / Codex / OpenCode
  -> server CLI or REST API
  -> OpenGUI backend task/execution services
  -> standby dispatch
  -> Android client
  -> execution socket action loop
```

## Trigger Guidance

Start this skill when the user asks for any of these:

- "Use OpenGUI to control my phone"
- "让 Codex 操控手机"
- "Use OpenCode to control my phone through OpenGUI"
- "Run this task on the Android device"
- "Use the OpenGUI CLI"
- "List OpenGUI devices"
- "Check / pause / resume / cancel an OpenGUI execution"
- "让 Claude Code 通过 OpenGUI 跑手机任务"

If the user asks to install or bootstrap OpenGUI from scratch, use `open-gui-bootstrap` first. After backend and Android client are running, return to this skill for task execution.

## Core Rules

- First obtain or locate a runnable OpenGUI checkout. The CLI cannot work without the repository.
- Use the repository CLI first: `cd server && pnpm opengui -- ...`.
- Use `--json` whenever the result will be parsed by the coding agent.
- Do not ask the user to run terminal commands that the agent can run.
- Ask the user only for physical phone actions, Android permissions, or missing secrets.
- Do not change Android socket event names or payloads.
- Do not use IM commands for this workflow.
- Do not bypass OpenGUI with coordinate-only `adb shell input` scripts.
- Treat `devices` as standby presence only; execution can still fail because of model config, Android permissions, app state, or device lifecycle.

## Repository Source

OpenGUI's runnable source checkout is:

```text
https://github.com/Core-Mate/OpenGUI
```

The checkout must contain both:

- `server/package.json`
- `client/start.sh`

If those paths are missing, the current directory is not the runnable OpenGUI checkout.

## Local Or Remote Workspace

Before using this skill, decide where OpenGUI should run.

Use a local workspace when:

- the Android phone is connected to the same machine by USB
- `adb reverse tcp:7777 tcp:7777` should be used
- the agent has terminal access to the developer machine

Use a remote workspace only when:

- the user explicitly asks to run OpenGUI on a remote host
- the remote host can reach the Android device or a device bridge
- the backend URL used by the Android client is reachable from the phone

Do not silently choose a remote host for phone control. USB debugging, `adb reverse`, Android build/install, and phone-side permissions are usually local-machine operations.

## Checkout Acquisition

If the current directory already contains the runnable checkout, use it.

If the current directory is a wrapper directory, search one level down for the runnable checkout before cloning:

```bash
find . -maxdepth 3 -type f -path '*/server/package.json' -print
find . -maxdepth 3 -type f -path '*/client/start.sh' -print
```

If no runnable checkout exists and the user wants the agent to set it up locally, clone the public repository:

```bash
git clone https://github.com/Core-Mate/OpenGUI.git
cd OpenGUI
```

If the destination already exists, do not overwrite it. Enter the existing directory, inspect `git status`, and pull only when the user asked for the latest code or when the checkout is clean enough to update safely.

If the user wants a remote setup, SSH to the remote host first, then perform the same checkout detection or clone on that host. Keep the backend URL and Android connectivity explicit; a backend running on a remote host will not be reachable through local `adb reverse` unless the user has provided a bridge.

## Preconditions

Before sending a task, verify these conditions:

- backend is reachable at `http://localhost:7777` unless the user gave another base URL
- Android client is installed and open
- `adb reverse tcp:7777 tcp:7777` has been applied for a USB-connected phone
- phone-side USB debugging is approved
- Accessibility Service is enabled
- overlay permission is enabled when needed
- at least one standby device is online

If the backend or client is not running, use the repository scripts:

```bash
cd server
./start.sh
```

```bash
cd client
./start.sh
```

## CLI Reference

Run commands from the `server/` directory.

List online standby devices:

```bash
pnpm opengui -- devices --json
```

Create and run a new task:

```bash
pnpm opengui -- do "观察当前手机屏幕，简要描述你看到了什么，然后结束" --json
```

Run a task on a specific device:

```bash
pnpm opengui -- do "打开设置，检查当前网络状态" --device <deviceId> --json
```

Run an existing task:

```bash
pnpm opengui -- run <taskId> --json
```

Check execution status:

```bash
pnpm opengui -- status <executionId> --json
```

Pause, resume, or cancel:

```bash
pnpm opengui -- pause <executionId> --json
pnpm opengui -- resume <executionId> "继续执行，但不要打开新的 App" --json
pnpm opengui -- cancel <executionId> --json
```

Use a non-default backend:

```bash
pnpm opengui -- devices --base-url http://localhost:7777 --json
```

Base URL priority:

```text
--base-url > OPENGUI_BASE_URL > http://localhost:7777
```

## Standard Workflow

### 1. Obtain the runnable checkout

Find or clone `https://github.com/Core-Mate/OpenGUI`.

Then work from the repository root that contains both:

- `server/package.json`
- `client/start.sh`

If the current directory is a wrapper repo, find the nested runnable checkout before running commands. If no runnable checkout exists, clone it or ask for the intended repository location.

### 2. Verify backend

Check the backend before task dispatch:

```bash
curl -fsS http://localhost:7777/docs >/dev/null
```

If this fails, start the backend:

```bash
cd server
./start.sh
```

If `start.sh` creates `.env` and exits, ask only for the missing model keys required for execution, then run it again.

### 3. Verify Android client

Prefer the repo script:

```bash
cd client