yueying

<!-- mcp-name: io.github.vsh5dvsch7-png/yueying -->
# yueying — let AI watch videos

**Point Claude, Cursor or any MCP client at a video and get back a timestamped transcript plus keyframe contact sheets — offline, no API key.** Local files first; URLs (YouTube, Bilibili, Douyin, Xiaohongshu, TikTok, Vimeo, …) are videos you are entitled to process, fetched via yt-dlp at ≤720p and deleted after processing by default.

[![PyPI](https://img.shields.io/pypi/v/yueying)](https://pypi.org/project/yueying/)
[![PyPI downloads](https://img.shields.io/pypi/dm/yueying)](https://pypi.org/project/yueying/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](pyproject.toml)
[![Add to Cursor](https://img.shields.io/badge/Add_to-Cursor-111111?logo=cursor&logoColor=white)](https://cursor.com/en/install-mcp?name=yueying&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ5dWV5aW5nIiwibWNwIl0sImVudiI6eyJQWVRIT05VVEY4IjoiMSJ9fQ==)
[![Install in VS Code](https://img.shields.io/badge/Install_in-VS_Code-0098FF?logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect?url=vscode%3Amcp%2Finstall%3F%257B%2522name%2522%253A%2522yueying%2522%252C%2522command%2522%253A%2522uvx%2522%252C%2522args%2522%253A%255B%2522yueying%2522%252C%2522mcp%2522%255D%252C%2522env%2522%253A%257B%2522PYTHONUTF8%2522%253A%25221%2522%257D%257D)

[中文说明 ↓](#中文说明)

Yueying (阅影) means "read video". One package gives you an **MCP server**, a **CLI** and an **agent skill**.

## What you get

![A 3x3 contact sheet: nine keyframes, each with a yellow "#number mm:ss" label bottom-left](docs/demo-grid.jpg)

*Contact sheet from a 24-second demo clip (four app screenshots with Chinese narration). The yellow label on every tile is the keyframe number and timestamp; the model cites them back to you.*

The transcript of the same clip — local speech recognition, language auto-detected as Chinese:

```
[00:00] 这是阅读,一个安静的桌面小说阅读器。整本书连续滚动,按段落记住进度。
        第二个画面是桌面模式,窗口变透明,只留文字浮在桌面上。
        第三个画面是伪装皮肤,一键变成代码编辑器。
        最后是伪装成表格的样子。
```

(The app is called 月读; ASR heard the homophone 阅读. Speech recognition does that to names — the model corrects it from the on-screen text in the frames.)

Every video becomes one folder:

```
report.md          index for the model: metadata, chapters, contact sheets, keyframes, transcript
transcript.txt     paragraphs with [mm:ss] timestamps
transcript.srt     subtitles for any player
grid_01.jpg …      3x3 contact sheets, 9 keyframes each, in time order
frames/            full-size keyframes, e.g. f003_00m15s.jpg
manifest.json      machine-readable result (paths, segments, chapters, options)
```

## Why yueying

- **Captions first, Whisper only when needed.** Platform subtitles are used when they exist. Otherwise local [faster-whisper](https://github.com/SYSTRAN/faster-whisper): `large-v3-turbo` on an NVIDIA GPU, `small` on CPU, automatic CPU fallback — nothing is uploaded, no key.
- **ffmpeg bundled.** Works on Windows 11 out of the box (imageio-ffmpeg); no PATH fiddling.
- **Token-efficient.** Keyframes are taken at scene changes, near-duplicates dropped, then packed into 3x3 contact sheets with burned-in timestamps. One sheet ≈ 1–2K tokens for nine moments; one transcript with `[mm:ss]` paragraphs.
- **Chinese platforms and the rest.** Bilibili (multi-part, collections, member videos with your browser login), Douyin, Xiaohongshu — and YouTube, TikTok, Vimeo, X and every other yt-dlp site.
- **Zero API keys, zero telemetry.** The only network traffic is the video site you name and one Whisper model download. See the [privacy policy](#privacy-policy).

Benchmark: a 6-minute Bilibili video → report in ~90 s on an RTX 5060 laptop; on CPU with `model=small` expect ~1–2 min per 10 min of speech.

## Quick start

1. Install [uv](https://docs.astral.sh/uv/) (Python is not required):
   ```bash
   winget install astral-sh.uv                         # Windows
   brew install uv                                     # macOS
   curl -LsSf https://astral.sh/uv/install.sh | sh     # Linux / macOS
   ```
2. Warm up and check everything once (installs the package, probes the GPU, downloads the speech model, runs a 2-second smoke test, prints config to paste):
   ```bash
   uvx yueying mcp --setup
   ```
3. Add the server to your client (below), then ask: *"Watch C:\videos\lecture3.mp4 and turn the steps into notes"* or *"What does this video say about docker compose: https://www.bilibili.com/video/BV…"*.

### Claude Desktop

`%APPDATA%\Claude\claude_desktop_config.json` (Windows) · `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS). Fully quit and reopen Claude afterwards.

```json
{ "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }
```

Windows note: Claude Desktop does not always see your PATH — if the server fails to start ("spawn uvx ENOENT"), use the absolute path, e.g. `"command": "C:\\Users\\<you>\\.local\\bin\\uvx.exe"` (`where uvx` prints it). Logs: `%APPDATA%\Claude\logs\mcp-server-yueying.log` (`~/Library/Logs/Claude/` on macOS). Keep `wait_seconds` at its default there; see [the RUNNING rule](#the-running-rule).

### Claude Code

```bash
claude mcp add --transport stdio --scope user yueying --env PYTHONUTF8=1 -- uvx yueying mcp
```

Or drop this repo's [`.mcp.json`](.mcp.json) into a project (it ships with `"timeout": 1800000` so one `watch_video` call can wait for a long video). To raise Claude Code's tool timeout globally, set `MCP_TOOL_TIMEOUT=1800000` (ms) in your environment. The repo is also a Claude Code plugin (`.claude-plugin/plugin.json`: server + skill).

### Cursor

Click the **Add to Cursor** badge above, or put the same JSON in `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project):

```json
{ "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }
```

### Cline

MCP Servers → Configure (`cline_mcp_settings.json`). `timeout` is in seconds; the five read-only tools are safe to auto-approve. Step-by-step agent instructions: [llms-install.md](llms-install.md).

```json
{
  "mcpServers": {
    "yueying": {
      "type": "stdio",
      "command": "uvx",
      "args": ["yueying", "mcp"],
      "env": { "PYTHONUTF8": "1" },
      "timeout": 1800,
      "autoApprove": ["get_transcript", "search_transcript", "get_frames", "get_frame_at", "list_videos"]
    }
  }
}
```

### Windsurf

`~/.codeium/windsurf/mcp_config.json`:

```json
{ "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }
```

### VS Code (Copilot agent mode)

Click the **Install in VS Code** badge above, or create `.vscode/mcp.json` (note the root key `servers`):

```json
{ "servers": { "yueying": { "type": "stdio", "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }
```

Direct links for hosts that accept custom URL schemes: `cursor://anysphere.cursor-deeplink/mcp/install?name=yueying&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ5dWV5aW5nIiwibWNwIl0sImVudiI6eyJQWVRIT05VVEY4IjoiMSJ9fQ==` and `vscode:mcp/install?%7B%22name%22%3A%22yueying%22%2C%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22yueying%22%2C%22mcp%22%5D%2C%22env%22%3A%7B%22PYTHONUTF8%22%3A%221%22%7D%7D`.

### Without uv (pip / pipx) and Windows one-click

```bash
pip install yueying            # or: pipx install yueying
yueying mcp --setup            # prints a config with the absolute path of the yueying-mcp executable
```

Use that absolute path as `"command"` with no `args` (Windows: `...\Scripts\yueying-mcp.exe`; also works as `python -m yueying mcp`). Windows users without Python tooling can double-click [`install.cmd`](install.cmd) from a checkout: it creates `%LOCALAPPDATA%\yueying\venv`, installs the Claude Code skill, runs `yueying mcp --setup` and prints the JSON block with the right path.

### GPU

```bash
uvx --from "yueying[cuda]" yueying mcp     # NVIDIA: adds the CUDA runtime wheels (cuBLAS, cuDNN)
pip install "yueying[cuda]"
```

Device and model are chosen automatically (`model=auto`: large-v3-turbo on CUDA, small on CPU); if the GPU trial fails, recognition falls back to CPU by itself.

### Docker

```bash
docker build -t yueying .
docker run --rm -i -v yueying-data:/data -v "$PWD/videos:/videos:ro" yueying
```

The image is CPU-only (containers get no GPU by default), so it defaults to the `small` model.
Mount your videos read-only and give the tools container paths (`/videos/lesson.mp4`); results and
the downloaded Whisper weights live in the `/data` volume. In a client config the `command` is
`docker` and `args` are `["run", "--rm", "-i", "-v", "yueying-data:/data", "-v", "/your/videos:/videos:ro", "yueying"]`.

## Tools

| Tool | When the agent uses it | What it returns | Limits |
|---|---|---|---|
| `watch_video(video, mode="full", language="auto", model="auto", frame_interval_seconds=None, cookies_from_browser=None, output_dir=None, refresh=False, wait_seconds=45, max_chars=12000)` | First call for any video: an absolute local path or a URL. `mode`: `full` (transcript + keyframes), `transcript`, `frames`. | `DONE` overview: title, source, duration, text source, folder, files, chapters, contact-sheet ranges, transcript in `[mm:ss]` paragraphs — or `RUNNING` with stage/percent/ETA, or `ERROR` with a plain-English hint. | Blocks up to `wait_seconds` (0–1500). Transcript truncated at `max_chars` with a `get_transcript` start time. Cached per video; `refresh=true` reprocesses. |
| `get_transcript(video, start="0", end=None, format="paragraphs", max_chars=8000)` | The overview was truncated, a specific time range, or exporting subtitles (`format="srt"`). | Header + `[mm:ss]` paragraphs / `[mm:ss-mm:ss]` segments / SRT blocks; `TRUNCATED — next_start="…"` when cut. | `max_chars` 1000–100000. Times: seconds, `mm:ss`, `h:mm:ss`. |
| `search_transcript(video, query, context_second