clipswarm

# clipswarm

[![CI](https://github.com/Shoberman2/clipswarm/actions/workflows/ci.yml/badge.svg)](https://github.com/Shoberman2/clipswarm/actions/workflows/ci.yml) [![npm](https://img.shields.io/npm/v/clipswarm)](https://www.npmjs.com/package/clipswarm) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

**Open-source, agent-native video clipping.** Point any number of AI agents at YouTube videos *or live streams*, and get ready-to-post viral shorts back: 9:16, a hook header, and word-by-word captions. Runs on your machine, with no API keys and no subscription.

<p align="center"><img src="docs/demo.gif" alt="A clipswarm viral clip: a white header card reading 'Astronauts celebrate July 4th from 250 miles up' above the full video of three astronauts on the ISS, with word-by-word captions below" width="300"></p>

<p align="center"><sub>Made with one command: <code>clipswarm clip https://youtu.be/75-H-i9ctkE 6 14.4 --viral --title "Astronauts celebrate July 4th from 250 miles up"</code><br>Footage: NASA (public domain). No endorsement implied.</sub></p>

```
"Get me the 3 best moments from each of these 10 podcasts as shorts"
        │
        ├─ clipper agent 1 ─┐
        ├─ clipper agent 2 ─┼─► clipswarm MCP ─► yt-dlp ─► ffmpeg ─► clips/*_viral.mp4
        └─ clipper agent N ─┘     shared queue, cached transcripts, retries
```

## Why clipswarm

OpusClip-style apps decide for you what's "viral". clipswarm gives **your agent** the tools to decide, then does the production work:

- **Viral format built in**: 1080×1920, a hook header card, the **whole** video frame (never cropped or zoomed), and animated captions that highlight each word as it's spoken. The layout adapts to the source's shape.
- **YouTube videos *and* live streams**: clip what just happened on a live stream (`start: -60, end: "now"`), or what aired at 8:41pm.
- **Captions from real word timings**: YouTube's auto-captions for regular videos, and local [whisper.cpp](https://github.com/ggml-org/whisper.cpp) for live streams. Free and offline.
- **Runs on autopilot**: give it a category and a schedule ("netflix stock, every 6h"), and it finds new videos, has AI pick the moments and write the hooks, and renders the clips in the background.
- **Built for swarms**: dozens of parallel jobs across many videos and many simultaneous agents, all through one shared queue. Transcripts are fetched once per video, and one bad job never sinks a batch.
- **Fast and frugal**: downloads only the seconds you clip (not the whole 3-hour stream), and uses hardware encoding on macOS.
- **Works with any ffmpeg**: text is rendered by clipswarm itself, so stock builds without `drawtext`/`libass` (e.g. Homebrew's) work fine.

## Install

**1. Dependencies** (Node 20+):

```bash
brew install yt-dlp ffmpeg whisper-cpp     # macOS
# Linux: pip install -U yt-dlp && sudo apt install ffmpeg   (+ whisper.cpp for live captions)
npx -y clipswarm setup                     # checks everything; downloads the caption model (~140MB)
```

`whisper-cpp` is optional. Without it, everything works, but live-stream clips render without captions.

> Keep yt-dlp up to date (`yt-dlp -U` / `brew upgrade yt-dlp`). YouTube regularly breaks older versions. This is the #1 cause of failures.

**2. Add it to your AI app:**

| App | How |
|---|---|
| **Claude Code** (recommended, includes the `clipper` agent) | `/plugin marketplace add Shoberman2/clipswarm` then `/plugin install clipswarm@clipswarm` |
| Claude Code (MCP only) | `claude mcp add clipswarm -- npx -y clipswarm mcp --out ./clips` |
| **Claude Desktop** | Settings → Developer → Edit Config, add the JSON below |
| **ChatGPT desktop app / Codex** | `codex mcp add clipswarm -- npx -y clipswarm mcp` (shared by Codex CLI, the IDE extension and the ChatGPT desktop app) |
| Cursor | `~/.cursor/mcp.json`, JSON below |
| VS Code | `code --add-mcp '{"name":"clipswarm","command":"npx","args":["-y","clipswarm","mcp"]}'` |
| Gemini CLI | `gemini mcp add clipswarm npx -y clipswarm mcp` |

```json
{ "mcpServers": { "clipswarm": { "command": "npx", "args": ["-y", "clipswarm", "mcp", "--out", "/Users/you/clips"] } } }
```

> **Why not claude.ai or ChatGPT on the web?** Those only connect to *hosted* servers. Hosting a YouTube downloader means datacenter IP blocks, Terms of Service exposure, and shipping 50MB videos through a chat window. clipswarm runs locally, where all three problems disappear.

## Use it

In Claude Code with the plugin installed:

> Spawn a clipper agent for each of these videos in parallel and get me the 3 strongest moments from each as viral shorts: <url1> <url2> <url3>

> Clip the last 60 seconds of https://www.youtube.com/@aljazeeraenglish/live as a viral short.

Each `clipper` agent reads its video's transcript, picks self-contained moments, writes a hook for each, and calls `create_clips`. They all run at once.

## The viral format

| Option | Default | |
|---|---|---|
| `style` | `"plain"` | `"viral"` for the 9:16 format |
| `title` | video title | The header hook. Agents should write a punchy one (≤10 words); `""` for no header |
| `background` | `"#000000"` | Colour behind the video (or `"blur"`) |
| `captions` | `true` | Word-by-word captions, 1–3 words at a time, current word highlighted |
| `accent` | `"#FFE600"` | Highlight colour |

**The full frame is always shown.** clipswarm never crops or zooms, because what matters in a video (a spreadsheet, a chart, a second speaker) can be anywhere in the frame. The layout follows the source:

| Source | Layout |
|---|---|
| Widescreen (16:9, screen recordings, podcasts) | Full width, header above, captions below, centred on black |
| Vertical (9:16 Shorts, phone video) | Fills the frame; header and captions sit over the video |
| Square / 4:3 / anything else | Scaled as large as fits, nothing cut off |

Captions come from YouTube's auto-captions (exact per-word timing) for regular videos, and from local whisper.cpp for live streams or videos without captions. If neither is available, the clip still renders and the result includes a `warnings` entry explaining why.

## Live streams

Paste a link to a stream that's live right now (a `watch?v=` or a channel's `/live` link). Times are relative to the live edge:

| You want | `start` | `end` |
|---|---|---|
| The last 30 seconds | `-30` | `"now"` |
| 2 minutes ago, 20s long | `"-2:00"` | `"-1:40"` |
| What aired at a specific moment | `"2026-10-08T00:51:00Z"` | `"2026-10-08T00:51:30Z"` |

- About the **last hour** is available (YouTube's DVR window).
- Plain live clips are stream-copied for speed and start on the previous keyframe (≤5s early). `live.from` / `live.to` in the result give the exact wall-clock range. Use `precise: true` for frame-accurate cuts. Viral clips are always frame-accurate.
- `end` can't be in the future. Retry once it has aired.

## Watches: clips that find themselves

Give clipswarm a category and a schedule. It searches YouTube for new videos, picks the best moments, and renders them as viral clips while you're away. A Mac notification appears when new clips land.

```bash
clipswarm watch add "netflix stock analysis" --every 6h          # what to look for, how often
clipswarm watch install                                          # background job (launchd on macOS)
clipswarm watch run --force                                      # or run it right now
clipswarm watch inbox                                            # what it found and clipped
```

Or just ask your agent: *"Watch for new AI agent news videos every 12 hours and clip the best moments."* (MCP tools: `create_watch`, `list_watches`, `run_watch`, `get_inbox`, `update_inbox_item`, `delete_watch`, `watch_scheduler`.)

**How it picks moments**, best first:

1. **AI.** If the [Claude Code](https://claude.com/claude-code) CLI is installed, Claude reads the transcript, picks self-contained moments and writes a hook for each header (e.g. *"Netflix at $67, but this model says $127"*). This runs on your own Claude account.
2. **Most replayed.** YouTube's replay peaks, on videos old enough to have them (usually several days).
3. **Transcript heuristic.** Free and offline: it scores stretches for questions, numbers, strong claims and energy.

If YouTube's captions are missing or rate-limited, the watcher downloads just the audio and transcribes it with whisper.cpp. Anything it can't clip lands in the inbox as `needs_agent` for your agent to handle.

| Option | Default | |
|---|---|---|
| `--every` | `6h` | `30m`, `6h`, `1d`… (min 5m) |
| `--max-age` | `72h` | Only videos uploaded within this window |
| `--min-views` | none | Skip small videos |
| `--videos` | `3` | New videos per run |
| `--clips` / `--seconds` | `2` / `45` | Clips per video, target length |
| `--picker` | `auto` | `ai`, `heatmap` or `transcript` to force one |
| `--live` | off | Also list streams that are live right now in the inbox |
| `--no-auto` | off | Only collect videos; don't clip |
| `--out` | `~/clipswarm/<id>` | Where clips go |

State lives in `~/.clipswarm/` (`CLIPSWARM_HOME` to move it). Each video is only processed once. On Linux, `watch install` prints a crontab line, and `clipswarm watch daemon` runs a foreground loop anywhere.

## How it works

1. **Find the moment.** The agent calls `get_video_info`, `search_transcript` and `get_transcript`. Transcripts come from YouTube's captions via yt-dlp and are cached.
2. **Cut it.**
   - **Regular videos:** yt-dlp downloads only the requested section.
   - **Live streams:** clipswarm reads YouTube's rolling HLS playlist (5-second segments with wall-clock stamps) and fetches just the segments covering your range.
3. **Make it viral.** clipswarm gets word timings and draws the header and caption frames (font → vector paths → PNG). ffmpeg composites them with the full, uncropped video on a 9:16 canvas.
4. **Share the machine.** Every job from every agent goes through one concurrency limiter, with retries for YouTube's i