soniccraft

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/brand/wordmark-dark.png">
    <img src="assets/brand/wordmark-light.png" alt="SonicCraft" width="420">
  </picture>
</p>

# SonicCraft

**A declarative, deterministic audio engine for AI agents.** Describe a production as a YAML (or JSON)
timeline: who says what, in which order, over which music, at what loudness. SonicCraft compiles it into
TTS speech, music beds and sound effects, mixed with sidechain ducking, mastered to an EBU R128 loudness
target, plus WebVTT/SRT subtitles that match the audio it actually rendered. It ships as a CLI, an MCP
server and an Agent Skill.

[![CI](https://github.com/user-github-me/soniccraft/actions/workflows/ci.yml/badge.svg)](https://github.com/user-github-me/soniccraft/actions/workflows/ci.yml)
[![License: MIT OR Apache-2.0](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue.svg)](#license)
[![crates.io](https://img.shields.io/crates/v/soniccraft.svg)](https://crates.io/crates/soniccraft)

## Use it from an AI agent

Give an agent (Claude Code, Codex, Cursor, Gemini CLI, ...) the link to this repository and ask for audio, for
example *"Use https://github.com/user-github-me/soniccraft to make a 30-second voiceover with background music
for my product video."* Everything it needs is below and in [AGENTS.md](AGENTS.md).

1. **Install the CLI** (prebuilt and checksum-verified, no Rust needed). Homebrew, Scoop, Docker and Cargo work
   too: see [Quickstart](#quickstart).

   ```bash
   curl -fsSL https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.sh | sh   # macOS, Linux
   ```

   ```powershell
   irm https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.ps1 | iex        # Windows
   ```

2. **Write a timeline.** `soniccraft init my-audio` starts one that renders offline: macOS voices on a Mac,
   elsewhere a mock test voice (not real speech). Real voices anywhere: Kokoro or Piper (free, local) or an
   ElevenLabs, OpenAI or Cartesia key, see [Providers](#providers). `soniccraft init --list` shows the
   templates, `soniccraft providers` the voices usable here, and `soniccraft schema` the JSON Schema; the
   [Agent Skill](skills/soniccraft/SKILL.md) explains the format.
3. **Check it, see the cost, render it:**

   ```bash
   soniccraft validate my-audio/timeline.yaml           # errors with a code, line, column and fix hint
   soniccraft plan my-audio/timeline.yaml               # free dry run: timing, ducking, characters to synthesize
   soniccraft render my-audio/timeline.yaml --preview   # mastered audio, subtitles, manifest, preview page
   ```

To keep SonicCraft in an agent for good, add the skill and the MCP server. Both use the CLI from step 1; with
Node.js 20 or newer the MCP server also runs with nothing installed: use `npx -y soniccraft-mcp` in place of
`soniccraft mcp` below.

| Agent | Skill | MCP server |
|---|---|---|
| Claude Code | `claude plugin marketplace add user-github-me/soniccraft`, then `claude plugin install soniccraft@soniccraft` | included in the plugin, or `claude mcp add soniccraft -- soniccraft mcp` |
| Codex | `npx skills add user-github-me/soniccraft -g -a codex` | `codex mcp add soniccraft -- soniccraft mcp` |
| Cursor | `npx skills add user-github-me/soniccraft -g -a cursor` | `{"mcpServers": {"soniccraft": {"command": "soniccraft", "args": ["mcp"]}}}` in `.cursor/mcp.json` |
| Gemini CLI and others | `npx skills add user-github-me/soniccraft -g` (choose the agent) | the same `mcpServers` entry (Gemini CLI: `~/.gemini/settings.json`) |

The server is also in the [official MCP Registry](https://registry.modelcontextprotocol.io) as
`io.github.user-github-me/soniccraft` (container image). Details: [MCP server](#mcp-server) and
[Agent Skill](#agent-skill).

## Why

Ask an agent for a two-voice explainer with background music and it will usually hand-write an ffmpeg
filter graph: `adelay` offsets computed from guessed speech lengths, an `amix`, perhaps a half-remembered
`sidechaincompress`. That fails in predictable ways:

- **Timing drift and overlaps.** Synthesized speech is never exactly as long as the agent guessed, so lines
  collide or leave dead air, and every script edit shifts every later offset.
- **No ducking.** Sidechain compression is fiddly to get right in a filter graph, so the music either
  drowns the voice or sits uniformly too low.
- **Loudness disparity.** Different providers and voices come out at different levels. Nothing measures
  integrated loudness or true peak, so the result is uneven and can clip.
- **Drifting subtitles.** Captions are written from estimated times instead of from the audio that was
  rendered.
- **Opaque, expensive iteration.** A broken graph is a cryptic ffmpeg error, and every retry re-bills every
  TTS call.

## How SonicCraft works

[HyperFrames](https://github.com/heygen-com/hyperframes) showed that agents make better video when they
write declarative HTML and a deterministic renderer does the frame-accurate work. SonicCraft applies the
same idea to audio. The agent writes **intent as data**; a **compiler** validates it, measures the real
durations, resolves all timing in integer sample frames, and renders natively in Rust. The same timeline,
TTS cache and binary always produce byte-identical audio, subtitles and manifests.

```yaml
version: "1.0"
output:
  format: "mp3"
  sample_rate: 48000
  bitrate: "320k"
  target_lufs: -16.0 # integrated loudness target, measured per EBU R128 / ITU-R BS.1770
  export_subtitles: true # Emits matching .vtt and .srt

tracks:
  - id: ambient_bed
    type: "music"
    source: "assets/audio/synthwave_loop.wav"
    loop: true
    volume: 0.65
    fade_in_ms: 1200
    fade_out_ms: 2500
    ducking:
      trigger_tracks: ["voice_primary", "voice_secondary"]
      reduction_db: -18.0
      attack_ms: 80
      release_ms: 350

  - id: voice_primary
    type: "speech"
    pan: -0.15 # Subtle left stereo placement
    sequence:
      - id: cue_01
        speaker: "narrator"
        provider: "elevenlabs"
        voice_id: "21m00Tcm4TlvDq8ikWAM"
        text: "Autonomous software development requires deterministic compilation targets."
      - pause_duration: 350ms
      - id: cue_03
        speaker: "narrator"
        provider: "elevenlabs"
        voice_id: "21m00Tcm4TlvDq8ikWAM"
        text: "By separating high-level planning from digital signal processing, systems eliminate runtime hallucinations."

  - id: voice_secondary
    type: "speech"
    pan: 0.15 # Subtle right stereo placement
    sequence:
      - id: cue_02
        start_relative:
          after_cue: "cue_01"
          offset_ms: 150
        speaker: "guest"
        provider: "local_kokoro"
        voice_id: "am_adam"
        text: "Exactly. SonicCraft treats audio composition the same way Hyperframes treated web-based video."

  - id: sfx_impact
    type: "sfx"
    source: "assets/sfx/whoosh.wav"
    volume: 0.8
    start_at_ms: 0
```

This is [`examples/tech-briefing/timeline.yaml`](examples/tech-briefing/timeline.yaml). Here is what
`soniccraft render` does with it:

1. **Validates everything at once.** Unknown keys, misspelled enum values, dangling anchors, missing
   voices, missing files and missing API keys are all reported together, each with a stable code, a path,
   a line and column, and a fix hint.
2. **Synthesizes and caches.** `cue_01` and `cue_03` go to ElevenLabs (with word timestamps); `cue_02` goes
   to a local Kokoro-FastAPI server (`local_kokoro` is an alias of the built-in `kokoro` provider). Each
   result is cached by a content hash, so later renders only pay for lines that changed.
3. **Levels the voices.** Each cue is trimmed of leading and trailing silence and normalized to -20 LUFS,
   so the cloud voice and the local voice sit at the same level.
4. **Lays out the timeline in sample frames.** `cue_01` starts at 0. `cue_02` starts 150 ms after `cue_01`
   ends. `cue_03` is sequential, 350 ms after `cue_01`, but that slot collides with `cue_02`, so the default
   overlap policy (`push`) **automatically moves `cue_03` to 350 ms after `cue_02` ends**, keeping its pause
   as the gap, and reports an info `cue_pushed`. The result is a natural three-turn exchange with no timing
   arithmetic by the agent.
5. **Places the music.** The bed is a looping clip without a length, so it runs to the end of the timeline
   with a 1.2 s fade-in; the timeline ends 2.5 s after the last word (the bed's `fade_out_ms`) and the bed
   fades out over exactly that tail. The whoosh fires at 0.
6. **Ducks the bed** by 18 dB whenever either voice speaks. Detection runs on the voices' own signal with
   an 80 ms lookahead (it defaults to the attack time), so the bed is already down when the first syllable
   lands; it recovers over 350 ms, and the default 250 ms hold bridges the 150 ms turn gap.
7. **Pans** the two voices to -0.15 and +0.15 (constant power).
8. **Masters** the mix to -16 LUFS integrated with a -1.0 dBTP true-peak ceiling (from the default
   `podcast` preset), iterating loudness gain and a 4× oversampled true-peak limiter until within 0.2 LU
   (the closest pass is kept and `loudness_target_missed` says by how much when no pass gets there; an
   unreachable target gives the loudest master within 6 dB of average limiting, whatever the target).
9. **Encodes** a 48 kHz, 320 kbps MP3 natively, decodes and measures the delivered file (re-limiting if the
   codec pushed a peak over the ceiling), and writes `timeline.vtt` and `timeline.srt` from the placed word
   timings, labelled `narrator` and `guest`.

Before spending anything, `soniccraft plan` (alias `dry-run`) shows the same layout from estimated
durations without calling a TTS API (abridged JSON, estimated numbers):

```json
{ "valid": true, "duration_ms": 20616.9, "estimated": true,
  "items": [
    { "id": "cue_01", "kind": "cue", "start_ms": 0.0,     "end_ms": 4838.7,  "duration_source": "e