docker-talkies

# talkies

[![Docker Pulls](https://img.shields.io/docker/pulls/psyb0t/talkies?style=flat-square)](https://hub.docker.com/r/psyb0t/talkies)
[![Docker Hub](https://img.shields.io/docker/v/psyb0t/talkies?sort=semver&label=Docker%20Hub&style=flat-square)](https://hub.docker.com/r/psyb0t/talkies)
[![License: WTFPL](https://img.shields.io/badge/License-WTFPL-brightgreen.svg?style=flat-square)](http://www.wtfpl.net/)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg?style=flat-square)](https://www.python.org/downloads/)

> **Self-hosted, OpenAI-compatible speech server.** 7 ASR backends, 2 TTS engines, voice cloning, MCP — one Docker container, one wire format.

```python
# Drop-in: point your existing OpenAI client at it, change the slug.
from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

c.audio.transcriptions.create(model="whisper-large-v3-turbo", file=open("a.mp3", "rb"))
c.audio.speech.create(model="qwen3-tts-0.6b", voice="alloy", input="hello").stream_to_file("out.mp3")
```

The same client you use against `api.openai.com` works here — only the base URL and the slug change. That's the entire story.

- **6 ASR backends** — Whisper (v3 / turbo), Parakeet-TDT, Canary-180M-Flash / 1B-Flash / Canary-Qwen-2.5B. Whisper-shape response across all of them; long files get sliced via Silero VAD and stitched back.
- **2 TTS engines, 3 backends** — Kokoro-82M (~41 voices across en/es/fr/hi/it/pt, sub-second on CPU) shipped in two flavors (`kokoro-82m` PyTorch and `kokoro-82m-nvidia` ONNX-via-ORT — NVIDIA's TensorRT-friendly export), plus Qwen3-TTS-0.6B (CUDA-only voice cloning).
- **Voice cloning** — drop a 10-30 s reference `.wav` into `/data/custom-voices/<name>.wav`, synth as `voice=<name>`. Nested paths preserved (`clients/acme/jane.wav` → `voice=clients/acme/jane`). Live re-scan, no restart.
- **Hot model swap + idle eviction** — one GPU pool serves both modalities, Ollama-style `/api/ps` for introspection, `DELETE /api/ps/<slug>` to evict.
- **MCP server built in** at `/v1/mcp` — Claude / Cursor / IDE-side LLMs can call transcribe + speak as tools.
- **Stereo diarization** without bolting on a separate model — left channel = speaker L, right = speaker R, chronological `L:` / `R:` turn lines.
- **CPU + CUDA images** — `psyb0t/talkies:latest` (CPU + Kokoro × 2 runtimes + 4 ASR models incl. multilingual Nemotron-3.5-ASR via parakeet.cpp) and `:latest-cuda` (everything, ~11 GB VRAM at full load).

## Quick start

```bash
docker run -d --name talkies \
  -v $HOME/talkies-data:/data \
  -p 8000:8000 \
  psyb0t/talkies:latest

curl -s http://localhost:8000/v1/audio/transcriptions \
  -F "file=@samples/hello.wav" \
  -F "model=whisper-large-v3-turbo" | jq
```

First boot downloads every model in `models.json` into `/data/models/<slug>/` (75 MB-3 GB each — bind-mount `/data` so they survive restarts). The CUDA image's full default set is ~30 GB on disk (the 5 Qwen3-TTS variants alone are ~24 GB). **Use `TALKIES_ENABLED_MODELS` to opt in to only what you actually need** — it whitelists slugs and the prefetch loop only downloads those, and the server only registers those backends:

```bash
# Only one ASR + one TTS — ~5 GB on disk instead of ~30 GB
docker run -d --gpus all \
  -e TALKIES_ENABLED_MODELS=whisper-large-v3-turbo,qwen3-tts-1.7b-custom \
  -v $HOME/talkies-data:/data \
  -p 8000:8000 psyb0t/talkies:latest-cuda
```

Unknown slug in the list → fail-fast at startup with the catalog listed. Empty / unset → every model in `models.json` gets downloaded (the default).

GPU: pull `psyb0t/talkies:latest-cuda` and add `--gpus all`.

<details>
<summary><b>More <code>curl</code> examples</b> — verbose JSON, SRT, stereo diarization, TTS, model management</summary>

```bash
# Verbose JSON — full Whisper shape with per-segment + per-word timestamps.
curl -s http://localhost:8000/v1/audio/transcriptions \
  -F "file=@samples/hello.wav" \
  -F "model=whisper-large-v3-turbo" \
  -F "response_format=verbose_json" \
  -F "timestamp_granularities[]=word" \
  -F "timestamp_granularities[]=segment" | jq

# SRT subtitle output (drop straight into a video player).
curl -s http://localhost:8000/v1/audio/transcriptions \
  -F "file=@samples/lecture.mp3" \
  -F "model=whisper-large-v3" \
  -F "response_format=srt" > lecture.srt

# Stereo diarization — left/right channels become speakers L/R.
curl -s http://localhost:8000/v1/audio/transcriptions \
  -F "file=@samples/interview-stereo.wav" \
  -F "model=whisper-large-v3-turbo" \
  -F "diarization=true" \
  -F "response_format=verbose_json" | jq

# Kokoro TTS — list the shipped voices, then synthesize an MP3.
curl -s http://localhost:8000/v1/audio/voices | jq
curl -s http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"kokoro-82m","input":"Hello from talkies.","voice":"af_heart","response_format":"mp3"}' \
  --output hello.mp3

# Which models are configured, which are loaded, evict one if you want.
curl -s http://localhost:8000/v1/models | jq
curl -s http://localhost:8000/api/ps | jq
curl -s -X DELETE "http://localhost:8000/api/ps/whisper-large-v3-turbo"
curl -s -X POST  http://localhost:8000/unload | jq    # evict everything
```

</details>

<details>
<summary><b>Table of contents</b></summary>

- [Quick start](#quick-start)
- [How it works](#how-it-works)
- [Supported models](#supported-models)
- [What's NOT supported](#whats-not-supported)
- [API — `POST /v1/audio/transcriptions`](#api--post-v1audiotranscriptions)
  - [Request fields](#request-fields)
  - [Response formats](#response-formats)
    - [`json` (default)](#json-default)
    - [`verbose_json`](#verbose_json)
    - [`text`](#text)
    - [`srt`](#srt)
    - [`vtt`](#vtt)
  - [Stereo diarization](#stereo-diarization)
  - [Translation (Canary X→Y)](#translation-canary-xy)
  - [Long files + VAD chunking](#long-files--vad-chunking)
  - [Error contract](#error-contract)
- [API — `POST /v1/audio/speech` (TTS)](#api--post-v1audiospeech-tts)
  - [Request body](#request-body)
  - [Voices (`GET /v1/audio/voices`)](#voices-get-v1audiovoices)
  - [Output formats](#output-formats)
  - [Error contract (TTS)](#error-contract-tts)
- [Resource-management endpoints (Ollama-style)](#resource-management-endpoints-ollama-style)
- [Server-side file staging (`/v1/files`)](#server-side-file-staging-v1files)
- [MCP endpoint (`/v1/mcp`)](#mcp-endpoint-v1mcp)
- [Bearer-token auth](#bearer-token-auth)
- [Configuration (env vars)](#configuration-env-vars)
- [CPU vs CUDA images](#cpu-vs-cuda-images)
- [Architecture](#architecture)
- [Customizing the model registry](#customizing-the-model-registry)
- [Development](#development)
- [Security notes](#security-notes)
- [Credits](#credits)
- [License](#license)

</details>

## How it works

`POST /v1/audio/transcriptions` with a multipart `file` + a `model` slug → text back. `POST /v1/audio/speech` with a JSON body (`model` + `input` + `voice`) → audio bytes back. Same wire shape as OpenAI for both.

Swap the ASR slug — `whisper-large-v3`, `whisper-large-v3-turbo`, `parakeet-tdt-0.6b-v3`, `canary-180m-flash`, `canary-1b-flash`, `canary-qwen-2.5b` — and the transcription contract stays identical. Behind the scenes the request is dispatched to the right backend (faster-whisper for the whisper family, NeMo for everything else), audio is normalized to 16 kHz mono WAV, long files are sliced via Silero VAD into ≤28-second speech regions, results are stitched back into one Whisper-shape timeline. None of that leaks into the wire shape. You just get text.

For TTS there are three slugs:

- `kokoro-82m` (Kokoro-82M, Apache 2.0, ~41 voices across en/es/fr/hi/it/pt) — the fast in-process PyTorch pipeline (via the `kokoro` PyPI lib + `misaki` G2P). Sub-second synthesis on CPU and trivial on GPU.
- `kokoro-82m-nvidia` (nvidia/kokoro-82M-onnx-opt, Apache 2.0, same voice catalog) — same Kokoro weights served via ONNXRuntime against NVIDIA's TensorRT-friendly ONNX export. No PyTorch on the hot path; CUDA EP on the CUDA image, CPU EP on the CPU image. G2P via `espeak-ng` + `phonemizer` (no `misaki` dep). Drop-in for `kokoro-82m` — same `voice` names, same wire format, same defaults.
- `qwen3-tts-0.6b` (Qwen3-TTS-12Hz-0.6B-Base, Apache 2.0, CUDA only) — voice cloning. Bring your own reference `.wav` (10-30 s of clean speech is plenty), drop it into `/data/custom-voices/<your-name>.wav`, and synthesize in that speaker's voice via `voice=<your-name>`. Supports nested paths (`/data/custom-voices/clients/acme/jane.wav` → `voice=clients/acme/jane`). Three sample voices (`alloy`, `echo`, `fable`) ship baked into the image.

Pass `model=<slug>`, an `input` string, and a `voice` from `GET /v1/audio/voices` — the server runs the matching backend's pipeline, encodes the raw PCM into your requested `response_format` (`mp3` / `opus` / `aac` / `flac` / `wav` / `pcm`) via ffmpeg, and streams the bytes back with the matching `Content-Type`.

Need stereo speaker diarization on transcription? Pass `diarization=true` and upload a stereo file — left channel = speaker L, right channel = speaker R, output gets per-segment `channel` tags and the text is split into chronological `L:` / `R:` turn lines. Two-mic setups (interview rigs, podcast splits, dual-track ham recordings) end up with a clean transcript without you having to bolt a separate diarization model onto your stack.

GPU variant (`psyb0t/talkies:latest-cuda` + `--gpus all`) ships everything; the CPU image (`psyb0t/talkies:latest`) ships the four ASR models that actually run reasonably without a GPU (the three Whisper variants + `canary-180m-flash`) plus Kokoro TTS. Parakeet-TDT, Canary-1B-Flash, Canary-Qwen-2.5B, and Qwen3-TTS need VRAM to be anything other than a space heater, so they're CUDA-only. Kokoro is fast enough on CPU that it ships in both images.

## Supported models

Eight ASR models + seven TTS slugs (engine mix: faster-whisper × 2, NeMo (Parakeet TDT + 3× Canary), parakeet.cpp/ggml × 1, Ko