cochlea

# cochlea

[![CI](https://github.com/richer-richard/cochlea/actions/workflows/ci.yml/badge.svg)](https://github.com/richer-richard/cochlea/actions/workflows/ci.yml)
[![docs](https://img.shields.io/badge/docs-book-blue)](https://richer-richard.github.io/cochlea/)

**A headless audio engine for agents.** Write a score as data, render it
offline to deterministic PCM, then *listen through numbers* — loudness,
onsets, pitch, key, spectrograms — and assert what you heard. Compose →
render → probe → verify, with no human ear (and no audio device) in the
loop.

![Mel spectrogram of first_light.ron: six note onsets followed by a reverb tail decaying to silence](docs/assets/first_light_spectro.png)

*What the agent sees: the mel spectrogram of `examples/scores/first_light.ron`
— the score used in the example below — after render and probe. No PCM in
sight.*

```rust
use cochlea_score::*;

let score = Score::new(SampleRate(48_000), Ppq(960))
    .time_signature(4, 4)
    .tempo(Ticks(0), Bpm(120.0))
    .track("lead", Instrument::preset("saw_lead"))
    .note("lead", bar(1).beat(1), Dur::quarter(), Pitch::A4, Vel(96))
    .automate("lead", Param::CUTOFF_HZ,
        keys![(bar(1), 400.0, ease_in_out()), (bar(3), 4_000.0)]);

let rendered = cochlea_render::render(&score)?;
rendered.write_wav("mix.wav")?;

use cochlea_verify::{VerifyExt, Tol, Ms, Cents, Db};
let report = rendered.verify(&score)
    .true_peak_below(-1.0)
    .pitch_matches_score("lead", Cents(10.0))
    .monotone("lead", Param::CUTOFF_HZ, bar(1)..bar(3))
    .silent_after(bar(5))
    .run();
assert!(report.passed);
```

Or entirely from the command line, score as RON:

```
cochlea render score.ron --out mix.wav --stems stems/ --verify
cochlea probe input.wav --json report.json --spectro spec.png
cochlea probe input.wav --digest --window-ms 500
cochlea probe input.mp3 --from 42.0 --to 44.5      # zoom into a window, any format
cochlea diff a.wav b.wav --tier2 --spectro delta.png
cochlea lint score.ron
cochlea spectro input.wav --out spec.png --annotate  # draw beats/onsets/pitch on the image
cochlea import song.mid --out score.ron              # SMF -> score, timing exact
cochlea reference    # the full score-authoring reference, generated from the live preset bank
```

`cochlea probe` works on **any** WAV — and FLAC (decoded bit-exact), mp3,
and ogg, still ffmpeg-free — no score required. That's the front door:
point it at audio you didn't render and get the same JSON report and
spectrogram an agent uses to review its own work.

## How an agent listens

compose → render → probe (JSON) → spectrogram (one vision call) → verify

1. **compose** a score as data (RON, or the Rust builder above).
2. **render** it to deterministic PCM — `cochlea render score.ron --out mix.wav`.
3. **probe** the mix into a compact JSON report (loudness, onsets, pitch,
   key, silence, clipping) — `cochlea probe mix.wav --json report.json`.
   No image, no audio: the agent reads numbers.
4. **look**, when numbers aren't enough — `cochlea spectro mix.wav --out
   spec.png` renders one small PNG the agent reviews in a single vision
   call instead of reasoning about raw samples.
5. **verify** — `cochlea render score.ron --verify` runs the score's
   embedded assertions and exits nonzero on failure, so an agent can retry
   without a human confirming "yes, that sounds right."

And when something in the middle of a long file needs a closer listen,
every read tool takes `--from/--to`: probe just bars 17–19, spectrogram
just the drop. The cut is frame-exact, report times are relative to it,
and `source.start_ms` says where it came from — the tier stack becomes a
zoom lens instead of a whole-file-only report.

The economics are the point, not an afterthought. The `first_light` render
above is 7 seconds of 48 kHz/32-bit-float PCM and weighs 2.7 MB; a
3-minute piece at the same settings is ~66 MB — not something to hand an
agent as text, let alone read sample-by-sample. Its probe report is a few
KB of JSON (schema v4, trimmed here to the interesting fields — note
`pitch.melody`: the piece's bass line and melody read back as *note
events*, the compose loop's read-back half):

```json
{
  "schema_version": 4,
  "source": { "sample_rate": 48000, "channels": 2, "duration_ms": 7035.708333333333, "start_ms": 0.0 },
  "loudness": { "integrated_lufs": -22.700454879284784, "true_peak_dbtp": -15.910817022082783, "lra": 10.607660373688798 },
  "onsets": { "count": 6, "times_ms": [1077.33, 2149.33, 2346.67, 3221.33, 4538.67, 5034.67] },
  "pitch": { "voiced_ratio": 0.9847560975609756, "median_f0_hz": 110.00194603797897,
             "melody": [ { "name": "A2", "start_ms": 0.0, "end_ms": 1045.3, "cents_off": 0.1 },
                         { "name": "E2", "start_ms": 1077.3, "end_ms": 2112.0, "cents_off": 0.3 },
                         { "name": "F#2", "start_ms": 2154.7, "end_ms": 3178.7, "cents_off": 0.2 },
                         { "name": "E2", "start_ms": 3210.7, "end_ms": 4384.0, "cents_off": 0.3 },
                         { "name": "E5", "start_ms": 4394.7, "end_ms": 5813.3, "cents_off": -0.4 } ] },
  "timbre": { "mfcc_mean": [-37.64, 14.47, -4.44, 1.24, "..."], "mfcc_std": ["..."], "frames": 656 },
  "key": { "tonic": "E", "mode": "major", "confidence": 0.8093960265638273 },
  "tempo": { "bpm": 55.97014925373134, "confidence": 0.6633739386089712, "stability": 0.3333333333333333,
             "candidates": [ { "bpm": 55.97014925373134, "salience": 0.6633739386089712 },
                             { "bpm": 112.5, "salience": 0.21588204941945222 } ] },
  "rhythm": { "grid_alignment": 0.8333333333333334, "grid": "straight", "offbeat_ratio": 0.4, "clear_rhythm": true },
  "stereo": { "width": 0.02967719705208343, "correlation": 0.9981362354107913, "balance": -0.0016380539212361243 },
  "structure": { "section_count": 1, "confidence": 0.0 },
  "silence": { "trailing_ms": 2485.708333333333 },
  "clipping": { "clipped_samples": 0, "true_peak_over_0dbtp": false }
}
```

And the spectrogram is one small image. Here's the `title_cue` demo — a
pad whose `cutoff_hz` automation sweeps 250 Hz → 5000 Hz across bars 1–3:

![Mel spectrogram of the title_cue demo: the quiet band at the top of the frame narrows across the first two bars as the filter sweep lets more high-frequency energy through](docs/assets/title_cue_spectro.png)

*The dark band at the top of the frame narrows as the sweep runs — more
high-frequency energy gets let through over time. An agent reads that
directly off the image; the demo's `Monotone(track: "pad", param:
"cutoff_hz", ...)` assertion checks the same thing numerically.*

For a whole piece in one image regardless of length, `--sheet` tiles the
spectrogram into a contact sheet instead of one long strip (two bars per
tile here, `--bars-per-tile 2`):

![Contact-sheet spectrogram of first_light.ron tiled two bars per row](docs/assets/first_light_sheet.png)

## Reading audio without a context window

`probe --digest` skips JSON entirely and prints a deterministic text
summary — one line per feature dimension, then a windowed timeline capped
at ~40 rows. Real output for the `drum_groove` demo (20.8 s, four tracks,
the wave-2 rhythm/stereo/structure dimensions in one screenful):

```
cochlea digest: 20.755s  2ch  48000Hz
loudness: integrated=-24.06  momentary_max=-22.42  true_peak=-5.95  lra=1.61
key: A# minor (conf 0.54)  pitch: voiced=23%  median=63.8Hz (C2 -42.8c)
melody: 6 notes  C2 C2 C2 C2 A1 A1
tempo: 110.3bpm (conf 0.79, stability 1.00)  alts: 54.9bpm(0.89), 36.6bpm(0.79)
rhythm: clear=true  grid_align=0.98 (straight)  offbeat=0.56
stereo: width=0.07 corr=0.99 bal=-0.01
structure: 1 section
onsets: count=58  rate=2.79/s
silence: leading=0ms  trailing=2545ms
clipping: clipped=0  over_0dbtp=false
timeline: window=1000ms  bucket=1x  rows=21
   idx        t(s)     rms   peak  ons     f0  flags
     0   0.000-1.000   -25.55  -7.36    4    64.0  -
     1   1.000-2.000   -25.61  -8.37    3    63.4  -
     ...
```

Tempo and rhythm are reported as *separate axes*, because they change
independently — a drum solo can hold a rock-steady pulse while its
pattern turns unrecognizable, and that difference is exactly what an
agent needs to see. Here the tempo reads 110.3 BPM (matching the
authored 110), `stability 1.00` says the speed never moves across the
piece, and the `alts` list surfaces the genuine half-tempo reading (54.9
BPM, salience 0.89 — actually the stronger raw peak; the octave prior
breaks the tie toward the beat). Metrical ambiguity is data an agent can
weigh, not a coin flip hidden inside the detector. The `rhythm` line
then reports how the *hits* relate to that pulse: 98% of onsets sit on
the beat-subdivision grid, 56% of them on off-beat subdivisions (an
eighth-note hat groove — syncopation as a number), so `clear=true`.
(Under the pre-0.2.0 metric this same groove read `clear_rhythm=false`
at confidence 0.01 — layering hats, kick, snare, and pad across three
metrical levels diluted every lag's share of a mass-fraction score. The
grid-based rule asks the right question instead.)

The `(straight)` tag is the grid *hypothesis test*: alignment is measured
against both straight sixteenths and eighth-note triplets, and the report
carries whichever more hits land on. A shuffle or swing take reads
`grid_align=1.00 (triplet)` — recognized as an aligned triplet rhythm —
instead of being force-fit to sixteenths and scored sloppy.

`cochlea diff` compares two files in feature space instead of byte-for-byte
— "did my change do what I meant," not "is the file bitwise equal." Real
output diffing `first_light.wav` against `title_cue.wav`:

```
verdict: different (duration, loudness, onsets, key)
duration     a->b +1264.3 ms
loudness     integrated -5.95 LU  true_peak +5.70 dB  lra -8.88 LU
onsets       matched=0  mean_offset=-  max_offset=-  unmatched_a=6  unmatched_b=5
pitch        delta +0.5 cents
key          a=E major (conf 0.81)  b=A minor (conf 0.86)  changed=true
segments     max_abs_rms