llm-cost-optimizer

## Overview

This skill walks through analyzing and reducing LLM spend on a Vellum assistant. There are three layers:

1. **Provider connections** — named auth configs (e.g. `anthropic-managed`, `my-personal-key`)
2. **Model profiles** — named presets (provider + model + effort + thinking + contextWindow). Four managed defaults, with UI labels. Note that the keys do not track the labels: read the key, not the name, when pinning a call site.
   - `balanced` → **Balanced** (the general agent-loop profile)
   - `quality-optimized` → **Quality** (the expensive escalation profile)
   - `cost-optimized` → **Cost** (the cheap utility/background profile, and the one to pin for spend reduction)
   - `latency-optimized` → **Speed** (the low time-to-first-token profile, used by live voice; faster but not cheaper than Cost)
3. **Call-site profile pins** (`llm.callSites.<id>.profile`) — optional per-task overrides of the shipped defaults.

The concrete model behind each managed profile depends on the install: platform-managed installs and BYOK installs resolve different providers/models, and the catalog changes over time. **Never assume which model a profile maps to** — read `assistant config get llm.profiles` and the usage breakdown by `model` to see what actually ran.

## How model selection works — read this before diagnosing

Every LLM call resolves exactly **one winning profile** through a strict first-usable-wins chain. Profiles never merge with each other:

1. **Per-conversation / per-run override** — the user's `/model` pick, an open `assistant inference session`, or a schedule's pinned profile
2. **`llm.activeProfile`** — applies to `mainAgent` (the chat loop) **only**; it IS the user's chat-model selection and outranks any `llm.callSites.mainAgent` pin
3. **`llm.callSites.<site>.profile`** — explicit per-site pin
4. **The call site's shipped default intent**, resolved through `llm.defaultProvider`
5. `balanced` intent (final anchor)

A rung only wins if its profile exists, is enabled, and carries its own provider + model; otherwise resolution silently falls to the next rung.

Consequences that change how you diagnose cost:

- **A missing or empty `llm.callSites` block is healthy, not a red flag.** Every call site ships with a sensible default intent: the agent loop and quality-sensitive sites (`mainAgent`, `subagentSpawn`, `compactionAgent`, `callAgent`, `patternScan`, `narrativeRefinement`, `memoryConsolidation`, `memoryV2Consolidation`, `memoryV3SelectL2`, `recall`, `conversationStarters`, `identityIntro`, `emptyStateGreeting`) default to `balanced`; everything else (classifiers, summarization, titles, copy generation, memory extraction/retrieval/sweeps, heartbeat, home-screen content, etc.) defaults to `cost-optimized`. Nothing "falls back" to an expensive model.
- **Do not write a full `llm.callSites` blob that mirrors the shipped defaults.** That freezes today's defaults into user config and silently opts the user out of future default improvements (and of tuning shipped alongside them, like cache and context-window settings). Pin only deliberate deviations.

## Step 1 — Measure current spend

```bash
# Monthly totals
assistant usage totals --range month

# Break down by conversation (what the user actually did — use this for presentation)
assistant usage breakdown --group-by conversation --range month

# Break down by call site (what kind of work is expensive — use for diagnosis)
assistant usage breakdown --group-by call_site --range month

# Break down by model (what actually ran)
assistant usage breakdown --group-by model --range month

# Break down by profile (which selection produced it)
assistant usage breakdown --group-by inference_profile --range month
```

Cross-reference the `call_site` and `inference_profile` breakdowns: a background call site showing spend under an expensive profile means an override or pin routed it there — that is the interesting finding, not the config defaults.

Add `--json` when you need token-level detail (input vs output vs `cache_creation` vs `cache_read`) — high input volume on a cheap model can outweigh low volume on an expensive one.

## Step 1b — Present costs in user-friendly terms

After gathering the data, present findings in a format the user can act on. Users think in terms of conversations they had and automations they set up — not call sites, inference profiles, or cache economics. Use the `call_site`, `model`, and `inference_profile` breakdowns for **diagnosis**, but lead the presentation with what the user recognizes.

**Presentation structure:**

1. **One-line headline** with total monthly cost.
2. **Conversations table** — the user's own activity, sorted by cost descending. Columns: conversation name, turns, cost, and a brief note on why it was expensive (e.g. "One heavy session drove 65% of your total spend"). Roll up small conversations into an "All other conversations" row to keep the table to 4-6 rows.
3. **"What I'd change to cut costs"** — 2-3 bullets in plain English, biggest lever first. No jargon. Instead of "drop the balanced profile effort from high to medium," say "your chat model is set to high effort — dropping to medium would save ~\$X/week." Each bullet states what to change, why it helps, and the estimated savings.
4. **A clear ask** — "Want me to make either of those changes?"

**What to omit from the user-facing presentation:**

- **Background work** (memory processing, heartbeats, health checks) — users can't control these individually and they're already on cheap models by default. Mentioning them adds noise without actionable signal.
- **Call site names, inference profile names, token counts, cache ratios** — these are diagnostic internals. Use them to figure out what's expensive, then translate to plain English.
- **Recurring automation costs** unless one is a significant contributor (>$4/month). A monthly $0.17 digest doesn't need its own line in the summary.

## Step 2 — Read the effective configuration

```b