smart-context-mcp

# smart-context-mcp

MCP server that reduces AI agent token usage by up to 90% through intelligent context compression (measured on this project).

[![npm version](https://img.shields.io/npm/v/smart-context-mcp.svg)](https://www.npmjs.com/package/smart-context-mcp)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## What it is

An MCP (Model Context Protocol) server that provides specialized tools for reading, searching, and managing code context efficiently. Instead of loading full files or returning massive search results, it compresses information while preserving what matters for the task.

**Real metrics from production use:**
- ~7M tokens → ~800K tokens (approximately 89% reduction)
- 1,500+ operations tracked across development
- Compression ratios: 3x to 46x depending on tool
- Context overhead is tracked separately so reports can show gross and net savings

**Workflow-level savings:**
- Debugging: ~85-90% token reduction
- Code Review: ~85-90% token reduction
- Refactoring: ~85-90% token reduction
- Testing: ~85-90% token reduction
- Architecture: ~85-90% token reduction

**Real adoption in non-trivial tasks:**
- Approximately 70-75% of complex tasks use devctx tools
- Most used: `smart_read` (850+ uses), `smart_search` (280+ uses), `smart_shell` (220+ uses)
- Primary reasons for non-usage: task too simple, no index built, native tools preferred

See [Workflow Metrics](./docs/workflow-metrics.md) and [Adoption Metrics](./docs/adoption-metrics-design.md) for details.

## Latest Release: `1.21.0`

Minor release built around a **semantic engine**. MCP grows from **20 → 22 tools** (`smart_code` + `smart_output`). SQLite schema bumps 8 → 9 (new `outputs` table; auto-migrates on first run). **Zero new runtime dependencies** — the TypeScript LanguageService comes from the `typescript` package you already have, and degrades to a fallback provider when it isn't there.

- **`smart_code` (new tool).** Semantic code navigation for JS/TS via the TypeScript LanguageService, resolving by **symbol name** (plus optional `filePath`) instead of forcing you to know a line and column. Actions: `definition`, `references`, `implementations`, `diagnostics`, `impact`, `rename`. Returns compact locations (file + 1-based start/end), never full file bodies. `includeTests=false` drops test paths; `maxResults` caps the payload.
- **Impact analysis.** `smart_code(action='impact')` splits **direct** hits (semantic: definitions, references, implementations) from **transitive** files (import graph, `maxHops` default 2) and related tests, plus coverage flags. `risk.level` is labelled `basis: 'heuristic'` and says so in its own `note` — it is a ranking aid, not semantic certainty.
- **Scope-aware rename, dry run by default.** `smart_code(action='rename')` uses `findRenameLocations` and **`dryRun: true` is the default**: the first call returns the planned per-file diff hunks (`before`/`after` per line) and writes nothing. Error conflicts (invalid or reserved `newName`, same name, unresolved target, path escape, missing file, blast radius over `maxFiles`) always block the write. `name-collision` is a heuristic warning that only blocks with `strict: true`, because the check doesn't verify scope overlap. After a real write, `diagnosticsAfter` reports remaining TypeScript errors in the touched files. `smart_edit` remains the tool for non-semantic textual replacements.
- **`smart_output` (new tool).** Persistent output/artifact store (`shell`, `test`, `build`, `lint`, `diff`) so a stack trace from twenty turns ago is recoverable **without rerunning the command**. Actions: `save`, `search`, `excerpt`, `summary`, `list`, `stats`, `prune`. Content is scrubbed for likely secrets before persistence and truncated head+tail above `DEVCTX_OUTPUT_MAX_BYTES` (256KB) so the failing tail always survives. Retention via `DEVCTX_OUTPUT_RETENTION_DAYS` (14) and `DEVCTX_OUTPUT_MAX_PER_KIND` (50). Identical output for the same `kind`+`command` is deduped into `repeat_count`. Automatic capture from `smart_shell`/`smart_test` is opt-in via `DEVCTX_OUTPUT_STORE=true` (then `smart_shell` returns an `outputRef`); explicit `save`/`search` always work. Degrades with `degraded: true` instead of throwing when SQLite is unavailable or locked.
- **Opt-in semantic expansion in `smart_context`.** `include: ['semantic']` expands context through real definitions, references and implementations rather than text similarity, and every item gains a `whyIncluded` explanation. Off by default until the precision@5 benchmark justifies otherwise.
- **Richer ADR/spec awareness and graph paths** carried over from the same roadmap: ADR sections are indexed as `kind='adr'` / `'adr-section'`, and `smart_context` can return graph **paths** between two files or symbols.

- **Shared `tokenBudget` across tools.** `smart_read`, `smart_read_batch`, `smart_context`, `smart_turn` (`start` + `end`) and `smart_resume` now accept `tokenBudget: number | { id?, maxTokens, shared? }`. When `shared:true` (or `id` set), the budget is reused across calls inside the same task — so a multi-step agent flow can stay under a hard token ceiling without per-call bookkeeping. Responses include `taskBudget`, `remainingBudget`, and `budgetDetails` (`scope`, `actions`, degraded mode) when the budget actually changed the output.
- **`smart_search` search modes.** New `mode: 'needle' | 'balanced' | 'semantic'` (default `balanced`). `needle` = literal exact only (no regex / no term expansion) — kills noise on debug queries. `balanced` = exact + regex + term expansion. `semantic` = exact-first plus the local semantic block only when exact signal is weak. The previous `semantic: true` flag remains as a legacy alias for `mode: 'semantic'`. Default `maxFiles` tightened 15 → 5. New `maxTokens` caps the whole response and compacts intelligently (matches first, then diagnostics, then semantic block). Per-file ranking is now inspectable via `matchedBy`, `boostSource`, `scoreBreakdown`, `whyRanked`. Response also returns `hasMore` / `totalFiles` / `nextSuggestedMaxFiles` and actionable `suggestions` when the query is too broad or empty.
- **`smart_read` persistent cache + budget-aware `full` degradation.** New SQLite `read_cache` table keyed by `(filePath, mode, selector, content_hash)`. Second read of an unchanged file is virtually free. Mode `full` is now an **explicit last resort**: if a `tokenBudget`/`maxTokens` is set, it degrades to lighter modes first (outline → signatures → truncated) and reports the real mode used in `fullMode` + `budgetDetails`. New `clearReadCachePersistent` + GC integration in `runStorageMaintenance`.
- **`smart_turn` simple-task skip heuristic.** When the prompt is short (≤ 40 chars after normalization), classified as a simple task, and no session/task is pinned, `smart_turn(start)` now returns `skipSmartTurn: true` with `recommendedPath.mode='simple_task_skip'` instead of paying the full orchestration cost. Saves continuity-resolution overhead on trivial prompts. `minimal` verbosity additionally compacts summary/refreshedContext to the fields agents actually consume.
- **`global_memory` noise hints.** Per-project, scrubbed noise telemetry persisted to `~/.devctx/global.db` (`noise_hints` table). New actions `noise_stats` and `noise_reset` (full or via `query`). Lets `smart_search` learn which queries the agent already discovered to be noisy in a given repo and adjust ranking, without ever leaking content.
- **KPI baseline infrastructure.** New scripts `evals/kpi-baseline.js` + `evals/kpi-utils.js` aggregate `harness.js` and `realworld-eval.js` runs into a single JSON snapshot with **top-5 precision, recall, reread task/call rate, and per-task-size buckets (short / long)**. Persists `kpi-baseline-latest.json` for regression detection across releases. New test suite `tests/eval-kpis.test.js`.

### Highlights from `1.20.0` (still current)

Same tool count as `1.19.0`, but several tools gained hard token-budget control, search-mode discipline and second-read cache reuse.

- **Shared `tokenBudget` across tools.** `smart_read`, `smart_read_batch`, `smart_context`, `smart_turn` and `smart_resume` accept `tokenBudget: number | { id?, maxTokens, shared? }`, reusable across calls inside one task so a multi-step flow stays under a hard ceiling.
- **`smart_search` modes.** `mode: 'needle' | 'balanced' | 'semantic'` (default `balanced`), `maxTokens` capping the whole response, and inspectable ranking via `matchedBy` / `scoreBreakdown` / `whyRanked`.
- **`smart_read` persistent cache.** SQLite `read_cache` keyed by `(filePath, mode, selector, content_hash)`; a second read of an unchanged file is virtually free. Mode `full` degrades to lighter modes under a budget and reports the real mode used.
- **`smart_turn` simple-task skip** on short trivial prompts, and **`global_memory` noise hints** that let `smart_search` learn which queries proved noisy per repo.
- **KPI baseline infrastructure** (`evals/kpi-baseline.js`) snapshotting precision@5, recall, reread rate and latency for regression detection.

### Highlights from `1.19.0`

Five-step quality jump executed as sequential commits with full dogfooding. MCP grew from **18 → 20 tools**, +68 tests, **zero new dependencies**.

- **`smart_playbook` (new tool).** Declarative composite workflows that run multiple `smart_*` tools in a single MCP call. Five built-in playbooks ship with the package: `preflight-merge` (review + affected tests + checkpoint), `debug-flake` (last failure + curated debug context + affected), `refactor-safe` (curated context + affected + checkpoint), `doc-sync` (ADR search + docs context), `ramp-up` (status + doctor + ADR overview). Project-level overrides via `.devctx/playbooks/*.{yaml,json}` with `{{args.X}}` interpolation, `when` / `label` / `stopOnFail` / `dryRun`. Tool allowlist restricted to `smart_*`. Zero deps: built-in minimal YAML parser.
- **Reactive FS watcher for the ind