forensic-deepdive

# forensic-deepdive

<!-- mcp-name: io.github.Dhevenddra/forensic-deepdive -->

> A persistent code knowledge graph + MCP server for AI coding agents. Five durable markdown artifacts as the human-readable projection. Apache-2.0.

`forensic-deepdive` analyzes any codebase (9 languages, polyglot) and produces:

1. **A persistent embedded graph** at `<repo>/.deepdive/graph.lbug` — File, Symbol, Module, Commit, Author, **Endpoint**, and **DbTable** nodes plus DEFINES, MEMBER_OF, IMPORTS, CALLS, EXTENDS, IMPLEMENTS, TOUCHED_BY_COMMIT, AUTHORED_BY, CO_CHANGES_WITH, and the cross-boundary HANDLES / CALLS_ENDPOINT / **ROUTES_TO** / INJECTS / PERSISTS_TO edges. **Every edge carries a confidence tag** (`EXTRACTED` / `INFERRED` / `AMBIGUOUS`) — no hidden heuristics. The single `Endpoint` join node unifies **five cross-boundary protocols** (HTTP, MCP tools, registry-dispatch, gRPC, messaging/AMQP), so a frontend call resolves to its backend handler across the stack as one `ROUTES_TO` edge.
2. **An MCP server** (`forensic serve`) exposing **9 composite tools** (`impact`, `context`, `archaeology`, `flow`, `query`, `record_insight`, `recall_insights`, `visualize`, `trace`) consumable by Claude Code, Cursor, Codex, Continue, Cline, Windsurf — and any other MCP-aware agent.
3. **Five durable markdown artifacts** under `<repo>/docs/codebase/`, regenerated from the graph on every extract:
   - **`MAP.md`** — what's where, ranked by centrality.
   - **`HOTPATHS.md`** — the dependency hot spots, with a per-row confidence-mix column so you see exactly how cleanly each symbol resolves.
   - **`ARCHAEOLOGY.md`** — why the code looks the way it does (git history, top authors with %, bus factor, co-change clusters, defect proximity).
   - **`MENTAL_MODEL.md`** — the doc the original author *would* write to onboard a new hire.
   - **`AGENT_BRIEF.md`** — ≤5 KB of assertive Never/Always rules with per-rule confidence tags. Drop-in `CLAUDE.md` for any project.
4. **Ten shims into the target repo** — 4 editor rule files (`CLAUDE.md`, `AGENTS.md`, `.cursor/rules/codebase.mdc`, `.continue/rules/codebase.md`), 5 single-intent Claude skills under `.claude/skills/codebase-{exploring,debugging,impact-analysis,refactoring,onboarding}/`, and a `.claude-plugin/plugin.json` manifest. All write-if-absent — hand-edited files are never overwritten.
5. **An agent-insight layer** — `record_insight` / `recall_insights` MCP tools backed by `<repo>/.deepdive/insights.jsonl` by default (zero dependencies, human-readable, git-friendly). The optional `[graphiti]` extra upgrades to a temporal knowledge graph backend above a 2-of-5 repo-size threshold.

Extract also regenerates **`ARCHITECTURE.md`** — a system-level Mermaid view of the cross-boundary graph (ROUTES_TO / INJECTS / PERSISTS_TO, confidence-styled), a *separate human-validation surface* (not one of the five contract artifacts, exactly like `forensic visualize` and `serve --ui`). Regenerate it on its own with `forensic diagram --repo <repo>`. Use it to sanity-check the graph — a wrong edge there is a wrong edge everywhere.

Add **`--emit-vault`** to also write an [Obsidian](https://obsidian.md)-friendly vault under `<output>/vault/` — every artifact gets `summary:`/`tags:` frontmatter, cross-references become `[[wikilinks]]`, and an `INDEX.md` MOC ties them together (with a `.obsidian/` config). A local-first second brain for humans (graph view, backlinks) and agents (triage by `summary:` without opening files, a traversable index). Opt-in; off by default.

## Status

**v0.10.0 "The Upgrade Path"** — an integrity release. v0.9 shipped correct, and then its findings run caught `--refresh-shims` being structurally unable to refresh half its targets, with 909 tests green over it. The cause wasn't the fix — it was that **every test in the suite wrote into an empty directory, so only the *first* run had ever been tested.** Everyone who isn't a new user lives on the untested path.

So 0.10 adds almost no surface. It makes the guarantees already made real: running Deepdive over a repo that already carries a previous release's output now provably **converges** (and provably leaves your hand-edited files alone), `extract` **tells you** when generated shims are stale instead of waiting for you to know the flag exists, and a stale `examples/` fails CI rather than being caught by eye mid-release. New: [`forensic extract --timings`](#quick-start), per-phase wall clock.

It also contains a **cancelled feature**, which is the more useful story. The planned performance work targeted PageRank. The profile required before touching it measured PageRank at **0.04 %** of a large extract, and the *already-batched* store writes at **79–87 %**. The optimization was disproven before it was written — see [`docs/findings/v0.10/PROFILE.md`](docs/findings/v0.10/PROFILE.md). Confirming a known technique had been applied had been mistaken for confirming it was fast.

The engine, the graph, the contract layer and the 5-artifact + 9-MCP-tool contract are **unchanged** from 0.9, and emitted content is byte-identical but for the version footer.

> **⚠ If you installed 0.8.0 or 0.9.0 and `forensic serve` stopped working — this is why.** Those
> releases declare `mcp>=1.27.1` with no upper bound. `mcp` 2.0.0 removed `mcp.server.fastmcp`,
> which the MCP server imports, so a **fresh install of 0.8.0 or 0.9.0 today** resolves to `mcp`
> 2.0.0 and fails on import. Our development lockfile pinned 1.27.1, so every local check stayed
> green and only the clean-environment wheel smoke in CI could see it. **Fix: upgrade to 0.10.0**
> (which caps `mcp<2`), or pin `mcp<2` yourself. Nothing else about those releases is affected.

**What's proven, and what isn't (honest framing).** Deepdive is an **assisted-analysis** tool. A real fresh-agent onboarding test confirmed it's **usable** and that an agent **auto-discovers** `AGENT_BRIEF.md` and routes to the right skill unprompted, and a grounded [MCP tool review](docs/findings/v0.7/mcp-tool-review.md) found the git-archaeology and curated briefs are the high-trust core. The **autonomous end-to-end** question, whether deepdive-seeding makes an agent *resolve* real issues measurably faster, is **still not proven**. A model-free localization **pilot** is recorded in [`experiments/fastcontext/RESULTS.md`](experiments/fastcontext/RESULTS.md), where the static seed turns out to be a *weak* prior, and the end-to-end measurement remains blocked on hardware (it needs a GPU plus a frontier main-agent endpoint). No autonomous-execution claims are made here. Accepted across real repos including Apache Superset, wagtail (Django), spring-petclinic, ripgrep, fastapi, and Iris-Nearby (Flutter/Dart). See [`docs/findings/`](docs/findings/).

## Quick start

```bash
# install from PyPI (puts `forensic` on PATH); or run ephemerally with uvx
uv tool install forensic-deepdive
forensic info            # banner + capability panel
forensic extract /path/to/repo

# …or from source for development:
git clone https://github.com/Dhevenddra/forensic-deepdive && cd forensic-deepdive
uv sync --all-extras

# what can it do? (banner + capability panel: artifacts, protocols, MCP tools, confidence legend)
uv run forensic info

# guided setup: analyze a repo, then wire it up as an MCP server
# (--yes takes every default: scriptable, and the one mode that needs no extra)
uv run forensic onboard --repo /path/to/repo

# run on any repo
uv run forensic extract /path/to/repo

# graph lands at <repo>/.deepdive/graph.lbug
# 5 markdown artifacts at <repo>/docs/codebase/
# 10 shims at <repo>/.claude/, .cursor/, .continue/, root

# trace a cross-stack feature slice (frontend call -> endpoint -> handler -> tail)
uv run forensic trace <symbol> --repo /path/to/repo

# interactive query REPL over one held-open store (needs the [interactive] extra)
# bare text = natural-language query (no LLM) · :cypher <q> = raw Cypher · :help · Ctrl-D exits
uv run forensic repl --repo /path/to/repo

# full-screen terminal graph browser — the loopback-free sibling of serve --ui
# 1/2/3 = Symbols/Files/Endpoints · type to filter · c/e/l = confidence/edge/language · Enter = context · i/f = impact/flow
uv run forensic browse --repo /path/to/repo

# the session shell: all of the above over ONE held-open graph, with history
# in-session: extract · query · trace · impact · flow · diagram · browse · onboard · serve
uv run deepdive --repo /path/to/repo

# query the graph as an MCP server (point it at the analyzed repo)
uv run forensic serve --repo /path/to/repo

# inspect every repo you've analyzed
uv run forensic list
```

## Install from PyPI

Published as **[`forensic-deepdive`](https://pypi.org/project/forensic-deepdive/)** —
no clone needed:

```bash
uv tool install forensic-deepdive        # puts `forensic` on PATH
forensic extract /path/to/repo

# …or run ephemerally, no install:
uvx forensic-deepdive extract /path/to/repo
```

Optional extras: `uv tool install "forensic-deepdive[semantic]"` (offline ONNX NL
query), `[interactive]` (the `forensic repl` query console, the `forensic browse`
TUI graph browser, and the `deepdive` session shell), `[openapi]` (YAML spec
parsing), `[graphiti]` (temporal insight backend).

### Upgrading from 0.8

Re-run `extract` with `--refresh-shims` once. The generated shims and skills under
`.claude/`, `.cursor/` and `.continue/` are written only if absent, so a plain
re-extract leaves your 0.8-era copies in place, and two of the skill files still cite
internal decision IDs that 0.9 removed:

```bash
forensic extract /path/to/repo --force --refresh-shims
```

Only files Deepdive generated are rewritten. A shim you have hand-edited is never
touched.
`pip install forensic-deepdive` works too if you're not on `uv`.

## Use it as an MCP server

`forensic serve` is a stdio MCP server exposing the 9 composite tools to any
MCP-aware agent (Claude Code, Cursor, VS Code/Copilot, Codex, Continue, Cline,
Windsurf). First build the graph once (`forens