wiki-retrieve
**wiki-retrieve** is a chunk-level hybrid retrieval system for the Compound Vault that combines contextual prefixes, BM25 sparse search, and cosine reranking to replace the v1.6 page-level read order. Based on Anthropic's September 2024 Contextual Retrieval research, it reduces retrieval failure by 35–67% and is opt-in via setup script, with data privacy controls that allow fully on-machine operation using synthetic prefixes and local Ollama reranking.
git clone --depth 1 https://github.com/AgriciDaniel/claude-obsidian /tmp/wiki-retrieve && cp -r /tmp/wiki-retrieve/skills/wiki-retrieve ~/.claude/skills/wiki-retrieveSKILL.md
# Retrieve relevant passages This extension derives search data from `wiki/` into `.vault-meta/`. It never changes canonical notes. Always pass the selected vault explicitly. Resolve the installed product root from this skill's own location, not from the vault or current working directory: ```bash PRODUCT_ROOT=/absolute/path/to/installed/claude-obsidian PREFIX="$PRODUCT_ROOT/scripts/contextual-prefix.py" BM25="$PRODUCT_ROOT/scripts/bm25-index.py" RETRIEVE="$PRODUCT_ROOT/scripts/retrieve.py" RERANK="$PRODUCT_ROOT/scripts/rerank.py" test -f "$PREFIX" && test -f "$BM25" && test -f "$RETRIEVE" && test -f "$RERANK" ``` ## Pipeline 1. `contextual-prefix.py` splits pages on paragraph boundaries and stores the raw chunk plus a short page-level prefix. 2. `bm25-index.py` builds a local, standard-library BM25 index over the contextualized text. 3. `retrieve.py` selects BM25 candidates, optionally reranks them, rejects invalid records, deduplicates by page, and returns paths and snippets. 4. The caller reads the returned pages and performs synthesis; retrieval output is not itself evidence. ## Provision locally Preview first, then build synthetic prefixes without network egress: ```bash python3 "$PREFIX" --vault "$VAULT" --all --no-llm --peek python3 "$PREFIX" --vault "$VAULT" --all --no-llm python3 "$BM25" --vault "$VAULT" build python3 "$RETRIEVE" --vault "$VAULT" "wiki" --top 1 --no-rerank --explain ``` Chunk and index files are disposable runtime state. Incremental prefixing skips records whose chunk and page hashes still match. A complete scan removes surplus records for deleted pages, and the prefixer invalidates the BM25 index before changing its chunk set so a mixed stale index is not served. Prefix and BM25 build operations share the vault-wide mutation lock with every other writer; a busy vault fails closed instead of publishing a partial index. ## Contextual-prefix privacy Synthetic prefixes use only local frontmatter and page text. The Anthropic API and `claude` subprocess tiers can send page bodies off-machine and therefore require the user's explicit consent plus `--allow-egress`. Never infer consent from an API key or installed binary. Preview the scope first and state which provider will receive what data. Remote Ollama endpoints also require explicit approval and `--allow-remote-ollama`; the default reranker accepts localhost only. ## Query For a strictly read-only lookup, use the prebuilt BM25 index: ```bash python3 "$RETRIEVE" --vault "$VAULT" "$QUERY" --top 5 --no-rerank --explain ``` For an explicitly requested rerank, omit `--no-rerank`. The default is Ollama's multilingual `nomic-embed-text-v2-moe` model (approximately 958 MB); the product never pulls it automatically. To use an already-installed, smaller, English-oriented v1.5 model, pass `--model nomic-embed-text` explicitly. Nomic models use `search_query:` for the query and `search_document:` for candidate text. Nomic v2 has a 512-token input context and Ollama truncates longer embedding inputs by default; BM25 still scores the complete chunk. Embeddings are cached by exact model, input scheme, and hash of the exact prefixed input. A missing local Ollama service, missing selected model, unusable vector, or any candidate embedding failure falls back for the complete result set to the original BM25 order; it never mixes cosine and BM25 score scales. Query input is bounded at 8,000 normalized characters and result counts must be between 1 and 1,000. Oversized queries and invalid limits fail with an actionable usage error instead of looking like an empty successful search. An untagged model request matches only the installed untagged name or its `:latest` alias; select any other tag explicitly. Use direct diagnostics when needed: ```bash python3 "$BM25" --vault "$VAULT" stats python3 "$BM25" --vault "$VAULT" query "$QUERY" --top 10 python3 "$RERANK" --vault "$VAULT" "$QUERY" --peek python3 "$RERANK" --vault "$VAULT" "$QUERY" --model nomic-embed-text --peek ``` ## Integrity rules - Accept only relative chunk and page paths whose resolved targets remain under `$VAULT/.vault-meta/chunks/` and `$VAULT/wiki/` respectively. - Reject hashless legacy chunk records and require chunk-body, page, and index hashes to match before a cached record can be built or served. - Reject absolute paths, symlink escapes, missing pages, mismatched chunk IDs, changed page hashes, and stale index/chunk hash pairs. - Rerank the full candidate set, then deduplicate by page, then apply `--top`. - An empty index is an honest no-result state. A missing or corrupt index makes `retrieve.py` exit 10 with a stable rebuild command; callers fall back to the standard vault query/text-search path and do not fabricate matches. - Do not cite benchmark percentages unless a reproducible vault-specific benchmark produced them. ## Checkpoint Observe cache readiness and privacy boundaries, think about whether lexical or semantic ranking is needed, verify returned paths and source freshness, and grow by measuring retrieval misses against a maintained local query set.
>
Ingest supplied source material into an Obsidian vault with provenance and claim tracking: pasted text, files staged in the selected vault's inbox or .raw archive, or explicitly approved URLs. Use for a single source or bounded batch, not for saving an assistant answer. Triggers: ingest, ingest this file, ingest this URL, process this source, read and file this source, batch ingest, ingest these sources.
Run a deterministic, read-only health check on an Obsidian wiki. Use for lint, vault health check, audit wiki health, find orphans, find dead links, frontmatter audit, provenance audit, or wiki audit. Reports graph, link, frontmatter, provenance-ledger, empty-section, and stale-index findings; it does not reason broadly or repair files.
Run a bounded, source-grounded research loop, draft a cited dossier, and optionally propose a separately reviewed canonical vault merge. Use when the user wants autonomous or deep research that may access the public web. Triggers: /autoresearch, autoresearch, research this topic, deep dive into, investigate, find everything about, research and file, go research, build a wiki on.
Create, inspect, and update Obsidian JSON Canvas boards with text, file, link, group, and edge nodes. Use for canvas status, canvas lists, visual maps, zones, spatial layouts, adding vault notes or media to a .canvas file, and requests such as create canvas, add to canvas, or put this on the canvas.
Save a user-selected answer, decision, insight, or session summary into an Obsidian vault as one reviewed transaction. Use only when the user explicitly asks to preserve specific conversation content, not when they supply a file or URL to ingest. Triggers: /save, save this, save that answer, file this conversation, save this analysis, keep this insight, preserve this chat result.
Initialize, adopt, and route work for a separate Obsidian knowledge vault through the portable claude-obsidian core. Use for vault setup, scaffolding, workspace selection, cross-project configuration, or choosing the correct wiki sub-skill. Triggers: /wiki, set up wiki, scaffold vault, create knowledge base, adopt this vault, Obsidian vault, second brain setup, persistent wiki.
Plan and, with explicit network consent, use an optional external Defuddle cleaner to extract article-like HTTPS pages as Markdown. Use for defuddle, clean this URL, strip page clutter, readable Markdown from a web page, or preparing a web source for later wiki ingestion.