rag-design-doc
Design a Retrieval-Augmented Generation system end to end. Use when asked to design a RAG pipeline, a 'chat with your docs' feature, a knowledge assistant, or to debug why a RAG system gives wrong/ungrounded answers. Produces a RAG design doc — ingestion & chunking, embeddings & index, retrieval & reranking, the generation prompt, grounding/citations, evaluation, and failure modes with mitigations.
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills /tmp/rag-design-doc && cp -r /tmp/rag-design-doc/plugins/pm-ai/skills/rag-design-doc ~/.claude/skills/rag-design-docSKILL.md
# RAG Design Doc Skill Most RAG systems fail not at generation but at retrieval — the model answers confidently from the wrong chunks. This skill forces the decisions that actually determine quality (chunking, retrieval, reranking, grounding) and pairs each with how you'll evaluate it, so "it hallucinates sometimes" becomes a diagnosable, fixable pipeline. ## Required Inputs Ask for these only if they aren't already provided: - **Corpus** — what's being retrieved over (docs, tickets, code, tables), size, and update frequency. - **Queries** — the kinds of questions users ask, and how precise/recall-sensitive they are. - **Grounding requirement** — must answers cite sources? Is "I don't know" acceptable (it should be)? - **Constraints** — latency budget, cost, privacy/tenancy (per-customer isolation?), and freshness needs. ## Output Format ### RAG Design: [system] **1. Goal & non-goals** — what questions it answers well, and what it explicitly won't do. **2. Ingestion & chunking** - Source connectors and refresh strategy (full re-index vs. incremental). - **Chunking:** strategy (fixed, recursive, semantic, structure-aware), size + overlap, and what metadata travels with each chunk (source, section, timestamp, permissions). Chunking is the highest-leverage choice — justify it. **3. Embeddings & index** — embedding model + dimension, vector store, and the index/filter strategy (incl. metadata filters and per-tenant isolation). **4. Retrieval** — top-k, hybrid (dense + keyword/BM25) vs. pure vector, metadata pre-filtering, and query transformation (rewriting, decomposition, HyDE) if used. **5. Reranking** — whether a cross-encoder/reranker narrows the candidate set before generation, and the final context budget. **6. Generation** — the prompt template, how retrieved context is formatted, the instruction to **answer only from context and say "I don't know" otherwise**, and how citations are produced and verified. **7. Evaluation** — retrieval metrics (recall@k, MRR) *separately from* answer quality (faithfulness/groundedness, correctness). Pair with an [`ai-eval-plan`](../ai-eval-plan/SKILL.md). **8. Failure modes & mitigations** — a table: symptom → likely stage → fix. | Symptom | Likely cause (stage) | Mitigation | |---|---|---| | Confident but wrong | retrieval missed the chunk | hybrid search, better chunking, rerank | | Right doc, wrong detail | chunk too large/small | tune size+overlap, structure-aware split | | Ignores retrieved context | prompt/format | stronger grounding instruction, fewer/cleaner chunks | | Stale answers | index freshness | incremental re-index, timestamp filter | ## Quality Checks - [ ] Retrieval quality is evaluated **separately** from answer quality (you can't fix what you can't isolate) - [ ] The system can say "I don't know" when context is insufficient — it's not forced to answer - [ ] Answers carry citations that are verified against the retrieved context - [ ] Chunking strategy and size are justified against the corpus structure, not copied from a tutorial - [ ] Per-tenant / permission isolation is handled in retrieval, not just at the UI - [ ] Hybrid (keyword + vector) retrieval is considered for queries with exact terms/IDs ## Anti-Patterns - [ ] Do not jump to "fine-tune the model" when retrieval is the problem — fix what's retrieved first - [ ] Do not evaluate only the final answer — a good answer from luck and a bad answer from bad retrieval look different and need different fixes - [ ] Do not force an answer when nothing relevant was retrieved — an honest "I don't know" beats a confident hallucination - [ ] Do not ignore metadata filtering — semantic similarity will happily return the right-sounding chunk from the wrong document or wrong tenant - [ ] Do not pick a chunk size by default — it's the single biggest lever on retrieval quality ## Based On Retrieval-Augmented Generation practice — hybrid retrieval, reranking, grounded generation, and faithfulness evaluation.
Conduct a structured ethical review of an AI or ML feature, model, or product. Use when preparing to deploy an AI system, assessing algorithmic risk, auditing a model for bias, or producing a responsible AI impact assessment. Produces a structured ethics review covering fairness, transparency, privacy, safety, accountability, and societal impact with a risk tier score, pre-deployment checklist, and prioritised mitigations.
Structure AI and ML product decisions with the rigour of any product decision. Use when building AI-powered features, evaluating LLM integrations, designing AI products, or assessing AI readiness. Produces a complete AI product canvas covering problem definition, model approach, data requirements, evaluation framework, UX design, responsible AI checklist, and launch monitoring plan.
Transform feature briefs into structured design briefs that give designers the context they need before opening Figma. Use when asked to write a design brief, create a design handoff, brief a designer on a new feature, or translate a PRD into design requirements. Produces a brief with user goal, emotional context, success criteria, constraints, edge cases, and out-of-scope boundaries.
Design statistically rigorous A/B tests and interpret experiment results. Use when asked to design an experiment, run an A/B test, calculate sample size, interpret test results, or assess whether an experiment was successful. Produces a complete experiment design with hypothesis, sample size, run time, success criteria, and risk flags — or a results interpretation with ship/iterate/kill recommendation.
Synthesises user signals from multiple research sources into a unified, weighted insight brief. Use when you have data from interviews, support tickets, NPS verbatims, app reviews, or sales calls and need to reconcile contradictions, surface the underlying need behind requests, or answer 'what are users really telling us'. Produces ranked insights with confidence ratings, source weighting rationale, divergent signal analysis by user segment, and a research gap identification section.
Structure a product data analysis, metric deep-dive, funnel analysis, or cohort study. Use when asked to analyse product metrics, investigate a drop in conversion, explain a data change to stakeholders, or find the root cause of a metric movement. Produces a structured analysis with question, root cause, confidence level, and recommended action.
Interpret product metrics against goals and surface actionable signals. Use when asked to analyse product health, review key metrics, investigate a performance issue, produce a health report, or assess product-market fit signals. Produces a structured health report with RAG status, trend analysis, root cause hypotheses, and prioritised actions.
Structure a retention analysis, churn investigation, or engagement deep-dive for any product team. Use when asked to analyse user retention, investigate churn, measure DAU/MAU, or build a retention improvement plan. Produces a retention snapshot with root cause hypotheses, aha-moment correlation, and prioritised interventions.