rag-design-doc
Design a Retrieval-Augmented Generation system end to end. Use when asked to design a RAG pipeline, a 'chat with your docs' feature, a knowledge assistant, or to debug why a RAG system gives wrong/ungrounded answers. Produces a RAG design doc — ingestion & chunking, embeddings & index, retrieval & reranking, the generation prompt, grounding/citations, evaluation, and failure modes with mitigations.
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills /tmp/rag-design-doc && cp -r /tmp/rag-design-doc/plugins/pm-ai/skills/rag-design-doc ~/.claude/skills/rag-design-docSKILL.md
# RAG Design Doc Skill Most RAG systems fail not at generation but at retrieval — the model answers confidently from the wrong chunks. This skill forces the decisions that actually determine quality (chunking, retrieval, reranking, grounding) and pairs each with how you'll evaluate it, so "it hallucinates sometimes" becomes a diagnosable, fixable pipeline. ## Required Inputs Ask for these only if they aren't already provided: - **Corpus** — what's being retrieved over (docs, tickets, code, tables), size, and update frequency. - **Queries** — the kinds of questions users ask, and how precise/recall-sensitive they are. - **Grounding requirement** — must answers cite sources? Is "I don't know" acceptable (it should be)? - **Constraints** — latency budget, cost, privacy/tenancy (per-customer isolation?), and freshness needs. ## Output Format ### RAG Design: [system] **1. Goal & non-goals** — what questions it answers well, and what it explicitly won't do. **2. Ingestion & chunking** - Source connectors and refresh strategy (full re-index vs. incremental). - **Chunking:** strategy (fixed, recursive, semantic, structure-aware), size + overlap, and what metadata travels with each chunk (source, section, timestamp, permissions). Chunking is the highest-leverage choice — justify it. **3. Embeddings & index** — embedding model + dimension, vector store, and the index/filter strategy (incl. metadata filters and per-tenant isolation). **4. Retrieval** — top-k, hybrid (dense + keyword/BM25) vs. pure vector, metadata pre-filtering, and query transformation (rewriting, decomposition, HyDE) if used. **5. Reranking** — whether a cross-encoder/reranker narrows the candidate set before generation, and the final context budget. **6. Generation** — the prompt template, how retrieved context is formatted, the instruction to **answer only from context and say "I don't know" otherwise**, and how citations are produced and verified. **7. Evaluation** — retrieval metrics (recall@k, MRR) *separately from* answer quality (faithfulness/groundedness, correctness). Pair with an [`ai-eval-plan`](../ai-eval-plan/SKILL.md). **8. Failure modes & mitigations** — a table: symptom → likely stage → fix. | Symptom | Likely cause (stage) | Mitigation | |---|---|---| | Confident but wrong | retrieval missed the chunk | hybrid search, better chunking, rerank | | Right doc, wrong detail | chunk too large/small | tune size+overlap, structure-aware split | | Ignores retrieved context | prompt/format | stronger grounding instruction, fewer/cleaner chunks | | Stale answers | index freshness | incremental re-index, timestamp filter | ## Quality Checks - [ ] Retrieval quality is evaluated **separately** from answer quality (you can't fix what you can't isolate) - [ ] The system can say "I don't know" when context is insufficient — it's not forced to answer - [ ] Answers carry citations that are verified against the retrieved context - [ ] Chunking strategy and size are justified against the corpus structure, not copied from a tutorial - [ ] Per-tenant / permission isolation is handled in retrieval, not just at the UI - [ ] Hybrid (keyword + vector) retrieval is considered for queries with exact terms/IDs ## Anti-Patterns - [ ] Do not jump to "fine-tune the model" when retrieval is the problem — fix what's retrieved first - [ ] Do not evaluate only the final answer — a good answer from luck and a bad answer from bad retrieval look different and need different fixes - [ ] Do not force an answer when nothing relevant was retrieved — an honest "I don't know" beats a confident hallucination - [ ] Do not ignore metadata filtering — semantic similarity will happily return the right-sounding chunk from the wrong document or wrong tenant - [ ] Do not pick a chunk size by default — it's the single biggest lever on retrieval quality ## Based On Retrieval-Augmented Generation practice — hybrid retrieval, reranking, grounded generation, and faithfulness evaluation.
Design a 360-degree feedback survey or write a structured 360 feedback report. Use when asked to build a 360 feedback process, write 360 feedback for a colleague, design a feedback survey, or produce a feedback report. Produces either a complete survey instrument with rating scales and open-ended questions, or a structured narrative feedback report with themes, strengths, and development areas.
Decode a 401k or workplace retirement plan — the real cost of its funds, the match's fine print, vesting math, and the plan features worth using or avoiding. Use when someone asks 'is my 401k any good', 'decode my 401k plan', 'which funds should I look at', or 'what fees am I paying'. Produces a fee decode in dollars-over-time, match and vesting math, a fund-lineup triage by cost, and the questions for HR or the plan administrator.
Design statistically rigorous A/B tests for product features, UI changes, onboarding flows, and pricing experiments. Use when asked to set up an experiment, design an A/B test, calculate sample size, or interpret test results. Produces a complete test plan with hypothesis, variant definitions, sample size, duration estimate, guardrail metrics, and a results interpretation guide.
Analyse a finished A/B test and write the readout — the result, whether it's statistically and practically significant, what it means, and the ship/no-ship call. Use when asked to analyse experiment results, write an A/B test readout, interpret test data, or decide whether to ship a variant. Produces a clear verdict with the lift and confidence, segment cuts, the risks (peeking, novelty, sample), and a recommendation. Distinct from planning a test — this reads results.
Generate a WCAG 2.2 accessibility audit checklist and remediation suggestions for any UI or design. Use when asked to audit for accessibility, check WCAG compliance, review a design for a11y issues, or create an accessibility remediation plan. Produces a prioritised checklist with pass/fail assessments and specific fixes.
Plan a trip that actually works with a disability or access need — confirm real accessibility (not just 'accessible' labels), book the assistance in advance, plan for equipment and medication, and build in the contingencies for when access breaks down. Use when someone says 'plan an accessible trip', 'travelling with a wheelchair/disability', 'book assistance for my flight', or 'will this hotel actually work for me'. Produces an access-verified itinerary, an assistance-booking checklist, an equipment/medication plan, and contingency scripts. Verify specifics with providers.
Request a reasonable accommodation at work or in education — frame it around the barrier and the adjustment (not your diagnosis), cite the right process, and navigate the back-and-forth constructively. Use when someone says 'I need a workplace accommodation', 'request reasonable adjustments', 'ADA/Equality Act accommodation', or 'how do I ask for accommodations for my disability/condition'. Produces the request letter, a barriers-and-adjustments map, disclosure guidance, and a plan for the interactive process. Not legal advice — routes to the formal process and to advocacy where needed.
Build a structured account plan for any key customer or target account. Use when asked to create an account plan, key account strategy, strategic account review, or territory plan. Produces a complete account plan with relationship map, growth opportunities, risks, and 90-day action plan.