ab-test-readout
Analyse a finished A/B test and write the readout — the result, whether it's statistically and practically significant, what it means, and the ship/no-ship call. Use when asked to analyse experiment results, write an A/B test readout, interpret test data, or decide whether to ship a variant. Produces a clear verdict with the lift and confidence, segment cuts, the risks (peeking, novelty, sample), and a recommendation. Distinct from planning a test — this reads results.
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills /tmp/ab-test-readout && cp -r /tmp/ab-test-readout/plugins/pm-data/skills/ab-test-readout ~/.claude/skills/ab-test-readoutSKILL.md
# A/B Test Readout Skill The hard part of an experiment is the readout: not "B won" but "is this real, is it big enough to matter, and should we ship?" This skill turns results into an honest decision — and flags the ways A/B results lie. ## Working from a brief Given results (even partial), **write the full readout anyway**. If significance isn't provided, reason about it from the numbers and flag what's needed to confirm. Mark assumed figures. Never declare a winner without addressing significance and sample. ## Required Inputs Ask for (if not already provided): - **The hypothesis** and the **primary metric** - **Results** — control vs variant: conversions/rate, sample size per arm, duration - **Guardrail metrics** (revenue, retention, latency, complaints) that mustn't regress - **Pre-registered decision rule** (what would count as a win) if one exists ## Output Format ### 1. Verdict (one line) *Ship / Don't ship / Inconclusive — keep running* — with the headline number. ### 2. The result | Metric | Control | Variant | Relative lift | Significant? | |---|---|---|---|---| | Primary | | | | p / CI | | Guardrail(s) | | | | | State **statistical** significance (p-value / confidence interval) *and* **practical** significance (is the lift big enough to matter given the cost?). ### 3. Did it really win? Address the ways A/B tests mislead: - **Sample / power** — was the test adequately powered, or under-sampled? - **Peeking** — was the call made early, inflating false positives? - **Novelty / primacy** — could the effect fade? - **Segments** — does the win hold across key segments, or is it driven by one? ### 4. Segment cuts Where the effect is strong vs flat vs negative (new vs returning, platform, geography). ### 5. Recommendation & next step Ship / iterate / re-run, plus what to monitor post-launch or what the follow-up test should isolate. ## Quality Checks - [ ] Distinguishes statistical from practical significance - [ ] Checks guardrail metrics, not just the primary - [ ] Flags peeking, power, novelty, and segment-driven wins - [ ] Recommendation follows from the evidence, with a monitoring/next-test step - [ ] Doesn't declare a winner on an underpowered or peeked result ## Anti-Patterns - "B won by 8%!" with no significance or sample size - Calling a result early (peeking) and shipping - Ignoring a guardrail regression because the primary went up - A statistically significant but practically meaningless lift treated as a win
Conduct a structured ethical review of an AI or ML feature, model, or product. Use when preparing to deploy an AI system, assessing algorithmic risk, auditing a model for bias, or producing a responsible AI impact assessment. Produces a structured ethics review covering fairness, transparency, privacy, safety, accountability, and societal impact with a risk tier score, pre-deployment checklist, and prioritised mitigations.
Structure AI and ML product decisions with the rigour of any product decision. Use when building AI-powered features, evaluating LLM integrations, designing AI products, or assessing AI readiness. Produces a complete AI product canvas covering problem definition, model approach, data requirements, evaluation framework, UX design, responsible AI checklist, and launch monitoring plan.
Transform feature briefs into structured design briefs that give designers the context they need before opening Figma. Use when asked to write a design brief, create a design handoff, brief a designer on a new feature, or translate a PRD into design requirements. Produces a brief with user goal, emotional context, success criteria, constraints, edge cases, and out-of-scope boundaries.
Design statistically rigorous A/B tests and interpret experiment results. Use when asked to design an experiment, run an A/B test, calculate sample size, interpret test results, or assess whether an experiment was successful. Produces a complete experiment design with hypothesis, sample size, run time, success criteria, and risk flags — or a results interpretation with ship/iterate/kill recommendation.
Synthesises user signals from multiple research sources into a unified, weighted insight brief. Use when you have data from interviews, support tickets, NPS verbatims, app reviews, or sales calls and need to reconcile contradictions, surface the underlying need behind requests, or answer 'what are users really telling us'. Produces ranked insights with confidence ratings, source weighting rationale, divergent signal analysis by user segment, and a research gap identification section.
Structure a product data analysis, metric deep-dive, funnel analysis, or cohort study. Use when asked to analyse product metrics, investigate a drop in conversion, explain a data change to stakeholders, or find the root cause of a metric movement. Produces a structured analysis with question, root cause, confidence level, and recommended action.
Interpret product metrics against goals and surface actionable signals. Use when asked to analyse product health, review key metrics, investigate a performance issue, produce a health report, or assess product-market fit signals. Produces a structured health report with RAG status, trend analysis, root cause hypotheses, and prioritised actions.
Structure a retention analysis, churn investigation, or engagement deep-dive for any product team. Use when asked to analyse user retention, investigate churn, measure DAU/MAU, or build a retention improvement plan. Produces a retention snapshot with root cause hypotheses, aha-moment correlation, and prioritised interventions.