a-b-test-design
The a-b-test-design skill structures rigorous A/B experiments by establishing clear hypotheses, isolating single variables between control and treatment variants, defining primary and secondary metrics, and calculating required sample sizes based on statistical significance and power. Use this skill when designing product experiments, feature changes, or user experience improvements where controlled comparison and quantifiable measurement can determine impact.
git clone --depth 1 https://github.com/Owl-Listener/designer-skills /tmp/a-b-test-design && cp -r /tmp/a-b-test-design/prototyping-testing/skills/a-b-test-design ~/.claude/skills/a-b-test-designSKILL.md
# A/B Test Design You are an expert in designing rigorous A/B experiments that produce actionable results. ## What You Do You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor. ## Test Structure ### 1. Hypothesis Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].' ### 2. Variants - Control (A): current design - Treatment (B): proposed change - Keep changes isolated — test one variable at a time ### 3. Primary Metric The single most important measure of success. Must be measurable, relevant, and sensitive to the change. ### 4. Secondary Metrics Supporting measures and guardrail metrics to detect unintended consequences. ### 5. Sample Size Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%). ### 6. Duration Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically. ## Common Pitfalls - Peeking at results before completion - Too many variants at once - Metric not sensitive enough to detect change - Sample size too small - Not accounting for novelty effects - Ignoring segmentation effects ## When Not to A/B Test - Very low traffic (insufficient sample) - Ethical concerns with withholding improvement - Foundational changes that affect everything - When qualitative insight is more valuable ## Best Practices - One hypothesis per test - Document everything before starting - Don't stop early on positive results - Analyze segments after overall results - Share learnings broadly regardless of outcome
Facilitate a structured team critique — framing, feedback rules, and actionable outcomes. Use when running a session with people in the room. For a solo expert review, use `heuristic-evaluation` (prototyping-testing).
Inventory and prioritise accumulated design inconsistencies across a product. Use when drift has built up over time. For token coverage specifically use `design-token-audit` (designer-toolkit); for WCAG gaps use `accessibility-audit` (design-systems).
Communicate design's contribution to business and user outcomes in stakeholder language. Use when reporting results upward. For choosing the metrics in the first place, use `metrics-definition` (ux-strategy).
Build a QA checklist for verifying that a build matches the design. Use at implementation review. For the spec engineers build from, use `handoff-spec`.
Establish review gates — criteria, checkpoints, and approval flow. Use when work ships without consistent review. For running one individual session, use `design-critique`.
Plan and facilitate a design sprint from challenge framing through prototype testing. Use when compressing discovery into days. For ongoing team cadence, use `team-workflow`.
Write the implementation handoff — measurements, behaviours, assets, states, and edge cases. Use when engineering picks up the work. For verifying the result afterwards use `design-qa-checklist`; for reusable library components use `component-spec` (design-systems).
Design the team's operating rhythm — task management, collaboration rituals, and tooling. Use when the day-to-day cadence needs structure. For a time-boxed sprint, use `design-sprint-plan`.