19-ab-test-setup
This Claude Code skill guides users through designing and executing statistically valid A/B tests for marketing materials including ads, landing pages, emails, and pricing. It helps teams without data science expertise select test variables, calculate required sample sizes, establish tracking methods, and analyze results for statistical significance by following seven core A/B testing principles and providing formulas for determining adequate test duration and participant numbers.
git clone --depth 1 https://github.com/minhnv0807/ai-business-skills /tmp/19-ab-test-setup && cp -r /tmp/19-ab-test-setup/skills/vi/19-ab-test-setup ~/.claude/skills/19-ab-test-setupSKILL.md
# A/B Test Setup ## Thu thap thong tin ### Buoc 0: Kiem tra context Doc `.agents/product-marketing-context.md` neu co. ### Buoc 1: Hoi user 1. **Test cai gi?** (Headline ads / Landing page / Email subject / Gia / CTA button / Creative video) 2. **Metric chinh?** (CTR / Conversion rate / CPMess / Revenue / Open rate) 3. **Traffic hang ngay cua trang/ads?** (De tinh sample size can thiet) 4. **Muc tieu test?** (Tang X% metric chinh / Tim ra version tot nhat) --- ## 7 Nguyen tac A/B test ### 1. Test 1 bien duy nhat **Quy tac cot loi.** Neu test 2 bien cung luc → khong biet bien nao tac dong. - XAU: Doi headline + doi CTA + doi anh → khong ket luan duoc - TOT: Chi doi headline (A vs B), moi thu khac giu nguyen ### 2. Gia thuyet co the kiem dinh **Format:** "Neu lam [X], metric [Y] se tang [Z%]" - TOT: "Neu doi CTA tu 'Dang ky' sang 'Nhan tu van mien phi', conv rate tang 15%" - XAU: "Copy moi tot hon" (khong cu the, khong do duoc) ### 3. Sample size du **Khong duoc dung test som.** - Toi thieu: **100 conversion moi variant** (khong phai 100 visitor) - Tot nhat: Tinh truoc (xem cong thuc ben duoi) ### 4. Thoi gian du **Test tung tuan tron** (khong 3 ngay, khong 10 ngay) — khach mua hanh vi khac nhau cac ngay. - Toi thieu: 7 ngay - Nen co: 14 ngay ### 5. Khong peek som **Khong check ket qua moi gio.** Dung ket qua cho den khi het thoi han + du sample. - Peek som + stop som = false positive. ### 6. Significance > 95% **Chi tin khi p-value < 0.05.** Neu khong: ngau nhien, khong phai do bien cua ban. ### 7. Documented **Ghi lai:** - Gia thuyet - Ngay bat dau / ket thuc - Sample size - Ket qua - Ket luan → De sau nay ban khong test lai cai da test. --- ## Cong thuc tinh Sample Size ### Cong thuc don gian ``` Sample size/variant = 16 × p × (1-p) / MDE² Trong do: p = Baseline conversion rate hien tai (vi du 0.02 = 2%) MDE = Minimum Detectable Effect (muc do thay doi toi thieu ban muon phat hien, vi du 0.2 = 20% tang) ``` ### Vi du **Landing page hien co conv rate 3%, muon phat hien thay doi 20%+:** - p = 0.03 - MDE = 0.2 × 0.03 = 0.006 (tuc tang tu 3% len 3.6%) - Sample size = 16 × 0.03 × 0.97 / 0.006² = 12,933 visit/variant => Can 25,866 visit total, neu 500 visit/ngay → 52 ngay. **Landing page hien conv rate 10%, muon phat hien 20%+:** - p = 0.10 - MDE = 0.02 (tu 10% len 12%) - Sample size = 16 × 0.10 × 0.90 / 0.02² = 3,600/variant => 7,200 visit total, 500/ngay → 14 ngay (kha thi). ### Quy tac nhanh | Traffic/ngay | Conv rate | Thoi gian can | Co nen test? | |-------------|-----------|--------------|-------------| | < 100 visit | Bat ky | 2+ thang | Khong — traffic thap | | 100-500 | 2-5% | 3-6 tuan | Co, nhung kien nhan | | 500-2K | 2-5% | 2-3 tuan | Co — ly tuong | | 2K+ | Bat ky | 1-2 tuan | Co — nhanh | **Neu traffic thap** (< 100/ngay) → khong test, thay vao do tap trung tang traffic truoc. --- ## 8 Yeu to nen test (uu tien) ### 1. Headline **Tac dong cao nhat.** 80% nguoi doc headline, 20% doc body → headline thay doi toi uu = bien nho nhat. Idea: - Question vs Statement - Number specific vs generic - Outcome-focused vs Feature-focused - Short (5-7 tu) vs Long (12-15 tu) ### 2. CTA Button **Do thay doi duoc de test.** Thuong tang 5-25%. Test: - Text: "Dang ky" vs "Nhan tu van mien phi" vs "Bat dau ngay" - Color: Xanh primary vs Do vs Cam - Size: Standard vs Lon - Position: Tren cung vs Cuoi trang ### 3. Hero Image / Video Test: - Anh san pham vs Anh nguoi dung - Video vs Anh tinh - Testimonial video vs Demo video ### 4. Price Display Test: - 299K vs 299,000 vs 299.000đ - Co discount strikeout vs khong - Monthly vs Yearly highlight ### 5. Social Proof Test: - So con so vs Review chi tiet - Logo bao chi vs So khach - Testimonial video vs text ### 6. Form Fields Test: - 3 fields vs 5 fields vs 7 fields - Label tren vs ben trai - Placeholder example vs khong co ### 7. Email Subject Line Test: - Question vs Benefit - Personalization ([Ten]) vs Khong - Emoji vs Khong (VN: khong emoji tot hon) ### 8. Ad Creative Test: - Hook 3s dau khac nhau - UGC vs Brand - Format: Single image vs Carousel vs Video --- ## Setup tracking ### Tool khuyen dung | Tool | Phu hop | Chi phi | |------|---------|---------| | **Meta Ads A/B test built-in** | Test ads (creative/audience/placement) | Free | | **TikTok Ads Split test** | Test TikTok ads | Free | | **Google Optimize** (deprecated) | Da ngung — dung thay the: VWO, Convert.com | Tu $199/thang | | **PostHog** | Test product features, landing | Free tier OK | | **Unbounce / Instapage** | Landing page built-in A/B | $90+/thang | | **Manual (split URL)** | Don gian, 50/50 URL redirect | Free | ### Setup don gian (no-tool) ``` 1. Tao 2 phien ban landing page: /page-a va /page-b 2. Split traffic 50/50: - Qua Meta Ads: 2 ad set cung audience, khac link dich - Qua Google Ads: 2 ad cung traffic, khac link - Qua organic: dung URL rewrite (rare) 3. Tracking conversion rieng tung page: - Meta Pixel: event "Lead" voi custom parameter "page_version" = A/B - GA4: UTM campaign=test-header, utm_content=a/b 4. Sau 14 ngay + du sample, export data ``` --- ## Phan tich ket qua ### Tinh significance (p-value) **Dung calculator online:** https://www.evanmiller.org/ab-testing/chi-squared.html **Input:** - Variant A: Visitor count + Conversion count - Variant B: Visitor count + Conversion count **Output:** - p-value: Neu < 0.05 → significant (95% confidence) - Uplift: Tang bao nhieu % ### Ket luan | p-value | Uplift | Ket luan | |---------|--------|---------| | < 0.05 | > 5% | B thang — implement B | | < 0.05 | < 5% | Significant nhung nho — can nhac cost | | 0.05-0.10 | > 10% | Ket qua borderline — keo dai test | | > 0.10 | Bat ky | Khong significant — giu A hoac test tiep | ### Bay thuong gap 1. **Peek som roi stop.** Test ngan → ket qua ngau nhien. 2. **Sample khong deu.** Neu split 30/70 thay vi 50/50 → kiem tra xem truyen thong co van de. 3. **Seasonality.** Test T6 Chu nhat khac T2-T3 → luon test du tuan
Ke hoach Fullstack Marketing — master skill tong hop chien luoc, noi dung, trien khai, hieu suat, va timeline. Goi cac skill con de hoan thien output.
Use when the user needs an overall marketing plan for a period — objectives, segments, positioning, channel mix, content direction, budget, timeline, KPIs, and a risk matrix, adapted to US, EU, SEA, or LATAM. Trigger on 'marketing plan', 'marketing strategy for next year', 'go-to-market roadmap', 'plan for Q3', 'we have no marketing plan', 'where should we spend next quarter'. Also use when a founder only says the business needs to grow and has no idea which channel to start with. Not for — one specific campaign, see `02-campaign-brief-global`; working backward from a revenue goal to spend, see `10-reverse-kpi-global`; splitting a budget across channels and months, see `61-budget-planning-global`; taking a brand new product to market, see `59-go-to-market-global`.
Use when the user needs a monthly or weekly content calendar — posting schedule per channel, funnel ratio, content pillars, repurposing matrix, and an owner per slot. Trigger on 'content calendar', 'posting schedule', 'what should we post this month', 'editorial calendar', 'social media plan', 'we post randomly with no plan'. Also use when the user only says the team keeps running out of things to post. Not for — the brief for one specific post, see `36-content-brief-global`; the caption text itself, see `37-social-caption-global`; deciding next period content from last period data, see `40-next-content-plan-global`; a paid campaign timeline, see `02-campaign-brief-global`.
Lich noi dung thang — len lich dang bai da kenh voi ti le pheu, content pillar, repurposing matrix, va scoring tieu chi.
Brief chien dich marketing — 9 phan day du tu Context den Risk, bao gom creative direction, RACI, budget allocation theo phase, va auto-trigger cac skill lien quan.
Use when a specific campaign needs a brief the whole team can execute from — context, objective, audience, core message, creative direction, channels, deliverables, week-by-week timeline, approval gates, and risks. Trigger on 'campaign brief', 'creative brief', 'launch brief', 'brief the team on this campaign', 'we are running a promo next month', 'what is the plan for this campaign'. Also use when the user describes a promotion out loud and asks for something the team can actually run. Not for — the whole period plan, see `00-marketing-plan-global`; the list of design files needed, see `41-campaign-asset-list-global`; assigning work to one person, see `64-team-brief-global`; a new product launch sequence, see `59-go-to-market-global`.
Danh gia hieu suat marketing — audit performance ads va organic, chan doan root cause, de xuat toi uu voi action plan 48h va checklist hang tuan.
Use when marketing numbers look bad and the user needs to know WHY — layered root-cause diagnosis across tracking, delivery, creative, landing page, offer, and audience quality, 5-Whys, regional benchmarks for US, EU, SEA, and LATAM, and a 48-hour action plan. Covers dropshipping ROAS and BE-ROAS. Trigger on 'why is CPA so high', 'my ads are not converting', 'ROAS dropped', 'we are burning budget', 'performance is bad this month', 'diagnose the funnel'. Also use when the user pastes a screenshot of numbers with no question attached. Not for — auditing account configuration and scoring health, see `21-ads-audit-global`; writing a report someone else reads, see `07-marketing-report-global`; turning a raw export into insight, see `13-data-analysis-global`.