cro-optimization

# CRO Optimization

Run conversion rate optimization as a structured discipline: audit → hypothesize → test → decide. Stack-agnostic. Tool-agnostic.

This skill is for running tests against existing pages and flows. For writing landing page copy from scratch, use `landing-page-copy`. For setting up the analytics that make CRO possible, use `analytics-strategy`.

---

## When to use

- Converting traffic at lower rate than expected
- Specific funnel step has high drop-off
- Pages with high traffic that could move the needle if optimized
- A/B testing infrastructure exists (or can be set up)
- Statistical significance and sample size questions

## When NOT to use

- Without sufficient traffic to test (under ~5,000 monthly conversions per variant)
- Pre-launch (no users to test on yet)
- Strategy or messaging-level questions that need qualitative research first
- Brand-defining choices (CRO can't optimize a fundamentally wrong brand)

---

## Required inputs

- The page or flow under optimization
- Current conversion rate and traffic volume
- Access to analytics (event tracking, funnel data)
- An A/B testing tool (or willingness to set one up)
- Time and budget for testing (typically 2 to 8 weeks per test)

---

## The framework: 4 phases

### 1. Audit

Diagnose before treating.

**Quantitative audit:**

- **Funnel data.** Where are users dropping off? The biggest drop is the biggest opportunity.
- **Segmentation.** Does the funnel perform differently by source, device, geography, audience type?
- **Performance data.** Are slow pages dragging conversions?
- **Search Console / on-site search.** What are users looking for that they can't find?

**Qualitative audit:**

- **Session replay.** Watch 20+ sessions of users on the target flow. Note friction, confusion, hesitation.
- **Heatmaps.** Where do users click? Where do they scroll? Where do they not?
- **User interviews / surveys.** Why did users not convert? Survey people who started but abandoned.
- **Form analytics.** Which fields cause abandonment? Which cause errors?
- **Customer support tickets.** What conversion-related questions come in?

**Heuristic audit:**

- Apply CRO heuristics to the flow:
  - Is the value proposition clear in 5 seconds?
  - Is there a single primary CTA per page?
  - Is the form length appropriate to the offer?
  - Is the trust/social proof present?
  - Are objections handled?
  - Is the page accessible? (Accessibility issues hurt conversion silently.)

The audit produces a list of suspected friction points. Each becomes a hypothesis candidate.

### 2. Hypothesis

A testable statement.

**Hypothesis structure:**

> Because [observation from audit], we believe that [change] will produce [predicted outcome] for [user segment], because [reason].

**Example:**

> Because session replays show users abandoning at the shipping step (audit), we believe that adding visible shipping cost to the product page (change) will increase add-to-cart conversion by 5 percent (outcome) for desktop users (segment), because users are surprised by shipping cost and abandon (reason).

**Hypothesis quality criteria:**

- Specific change (not "improve the design")
- Measurable outcome (with a target)
- Grounded in evidence (audit, research, prior tests)
- Tied to a known mechanism (why would this work?)

**Hypothesis prioritization (ICE or PIE):**

- **Impact:** How much could this move the metric?
- **Confidence:** How likely is the hypothesis to be right?
- **Ease:** How easy to test? (Time, complexity, risk)

Score each 1 to 10. Highest combined scores test first.

### 3. Test design

A test that produces an unambiguous answer.

**Sample size and duration:**

Use a sample size calculator (most A/B tools have one) before launching. Inputs:

- Baseline conversion rate
- Minimum detectable effect (the smallest lift you'd care about)
- Statistical power (typically 80%)
- Significance level (typically 95%)

This produces required sample size per variant. Run the test until that sample is reached, OR for a minimum duration that captures full business cycle (typically 2 weeks minimum, to cover weekends and weekly patterns).

**Common test setup mistakes:**

- Stopping the test the moment significance is hit (peeking)
- Running tests for too short to capture a full business cycle
- Running multiple overlapping tests on the same flow
- Testing during atypical periods (Black Friday, holidays, major campaigns)
- Excluding mobile when 50%+ of traffic is mobile (or vice versa)
- Testing on too small a slice of traffic (low statistical power)
- Not segmenting analysis (overall lift can hide negative impact on a segment)

**Test parameters to define before launch:**

- Primary metric (one)
- Guardrail metrics (do not go down)
- Sample size
- Duration (minimum and maximum)
- Decision criteria (when to ship, when to kill, when to extend)
- Segments to analyze in addition to overall

### 4. Decide

After the test concludes.

**Decision framework:**

| Outcome | Decision |
|---|---|
| Variant clearly wins (>95% significance, exceeds minimum effect) | Ship variant. Document. Continue testing. |
| Variant clearly loses | Kill. Capture the lesson. Iterate hypothesis. |
| Inconclusive (neither significant) | Larger test, different angle, or move on. Don't ship "tied" variants. |
| Small lift, lots of variance | Probably not worth shipping. Even if "winner," may not replicate. |
| Wins overall, loses for important segment | Investigate segment. Consider segment-specific solution. |

**Anti-patterns:**

- "It looks like it's winning, ship it" before reaching significance
- Shipping a variant because the team wants to (HiPPO - highest paid person's opinion)
- Killing tests too early because they look bad
- Re-running tests until they "win" (false positive risk)
- Not capturing the learning when a test loses

---

## Statistical foundations

### Significance and confidence

A 95% significance level means: if there were truly no difference between variants, th