Design statistically rigorous A/B tests for product features, UI changes, onboarding flows, and pricing experiments. Use when asked to set up an experiment, design an A/B test, calculate sample size, or interpret test results. Produces a complete test plan with hypothesis, variant definitions, sample size, duration estimate, guardrail metrics, and a results interpretation guide.
SKILL.md
A/B Test Planner Skill
Design experiments that produce trustworthy results — not just directional signals. Every test output includes hypothesis, success metrics, sample size, duration, and a results interpretation guide.
Required Inputs
Ask the user for these if not provided:
What is being tested (feature, UI change, copy, pricing, onboarding step)
Single primary metric (plus up to 2 guardrail metrics)
Minimum detectable effect (MDE) defined
Sample size calculated
Test duration estimated
Segment isolated (no overlap with other running tests)
Rollback plan defined
Hypothesis Template
"We believe that [change] will cause [primary metric] to [increase/decrease] by [X%] for [user segment], because [rationale based on data or insight]."
Never run a test without a directional hypothesis. "Let's just see what happens" is not a hypothesis.
Sample Size Calculator Logic
Use this formula (provide the output, not the formula, to the user):
Baseline conversion rate: Current rate of primary metric
MDE: Smallest change worth detecting (recommend 10–20% relative lift for most features)
Statistical power: 80% (standard)
Significance level: 95% (p < 0.05)
For common scenarios, provide pre-calculated estimates:
Baseline Rate
MDE (Relative)
Required Sample per Variant
5%
20%
~19,000
10%
15%
~14,000
20%
10%
~15,000
40%
10%
~9,500
60%
5%
~42,000
Always warn: "These are estimates. Use a tool like Evan Miller's calculator or Statsig for precision."
Flag if traffic is too low to reach significance in under 8 weeks — recommend a different approach (e.g., holdout test, qualitative research).
Output Format
A/B Test Plan — [Test Name] — [Date]
Hypothesis:
[Filled hypothesis template]
Variants:
Control (A): [Current experience]
Treatment (B): [Changed experience — be specific]
Primary Metric: [Metric name + how measured]
Guardrail Metrics: [Metrics that must not degrade]
Target Segment: [Who sees the test — % of traffic, user type]
Traffic Split: [50/50 recommended unless ramp-up needed]
Sample Size Required: ~[N] users per variant
Estimated Duration: [X] weeks (based on [Y] daily eligible users)
Significance Threshold: 95% confidence, 80% power
Exclusions: [Any user segments to exclude and why]
Rollback Trigger: If [guardrail metric] degrades by [X%], stop the test immediately.
Results Interpretation Guide:
✅ Ship if: Treatment shows [X%]+ lift on primary metric at 95% confidence AND guardrail metrics are stable
🔄 Iterate if: Direction is positive but not significant — consider extending or redesigning
❌ Reject if: No lift or negative direction at significance
⚠️ Inconclusive: Do not ship. Do not call it a win.
Guidelines
Always recommend against peeking at results before the test reaches planned sample size — explain p-hacking risk
If user wants to test multiple variants, explain the multiple comparisons problem and recommend a Bonferroni correction or a Bayesian approach
If traffic is very low (<1,000 users/day), recommend qualitative alternatives: moderated testing, 5-second tests, or user interviews
Never approve a test with no guardrail metrics — always protect revenue, retention, or core engagement
Anti-Patterns
Do not run a test without a directional hypothesis — "let's see what happens" produces uninterpretable results
Do not declare a winner before reaching the pre-planned sample size — peeking at results inflates false positive rates
Do not test multiple independent changes in a single variant — you won't know which change caused the result
Do not use engagement metrics (clicks, time-on-page) as the primary metric when the goal is revenue or retention — proxy metrics mislead
Do not ignore guardrail metrics — a conversion lift that causes a support ticket spike is not a win
Deeper Materials
This skill ships with support files — use them when they are available:
references/test-validity-traps.md — The Validity Traps That Quietly Invalidate A/B Tests. Apply it while producing the output; it carries the calibration and judgment calls the method summary above compresses.
templates/test-plan.md — a fill-in version of the deliverable with the quality gates inline. Offer it when the user wants to work the document themselves rather than have it generated.
Scoring Rubric (0–40)
Score any output of this skill before handing it over; 32+ is ship-quality.
Dimension
0
5
10
Statistical rigour
No sample size, or a number with no stated baseline/MDE behind it
Sample size present but MDE is guessed or copied from the lookup table without checking the actual baseline; power/significance unstated
Sample size derived from the stated baseline and MDE at 80% power / 95% confidence, duration checked against real daily traffic and the 2–4 week window, and the low-traffic escape hatch invoked if it doesn't fit
Hypothesis discipline
"Let's see what happens" — no direction, no magnitude, or multiple changes bundled into one variant
Directional hypothesis but missing magnitude, segment, or the evidence-based because; variant purity not confirmed
Full template filled (change, metric, direction, magnitude, segment, rationale citing data), and the treatment isolates exactly one change with excluded ideas named as follow-up tests
Guardrails & rollback
No guardrail metrics, or a rollback line with no threshold
Guardrails named but denominators/definitions ambiguous; rollback trigger vague ("if things look bad")
1–2 guardrails protecting revenue or core engagement with pre-agreed definitions, concrete rollback thresholds, and the peeking-vs-harm-monitoring distinction handled explicitly
Decision readiness
No interpretation guide; results will be argued about after the fact
Ship/iterate/reject listed but thresholds fuzzy; inconclusive outcome missing or treated as a soft win
All four outcomes (ship / iterate / reject / inconclusive) mapped to pre-committed thresholds, including what an inconclusive result costs and what each outcome changes next
Quality Checks
Hypothesis is directional (predicts a specific direction and magnitude, not "let's see")
Primary metric is singular (guardrail metrics are secondary)
Sample size is calculated from actual MDE and baseline (not guessed)
Test duration accounts for weekly seasonality (minimum 2 weeks)
Guardrail metrics are defined (at least one to protect revenue or core engagement)
Rollback trigger is specified with a concrete threshold
