ab-test-setup

LeoYeAI/openclaw-marketing-skillsMarketingAudit passedSnapshot de8ab94d917c

Summary

When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," or "how long should I run this test." Use this whenever someone is comparing two approaches and wants to measure which performs better. For tracking implementation, see analytics-tracking. For page-level conversion optimization, see page-cro.

SKILL.md

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first: If .agents/product-marketing-context.md exists (or .agents/product-marketing-context.md in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

Test Context - What are you trying to improve? What change are you considering?
Current State - Baseline conversion rate? Current traffic volume?
Constraints - Technical complexity? Timeline? Tools available?

Core Principles

1. Start with a Hypothesis

Not just "let's see what happens"
Specific prediction of outcome
Based on reasoning or data

2. Test One Thing

Single variable per test
Otherwise you don't know what worked

3. Statistical Rigor

Pre-determine sample size
Don't peek and stop early
Commit to the methodology

4. Measure What Matters

Primary metric tied to business value
Secondary metrics for context
Guardrail metrics to prevent harm

Hypothesis Framework

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Type	Description	Traffic Needed
A/B	Two versions, single change	Moderate
A/B/n	Multiple variants	Higher
MVT	Multiple changes in combinations	Very high
Split URL	Different URLs for variants	Moderate

Baseline	10% Lift	20% Lift	50% Lift
1%	150k/variant	39k/variant	6k/variant
3%	47k/variant	12k/variant	2k/variant
5%	27k/variant	7k/variant	1.2k/variant
10%	12k/variant	3k/variant	550/variant

Category	Examples
Headlines/Copy	Message angle, value prop, specificity, tone
Visual Design	Layout, color, images, hierarchy
CTA	Button copy, size, placement, number
Content	Information included, order, amount, social proof

Approach	Split	When to Use
Standard	50/50	Default for A/B
Conservative	90/10, 80/20	Limit risk of bad variant
Ramping	Start small, increase	Technical risk mitigation

Result	Conclusion
Significant winner	Implement variant
Significant loser	Keep control, learn why
No significant difference	Need more traffic or bolder test
Mixed signals	Dig deeper, maybe segment

ab-test-setup

Summary

A/B Test Setup

Initial Assessment

Core Principles

1. Start with a Hypothesis

2. Test One Thing

3. Statistical Rigor

4. Measure What Matters

Hypothesis Framework

Structure

Example

Test Types

Sample Size

Quick Reference

Metrics Selection

Primary Metric

Secondary Metrics

Guardrail Metrics

Example: Pricing Page Test

Designing Variants

What to Vary

Best Practices

Traffic Allocation

Implementation

Client-Side

Server-Side

Running the Test

Pre-Launch Checklist

During the Test

The Peeking Problem

Analyzing Results

Statistical Significance

Analysis Checklist

Interpreting Results

Documentation

Common Mistakes

Test Design

Execution

Analysis

Task-Specific Questions

Related Skills

Related skills