Experiment Readout Skill

A test result is only a decision if the statistics are sound — and "variant looks higher" is not a result. This skill computes the lift, the p-value, and a confidence interval from the raw counts, checks the guardrails, and writes an honest readout with a clear ship/no-ship call — flagging the traps (peeking, underpowered, novelty, a significant but tiny effect) that make teams ship noise.

Required Inputs

Ask for these only if they aren't already provided:

• The metric & data — for a conversion test: users and conversions per variant (control vs. treatment). For a continuous metric: mean, SD, and n per variant.
• The hypothesis — what you expected and the minimum effect that matters.
• Guardrail metrics — what shouldn't get worse (revenue, latency, retention).
• Test setup — planned sample size/duration, and whether it ran to plan (for the peeking check).

Output Format

Experiment Readout: [test name]

1. Result — computed (use the helper): control vs. treatment rate, absolute & relative lift, p-value, and the confidence interval on the difference.

Variant	N	Conversions	Rate
Control
Treatment

→ Lift: X% (CI: [a%, b%]) · p = 0.0xx

2. Verdict — significant at the stated bar or not, and whether the effect is big enough to matter (a significant +0.2% may not be worth the complexity). Distinguish statistical from practical significance.

3. Guardrails — did anything you promised not to harm move? A win that tanks a guardrail isn't a win.

4. Validity checks — was it run to the planned sample (no peeking/early-stopping)? Sample-ratio mismatch? Novelty/seasonality? Call out anything that undermines the result.