Official skill · Product
A/B test plan
Writes an A/B test brief: hypothesis, primary and guardrail metrics, MDE, sample size and duration with the math, stop rules, and a decision rule set in advance. Use it when before launching an A/B test on a page, flow, email, or price.
AI skill
Have your AI plan an A/B test you can trust
14 days free · No card needed
Works in Claude, ChatGPT, and coding agents. On Team and Business, the official skill in your pod updates itself when we improve it.
When to use it
- Before launching an A/B test on a page, flow, email, or price
- Someone asks how long to run a test or how many users it needs
What you give it
- The change to test and the reason
- Primary metric baseline
- Weekly traffic in the test
What you get
A one-page experiment plan with hypothesis, metrics table, sample size math, duration, stop rules, a ship/don't-ship rule, and risks.
How it works
- Write a falsifiable hypothesis
- Pick one primary metric and 1 to 3 guardrails
- Set the minimum detectable effect
- Estimate sample size and duration, and show the method
- Write the stop rules
- Write the decision rule before launch
- List risks and the readout date
Example
What you give it
We want to test a shorter sign-up form (3 fields instead of 7). Sign-up rate is 4% of pricing-page visitors. About 6,000 visitors a week. We'd ship it if it adds a point.
What you get back (the start of it)
Experiment plan: Short sign-up form
Owner: [person] · Decision maker: [person] · Start: [date] · Readout: start + 3 weeks
Hypothesis
Because [check: evidence, e.g. form analytics showing drop-off at field 4], we believe a 3-field sign-up form for pricing-page visitors will increase sign-up rate. We'll know it's true if sign-up rate rises by at least 1 percentage point (4% to 5%).
Variants
- Control: 7-field form
- Variant B: 3-field form (email, password, company name [to confirm])
- Split: 50/50 · Audience: new visitors to the pricing page; logged-in users excluded
The rest of this output, the full steps and the checks the AI runs before it answers are in your pod once you add the skill.
Use this skill in your AIShare it with your team
Add it to an AI Skill Pod and everyone on the team, contractors included, uses the same version from their own AI. Read the best ways to share AI skills with a team
Changelog
- v1.0.0 · · First release.
Sources
- How Not To Run an A/B Test - Evan Miller (read 2026-11-04)
- Power Analysis - Statsig Docs (read 2026-11-04)
- Diagnosing Sample Ratio Mismatch in A/B Testing - Microsoft Research (read 2026-11-04)