A/B Test Results Analyzer

Prompt

Analyze my A/B test results and tell me whether to ship the variant.

Experiment name: [EXPERIMENT NAME]
Hypothesis: [WHAT YOU EXPECTED TO HAPPEN]
Primary metric: [METRIC NAME, e.g. conversion rate, revenue per user]
Secondary metrics: [LIST ANY OTHER METRICS YOU TRACKED]
Test duration: [START DATE] to [END DATE]

Results:
Control (A):
- Sample size: [N]
- Conversions / Events: [N]
- Metric value: [VALUE]

Variant (B):
- Sample size: [N]
- Conversions / Events: [N]
- Metric value: [VALUE]

Please produce:
1. **Statistical Significance** — p-value (two-tailed), confidence level
2. **Effect Size** — relative lift %, absolute difference, Cohen's d or h
3. **95% Confidence Interval** for the difference
4. **Power Analysis** — was the sample size sufficient?
5. **Novelty Effect Check** — did performance hold over the full test period?
6. **Guardrail Metrics** — did anything break?
7. **Recommendation** — SHIP / DO NOT SHIP / EXTEND TEST, with reasoning
8. **Python code** to reproduce this analysis (scipy.stats)