Here is an illustrative A/B test of two web pages. Version A converts 50 of 1,000 visitors (5%) and version B converts 70 of 1,000 (7%). A two-proportion z-test checks whether the difference is likely to be real or just chance.
| Step | Working |
|---|
| Hypotheses | H₀: the conversion rates are equal; H₁: they differ |
| Pooled proportion | (50 + 70) ÷ 2,000 = 0.06 |
| Standard error | √(0.06 × 0.94 × (1/1,000 + 1/1,000)) ≈ 0.0106 |
| Test statistic | z = (0.07 − 0.05) ÷ 0.0106 ≈ 1.88 |
| p-value (two-tailed) | ≈ 0.06 |
| Decision at 5% | Do not reject H₀: the evidence is not strong enough |
In business terms: B looks better, but with this sample the difference could be chance, so the firm should collect more data before switching. A one-tailed test, if justified in advance, would give p ≈ 0.03.