Short version:
"Did one version beat the other by enough that the result is unlikely to be random?"
First, the tool measures how far apart the two groups were before the test began. That becomes the baseline.
It then measures how far apart they were after the test started and checks whether that gap changed materially during the test period.
Pre-test gap: Dpre = Avg(Variantpre) - Avg(Controlpre)
Test gap: Dtest = Avg(Varianttest) - Avg(Controltest)
Winner signal: Effect = Dtest - Dpre
If that effect is positive, the variant improved more than the control. If it is negative, the control performed better after adjustment.
The tool also compares the daily gap before the test with the daily gap during the test. This is the significance step. It calculates a p-value from those daily differences and converts that into a confidence percentage.
Daily gap: gday = Variantday - Controlday
Confidence: Confidence % ≈ (1 - p-value) × 100
Formally, the tool estimates a difference-in-differences effect and then checks whether that effect is likely to be genuine rather than random variation.
