First, the app measures how far apart the two groups were before the test began. That becomes the baseline.
It then measures how far apart they were after the test started and checks whether that gap changed materially during the test period.
Pre-test gap: Dpre = Avg(Variantpre) - Avg(Controlpre)
Test gap: Dtest = Avg(Varianttest) - Avg(Controltest)
Winner signal: Effect = Dtest - Dpre
If that effect is positive, the variant improved more than the control. If it is negative, the control performed better after adjustment.
The app also compares the daily gap before the test with the daily gap during the test. This is the significance step. It calculates a p-value from those daily differences and converts that into a confidence percentage.
In notation:
Daily gap: gday = Variantday - Controlday
Confidence: Confidence % ≈ (1 - p-value) × 100
Formally, the app estimates a difference-in-differences effect and then checks whether that effect is likely to be genuine rather than random variation.
In practical terms: did the variant beat the control by enough that the result is likely to be real?
