Chapter 19 · 14 min · Basic
The four ways a readout fools you
Assume everything the last four chapters asked for. The rows are people, the denominator is the randomised population, the split is even, nobody is in both arms, the distribution is not hiding a whale, and the result clears 1.96.
All four traps below survive that. Each one produces a readout that passes every check so far and still points the wrong way, and each one is found by slicing the result along an axis the summary collapsed — time, calendar week, segment, or a second metric nobody was asked to report.
One experiment runs through all four. It is a checkout change, four weeks, staged rollout. On 8 March it is a significant win. By 29 March it is a significant loss, and each of the four sections below is a different reason the first reading was wrong.
The dataset
One checkout experiment over 28 days, one row per day per arm, carrying users, conversions and cancellations. The rollout was staged — treatment held 10% of traffic in week 1 and 50% afterwards — and week 1 also ran a launch promotion, so both arms converted better in it. Neither fact is recorded anywhere except in the counts.
Schema
| day | date |
| variant | text |
| users | int |
| conversions | int |
| cancellations | int |
Example data
Trap one: you looked early
Compute the cumulative conversion rate for each arm after every day, and run the z-test from the previous chapter on each running total.
On 8 March the statistic reads 2.151. That clears 1.96, the dashboard turns green, and a reasonable person ships the feature. Three weeks later the same statistic reads −0.807, and the point estimate has crossed from +3 points to −0.31.
Nothing went wrong on 8 March. The arithmetic was correct and the threshold was met. The problem is that a 5% threshold means a 5% chance of a false positive per look, and checking every day for four weeks is twenty-eight looks. Under repeated peeking a test with no real effect crosses 1.96 somewhere in the run far more often than not — the threshold is only a 5% guarantee if you test once, at a time fixed in advance.
The defences are all forms of deciding when to stop before you start: fix the end date, fix the sample size (the query that sizes it takes ten seconds), or use a sequential test with thresholds built for repeated looks. The one that does not work is the natural one — looking daily and stopping when it goes green.
The final result is negative. Do you expect the running z-score ever crossed +1.96 on the way there?
Basic
The rest of this chapter is Basic
Learning SQL is free here, forever. Both data-science tracks are the paid half — this one is the level-up: grain, reshaping, change over time, and whether the number really moved.
RM 25/mo · cancel anytime
Still to come in this chapter
- 02Trap two: the first week was not the product
- 03Trap three: it wins every week and loses overall
- 04Trap four: the metric nobody put in the readout
- 05The reading that survives all four
- 06The end of the track, and where it goes next