Chapter 17 · 12 min · Basic
Distributions before means
The split was even and every user is in exactly one arm. Now somebody wants revenue per user, and AVG is right there.
The mean is the correct summary for a symmetric distribution with no long tail. Almost nothing an analyst measures is that. Revenue, session length, order value, time to first purchase, items per basket — all of them are piles of small numbers with a few enormous ones on the right, and the mean of that shape is a number that describes almost none of the people in it.
This chapter has one instruction and several ways of carrying it out: look at the distribution before you summarise it. In this dataset the mean says treatment tripled revenue and every other summary says treatment lost, and the disagreement is one user.
The dataset
Forty users in an even A/B split, with the total revenue each one produced over the test. Several users spent nothing, which is normal and not a data problem. One user in treatment spent RM 2,000, which is also not a data problem, and is the entire subject of this chapter.
Schema
| user_id | int |
| variant | text |
| revenue_myr | numeric(8,2) |
Example data
The mean says treatment tripled revenue
Twenty users per arm, one AVG, and a result that would end any discussion: RM 51.75 per user in control against RM 138.50 in treatment. A 167.6% lift.
This is the number that gets shipped. It is correctly computed, the arms are the same size, the denominator is the randomised population — everything chapter 15 asked for. There is no error to find.
And it is still wrong, in the sense that it will lead to a decision that loses money. Not because the arithmetic failed, but because the mean answers "if the total were shared out equally, how much each?" and nobody asked that question. What was actually asked is closer to "did users spend more", and those two questions come apart the moment one user is different from the others.
Everything below is the work of finding out whether they came apart here.
A 167.6% lift from a product change. Higher or lower than any real checkout experiment you would believe?
Basic
The rest of this chapter is Basic
Learning SQL is free here, forever. This track is the paid half: what to do when the data is dirty, duplicated and undocumented, and somebody still wants a number.
RM 25/mo · cancel anytime
Still to come in this chapter
- 02Look at the shape before you summarise it
- 03Buckets hide it too, if the top one is open
- 04Trimming, and where you put the cut
- 05Zero is a data point
- 06Report the disagreement, not the winner
- 07Now find the shape you were not shown