Chapter 15 · 12 min · Basic
Reading an experiment readout
A product manager ships a new checkout and sends you a two-row table. Treatment is up. They want a sentence for the Monday deck.
The table is not the experiment. It is somebody's summary of the experiment, and every summary is a stack of choices — what counts as a conversion, who is in the denominator, which of several defensible numbers to print. Those choices were made before the table reached you, usually by whoever had the SQL open, and none of them are visible in the two rows.
This chapter is the first of five on measurement, and it is the least statistical of them. No significance test appears here. The job is narrower and comes first: know what the rows are, know which number you are quoting, and know how much it would take to move it.
The dataset
One experiment on a new checkout flow: 24 users split evenly into control and treatment, the orders they placed, and readout_as_sent — the summary table a colleague actually mailed round. The raw rows and the summary disagree, and the reason is the first thing this chapter looks at.
Schema
| user_id | int |
| variant | text |
| order_id | int |
| user_id | int |
| revenue_myr | numeric(7,2) |
| variant | text |
| users | int |
| conversions | int |
| revenue_myr | numeric(8,2) |
Example data
What the table in front of you actually is
Start by reading the readout exactly as sent, and doing nothing clever to it.
Treatment converts at 41.7% against control's 25.0%, and brings in RM 31.67 per user against RM 25.00. On the face of it that is a large, clean win, and a sentence for the deck writes itself.
Now look at the shape of what you are holding. Two rows, four columns, no user ids, no order ids, no dates. There is nothing in this table you can check against anything. The word conversions is not defined anywhere on it, and the number of people who could tell you what it means without re-running the query is one.
A readout is a claim, not a measurement. Before quoting it, go and find the rows it was built from — which in this dataset are sitting right next to it.
Both arms have 12 users. What would you need to see before you would put the 41.7% in a deck?
Basic
The rest of this chapter is Basic
Learning SQL is free here, forever. Both data-science tracks are the paid half — this one is the level-up: grain, reshaping, change over time, and whether the number really moved.
RM 25/mo · cancel anytime
Still to come in this chapter
- 02One row per user, not one row per event
- 03The difference and the lift are two different claims
- 04Which revenue did you mean?
- 05How much would it take to move this?
- 06Now read one you did not build