Chapter 16 · 13 min · Basic
Was the split even?
The previous chapter took a readout apart and rebuilt it honestly. This one asks the question that comes before that: was there a valid experiment at all?
Randomisation is the entire reason an A/B test means anything. Every claim of the form "the change caused the lift" rests on the two groups having been identical except for the change. Nothing enforces that. A bucketing service can fail halfway, a logging call can fire on one arm and not the other, a user can be re-bucketed on their second visit — and the resulting table looks exactly like a healthy one. Two variants, plausible counts, a difference in the metric.
There are two checks, both cheap, and you run them before you look at the outcome. Is every user in exactly one arm, and are the arms the size they were configured to be? In this chapter the answer to both is no, and the readout swings by twelve percentage points once they are fixed.
The dataset
One experiment configured as a 50/50 split of 120 users. bucketing_service_log is what the assignment service decided; assignment_log is what the analytics table actually received; orders holds the conversions. chi_square_critical is a seeded lookup — Postgres has no chi-square distribution function, so the critical values are stored as rows, exactly as you would read them off a printed table.
Schema
| user_id | int |
| variant | text |
| row_id | int |
| user_id | int |
| variant | text |
| user_id | int |
| revenue_myr | numeric(7,2) |
| df | int |
| alpha | numeric(4,3) |
| critical_value | numeric(6,3) |
Example data
Rows are not users
Every check in this chapter is a count, so the first job is knowing what you are counting.
assignment_log holds 104 rows and 100 distinct users. Those are different numbers, and until you know why, every rate you compute out of this table has an unknown denominator.
Four extra rows is not a rounding detail. Assignment logs are append-only and written by whatever code path happens to fire, so duplicates are the normal state of one: a retry, a page reload, a second session, a re-bucketing. Some of those are harmless. One of them is not.
Count rows and users as separate columns in every experiment query you write. It costs one count(distinct ...) and it is the only reason you will notice.
The experiment was configured for 120 users split evenly. Before running this — what would worry you more, too many rows or too few?
Basic
The rest of this chapter is Basic
Learning SQL is free here, forever. This track is the paid half: what to do when the data is dirty, duplicated and undocumented, and somebody still wants a number.
RM 25/mo · cancel anytime
Still to come in this chapter
- 02The same user in both arms
- 03Sample ratio mismatch
- 04The result you must not ship yet
- 05Where the missing users went
- 06What the experiment actually said
- 07Now run the checks where nobody kept a second log