When pouring tea, do you add the tea first or the milk first? If you think it can’t possibly matter, you’re unfortunately wrong — as Dr. Ronald Fisher proved at an innocuous tea party where he conducted an experiment that changed statistical science forever.
Our story took place in the 1920s, in Rothamsted, England. The exact year doesn’t matter, but the time does, because it was four o’clock in the afternoon, and that was tea time, as far as the group of academics gathered in Rothamsted was concerned. Among the academics was Dr. Blanche Muriel Bristol, a biologist studying algae, who, when offered a cup of tea by a colleague, turned it down. The man had poured the tea into the tea cup first, and Dr. Bristol only cared for tea when the milk was poured into the tea cup first, and the tea afterwards.
The man doing the offering was Dr. Ronald Aylmer Fisher, a biologist and mathematician. He was taken aback and laughed, saying that surely she could not tell the difference. Dr. Bristol insisted that she could, and so the group set out to test Dr. Bristol’s abilities.
Testing Bristol was not as simple as some people assume. The group wanted an experiment designed to be without any complicating factors, and with results that gave them confidence in either her abilities, or the lack thereof. What they did that afternoon isn’t entirely known. What is known is Fisher’s formal design for the experiment, which he published in 1935. It formed the central example in his book, The Design of Experiments, which was lauded for its explanation of the importance of randomization, its clarity, and its determination of what results would be considered acceptable evidence to prove or disprove a claim.
In the case of the “lady tasting the tea,” Fisher believed that she could not tell the difference between milk being added before the tea or after the tea. Bristol was going to have to disprove him. How many cups would it take, and how many would she have to get right? Fisher decided that Bristol would be presented with eight cups of tea with milk: four in which the milk preceded the tea, and four in which the milk had succeeded the tea, all prepared in as uniform a way as possible. She would sort them into two groups of four. And she needed to get every single one right.
To understand why, let’s look at how many ways there is to be wrong in the experiment. The eight cups were presented in random order, but for simplicity’s sake, let’s assume that in our experiment, the first four cups are milk-first, and the second four cups are tea-first. There’s only one way to get them right; sorting first four cups into the milk-first pile, and the last four cups into the tea-first pile. There’s also only one way to get them all wrong; sorting all milk-first cups and all the tea-first cups into the wrong piles.
There’s more than one way get one pair of cups wrong. Let’s say Bristol sorted the first cup, a milk-first cup, into the wrong pile. Because she knows that each pile needs four cups, she’ll sort one of the tea-first cups into the wrong pile as well. It could be the fifth cup, the sixth cup, the seventh cup, or the eighth cup. So if she gets even one milk-first cup wrong, there are four variations on that wrongness. And since there are four milk-first cups, each with four variations of wrong, there is a total of 16 ways to only get one pair of cups wrong. Using the same reasoning, there are 16 ways to only get one pair of cups right.
So now we’ve covered her getting them all right, all wrong, one pair wrong, and three pairs wrong. Here’s the big one. How many ways are there to get two pairs of cups wrong? Let’s look at the possible variations in the first four cups, all of which are milk-first.
Wrong, wrong, right, right
Wrong, right, wrong, right
Wrong, right, right, wrong
Right, wrong, wrong, right
Right, wrong, right, wrong
Right, right, wrong, wrong
Phew! That is six ways to get the first four cups wrong. And remember, each of those six ways of wrongness can be paired with the same six ways to get the last four cups wrong, leading to a whopping 36 ways to get two pairs of cups mixed up.
One way to get them all right, one way to get them all wrong, 16 ways to get only one pair mixed up, 16 ways to get only one pair right, and 36 ways to get two pairs mixed up. That’s a total of 70 possibilities.
So, now that we’ve seen the way this breaks down, we know that there are 70 ways that the experiment can go, but only one way that would make Fisher believe that Bristol could taste the order in which her milk and tea had been poured. By looking at those variations we know why Bristol would lose her tea-tasting credibility if she mixed up even a single pair of cups. There’s only a one in seventy chance that she could have coincidentally gotten them all right. However, there’s a 16-out-of-70 chance that she coincidentally could have gotten one pair wrong, and three pairs right. Although it seems like a single mistake, miss-sorting just one pair of cups makes it much more likely that she just got lucky.
Fisher never revealed what the outcome of the test was. (We don’t even know that he used this exact test.) However, other colleagues who were at one of the most famous tea parties in the history of the world claim that Bristol sorted all of the tea cups correctly. If any io9 reader can tell us how she did it, it would be much appreciated.