A Trick With Horses
I am going to teach you a trick with horses. You can do it at home if you have enough horses. This old trick was famously done by TV’s Derren Brown.
We will run a with race with two horses, we can use more horses but with two it is easiest to explain. First find the names and addresses of 1024 people. Write to 512 to say horse A will win the race. To the other 512 say that horse B will win. Say horse B wins. Throw away the names of people to whom you said horse A would win. Next week, new race. To 256 of the remaining 512 say that horse C will win and to the other 256 say horse D. Horse C wins. Throw away the names of the people you gave the wrong prediction to. Same again the following week. Now there will be 128 people you have given a winning horse to three times. Do it again, 64 people and four wins. Then 32 people and five wins. And so on. Eventually there will be one person who thinks you are most amazing expert in horse racing, you have predicted 10 winners in 10 races. They are now primed for the big scam.
On TV, Derren gets this one person to bet all their life savings on the next race, on his prediction. And the sucker loses. Derren is distraught. This all makes great TV.
It has a happy ending though, because Derren has in secret bet on the horse that actually won, and he gives his winnings to the aforementioned sucker. So everyone is happy. (Of course, in secret Derren had bet on all the horses in this last race, so he can’t lose. They don’t show this on the TV though.)
In statistics this trick is a bit like what called is “p hacking.”
A scientist says he is 95% confident that eating peas causes spots. Where did he get that 95% from? He does lots of statistics on lots of data involving people who do and don’t eat peas. He writes research papers, becomes famous in the vegetable community and is hired by the Cabbage Marketing Board to promote the health advantages of cabbage. Cabbages good, peas bad.
Problem is that for years he does this research on many people, and many vegetables. He studies 1,000 people, and none of them show any bad effects from eating parsnips or mange tout or carrots. Not even broccoli. This is no good, he cannot write a research paper saying vegetables are good for you. Everyone does that. So he finds another 1,000 people. And then another 1,000. Then just 500. Or 200. With a small number of people he is more likely that one of them will be the school boy who allergic to everything. So one day, hurray, he find some very spotty people in his small sample. It had to happen eventually. Naughty scientist.
Mathemagical Thinking Lesson
You see this a lot with beauty products for which there are lower standards for claims than with, say, medical products. Look out for the small print telling you how large was the sample of people used in the trial. Eight out of 10 cats etc. Is that 80% of 10,000 cats? Or maybe just four out of a sample of five cats.
Are you looking into which school to send your genius progeny? You might find that a fairly local school has a very high recent ranking, as measured by exam results. But is that because that local school is very small and has one particularly bright child in one year? One out of a small sample, one is relatively large compared to one out of a large sample. As soon as that child leaves the school will fall back in the ranking.