Lies, damned lies, and statistics
22nd April 2017 | Professor Nick Lee
Nick Lee on… Statistical significance testing
List time around, I explained the concept of endogeneity bias, and how it might influence how much trust you can have in the results of studies you see reported in the media, and also research projects and white papers you might read yourself. This time, I’m going to deal with the other major problem in drawing conclusions from research – that of tatistical significance testing.
Now, you might remember learning about statistical significance testing either at school, or university. It’s one of the foundational topics in any statistics, market research, psychology, or organizational research course. And now I’m going to tell you not to trust it? The short answer is yes, I am. The longer answer is that (like most things) it’s complicated.
When we use statistical significance testing for its correct purpose, it’s fine. However, most of the time we don’t. This has led to many problems, and a lot of the results of high-profile studies are not to be trusted because of this. In fact, in the psychological research fields, it is not overstating it to say that some academics are talking of a “crisis”, in part because of this issue.