Late last year, the psychology world was rocked by a study that seemed to prove that many popular psychological claims were overstated, at best. Researchers set out to reproduce 100 studies, many of which form a foundation for much of what we know about psychology. In over 60 cases, they couldn’t reproduce the original results.
The Reproducibility Project not only brought into question a lot of what we think we know, it also brought ethical concerns to light. Reproducibility is an important part of research. Results are only good if they can be consistently repeated–otherwise, we may be looking at errors in methodology and science that doesn’t do anyone much good. If researchers were regularly overstating their results, that would be a problem. If studies weren’t reproducible and no one had any idea, clearly we were far too quick to adopt new science.
All fair points to make, but it turns out that the Reproducibility Project has problems of its own.
Slate has a great article breaking down not only the criticisms of the project but the counterpoints to those criticisms. This isn’t a callout, after all–it’s the scientific method at work.
Upon investigating the study further, the researchers identified a second and more crucial problem. Basically, the OSC researchers did a terrible job replicating those 100 studies in the first place. As King put it: “You’d think that a test about replications would actually reproduce the original studies.” But no! Some of the methods used for the reproduced studies were utterly confounding—for instance, OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam. Others simply didn’t use enough subjects to be reliable.
The new analysis “completely repudiates” the idea that the OSC study provides evidence for a crisis in psychology, says King. Of course, that doesn’t mean we shouldn’t be concerned with reproducibility in science. “We should be obsessed with these questions,” says King. “They are incredibly important. But it isn’t true that all social psychologists are making stuff up.”
So the good news is that the underpinnings of modern psychology haven’t been dashed to pieces. The bad news is that we still don’t really know how reproducible some of these results are. But we might yet–if the Reproducibility Project does nothing else, it gives us good reason to look twice at the science we take for granted.[Slate]