A recent large-scale study that came out of the University of Virginia has made a lot of headlines of late. A group known as the Reproductability Project replicated 100 social psychology experiments and found that more than half failed to replicate, meaning that the new studies came up with different results. This gained a lot of attention in the popular media and has caused many people to question their trust in psychology research openly. As cognitive-behavioral therapists pride themselves on clinical practice supported by research, this is a relevant issue for those using CBT.
The primary issue concerns what to make of two identical studies with un-identical results. The insinuation in popular media is that there may be some fabrication of results. This is an understandable conclusion, especially in light of notable incidences of this in the medical field in the recent past. However, this is not the only conclusion. Nor is it the most likely one.
The reality is that it is fairly common for replication studies to produce results different from the original study. This is expected because even when there are efforts to replicate a study such that the second study is identical to the first, some conditions are not controlled for and thus are different from those in the original. A slight change in the conditions of an experiment can yield widely divergent results. A different sample population, an updated psychometric measure, or even a change in temperature can result in a large degree of variability in the data.
In actuality, when replicated experiments result in inconsistent data, this is a cause for excitement in the scientific community. It means we do not understand something as well as we thought we did. In essence, the previous data was only true under certain conditions. If we can learn what was so influential about those conditions, we can have an even more sophisticated understanding of the phenomenon being observed and hone the experimental hypothesis. Divergent data are just as helpful, if not more so, than identical results.
This has certainly been the case with cognitive-behavioral clinical studies. Gold-standard treatments such as prolonged exposure for PTSD have been the source of numerous studies and replication studies. Through continual testing and re-examining, we have a better idea of how these treatments work and, when they don’t, what factors are responsible. Most importantly, when a clinical practice is deemed evidence-based, it is not because of having merely one study to support it. Evidenced-based denotes having numerous studies evidencing the effectiveness or efficacy of a particular clinical practice. CBT has been the subject of hundreds of studies in the last two decades, and as a result, we know which practices are effective and under which conditions.