by: Ashley coleman
History of the Replication Crisis
In 2015, a large group of researchers (Open Science Collaboration) published collaborative results obtained from conducting 100 replication studies. They claimed that replication is not rewarded in research in the same way that novel findings are. However, they argued that replication findings are necessary to further current scientific understanding and promote development. They hypothesized that factors such as status of the researcher, random error, reporting only significant results, purposefully using specific analysis techniques, and poor specification of methodology could be related to the reproducibility of studies.
Open Science Collaboration (2015) found that the average effect sizes found in the replication studies were approximately half of the average effect sizes reported in the original studies, indicating that overall, the effects studied could be weaker than the original studies reported. In addition, almost all (97%) of the original studies found significant results, whereas approximately one-third (36%) of replication studies found significant results. Furthermore, in terms of predicting reproducibility, the authors found that the strength of the original finding predicted whether the finding could be replicated. In other words, the stronger the original finding, the more likely the finding could be replicated.
One replication attempt (Sinclair, Hood, & Wright, 2014) analyzed the Romeo and Juliet effect originally found by Driscoll, Davis, and Lipetz (1972). Driscoll et al. (1972) found that couples’ relationships were strengthened with increased parental interference into the relationship. However, Sinclair et al. (2014) found support for the opposite idea, that couples’ relationships were weakened by parental interference. Their findings were consistent with another theory, the “social network effect,” which claims that disapproval by the social network (including parents) leads to poorer relationship quality. Sinclair et al. (2014) suggested that limitations in Driscoll et al.’s (1972) original study may have influenced their results. For example, operationalizing interference as disapproval and only collecting data about parents’ approval (rather than the larger social context, including friends) may have contributed to their findings.
This replication study example suggests that issues with replication may not indicate faulty research techniques or even intentional bias; rather, this example highlights that findings may not replicate if the conceptualization of the ideas differs between researchers or has changed over time.
In Response to the Replication Crisis
Although the large-scale replication meta-analysis (Open Science Collaboration, 2015) claimed that reproducibility of psychological findings was quite low, some researchers vehemently disagreed. For example, Gilbert, King, Pettigrew, and Wilson (2016) argued that there were methodological flaws with the replication meta-analysis that may have led them to draw incorrect conclusions about the reproducibility of findings. They argued that the Open Science Collaboration failed to account for adequate sampling error that occurred as a result of sampling from new populations and using different procedures than reported by the original studies, as well as methodology that underestimated the replication results.
Furthermore, they reported that replication studies in which the protocol was endorsed by the original authors were more likely to replicate than the replication studies in which the protocol was not endorsed by the original authors. In other words, they suggested that a higher degree of similarity between the original and replication protocols predicted the likelihood of successful replication, and they argued that had all protocols been approved by the original authors, the results would have been within the confidence intervals for the expected replication rates. Therefore, they concluded that no replication crisis exists in psychology.
Why Replication Matters
Klein et al. (2014) argued that a lack of replication may lead to overgeneralization of effects. In other words, although a finding may be true in the particular setting that it was originally found in, it may not hold true across settings. However, if studies are not replicated (and extended to other individuals and contexts), we may assume that a narrow finding holds true across a broad range of contexts. Therefore, replication and extension allows for a better understanding of scientific phenomena (Lynch, Bradlow, Huber, & Lehmann, 2015). However, the question remains as to how to increase replication, if replication is a necessary component of the scientific method. Everett and Earp (2015) suggested that replicating a previous finding could be a requirement for psychologists before earning a Ph.D. They argued that requiring replication as a condition of obtaining a doctorate degree would increase the number of replication studies in psychology.
However, while replication is important, whether or not a finding replicates in one replication study is not what determines whether the finding is valid or generalizable. Therefore, while replication studies are necessary, the findings from one replication study do not solve the issue with replication. Scientific research requires continued replication and expansion of findings to understand under which conditions a finding holds true or under which the relationship between the variables is not as well understood. In conclusion, the findings from previous replication studies do not indicate a replication crisis in psychology. They do, however, highlight the important of replication for psychological research.
References
- Everett, J. C., & Earp, B. D. (2015). A tragedy of the (academic) commons: Interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Frontiers In Psychology, 6.
- Gilbert, D.T., King, G., Pettigrew, S., Wilson, T.D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 351(6277). doi: 10.1126/science.aad7243
- Klein, R. A., Ratliff, K. A., Vianello, M., Adams Jr, R. B., Bahník, Š., Bernstein, M. J., ... & Cemalcilar, Z. (2014). Investigating variation in replicability. Social Psychology, 45(3). 45(3):142–152. doi: 10.1027/1864-9335/a000178
- Lynch, J. G., Bradlow, E. T., Huber, J. C., & Lehmann, D. R. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333-342.
- Open Science Collaboration. (2015). Science, 349(6251). doi: 10.1126/science.aac4716
- Sinclair, H. C., Hood, K. B., & Wright, B. L. (2014). Revisiting the Romeo and Juliet effect (Driscoll, Davis, & Lipetz, 1972). Social Psychology, 45(3), 170-178. doi: 10.1027/1864-9335/a000181