by: lediya dumessa
Overview:
Replication, which confirms the accuracy of empirical findings in research studies, is crucial to psychological science (Brandt et al., 2013). For replication study to be considered successful, it requires using similar conditions present in the initial experiment and yielding the same effects (MLP; Klein et al., 2014). Unfortunately psychological science has been facing a replication crisis in which replication rate is low. However, there are researchers who argue that replication rate may be underestimated and that it is much higher that what is commonly reported.
In the Open Science Collaboration (OSC, 2015), researchers conducted replications 100 experimental and correlational studies. Their findings were disheartening in that out of the ninety-seven percent of original studies with significant results, only 36% of the replicated studies resulted in p values less than .05. Moreover the effect size of the replicated studies was only about half of that of the original studies.
What are the potential reasons for the lack of replicability?
The low replicability rate maybe related to factors such as using small sample sizes, p-hacking (increasing sample size until the p-value becomes significant), the file-drawer phenomenon (not presenting results with null findings) and publication bias (negative results have a low probability of getting published) that may lead to high rate false positives. The publish or perish phenomenon present in the academia world for example put undue pressure on scientist to produce publications that have positive results at a very high rate.
Replication, which confirms the accuracy of empirical findings in research studies, is crucial to psychological science (Brandt et al., 2013). For replication study to be considered successful, it requires using similar conditions present in the initial experiment and yielding the same effects (MLP; Klein et al., 2014). Unfortunately psychological science has been facing a replication crisis in which replication rate is low. However, there are researchers who argue that replication rate may be underestimated and that it is much higher that what is commonly reported.
In the Open Science Collaboration (OSC, 2015), researchers conducted replications 100 experimental and correlational studies. Their findings were disheartening in that out of the ninety-seven percent of original studies with significant results, only 36% of the replicated studies resulted in p values less than .05. Moreover the effect size of the replicated studies was only about half of that of the original studies.
What are the potential reasons for the lack of replicability?
The low replicability rate maybe related to factors such as using small sample sizes, p-hacking (increasing sample size until the p-value becomes significant), the file-drawer phenomenon (not presenting results with null findings) and publication bias (negative results have a low probability of getting published) that may lead to high rate false positives. The publish or perish phenomenon present in the academia world for example put undue pressure on scientist to produce publications that have positive results at a very high rate.
"Incentives for achievement are similar across disciplines. Publication is essential, and positive, novel, tidy results increase the likelihood of getting published everywhere."- Nosek, 2015
The bias against null results further compromises the integrity of the field, as researchers are aware that their work would have a low probability of publication. This may partially explain selective reporting for example. Selective reporting refers to the tendency to report only the results that were significant. That is, if even if a study has multiple hypotheses only the ones that generated significant results will be presented.
"Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both."- Nosek & Lakens, 2014
Why is this a problem?
Replication increases precision of effect size, establishes generalizability of effect, and replication studies that do not yield same results as the original studies provide information on the necessary conditions for the expected effects (Nosek & Lakens, 2014). Given that one of the goals of research is to yield results that are externally valid or generalizable across individuals and contexts, it can lead to inaccurate implementation of results if they do not get replicated in other studies. Failure to identify the necessary conditions that a phenomenon occurs in can lead to overgeneralization of results (Henry, MacLeod, Phillips, & Crawford, 2004; MLP; Klein et al., 2014)
What could be leading to the low replicability statistics?
Gilbert and colleagues (2016) present potential reasons for the low replicability of original studies, specifically those conducted in the open science collaboration. They argue that one potential explanation could be an error in the methodologies that replications use rather than the effects of the original studies not being reproducible. OSC had multiple sources of error in their data including research method infidelities that were taken into account when reporting the results (Gilbert, King, Pettigrew, & Wilson, 2016). Moreover they argue that the OSC studies were underpowered. Finally, Gilbert and colleagues (2016) state that the OSC replication studies may have been biased toward failure; that is, they were expecting low replicability of the original studies and their methodologies and results were utilized to confirm that bias.
Conclusion:
So, Is the field of psychology facing replication crisis? Lynch and colleagues (2005) argue that non-replication does not imply falsehood. Given that exact replication of studies is impossible and even unnecessary, replication can fail due to differences in operational variables and the construct (concept) itself.
Replication increases precision of effect size, establishes generalizability of effect, and replication studies that do not yield same results as the original studies provide information on the necessary conditions for the expected effects (Nosek & Lakens, 2014). Given that one of the goals of research is to yield results that are externally valid or generalizable across individuals and contexts, it can lead to inaccurate implementation of results if they do not get replicated in other studies. Failure to identify the necessary conditions that a phenomenon occurs in can lead to overgeneralization of results (Henry, MacLeod, Phillips, & Crawford, 2004; MLP; Klein et al., 2014)
What could be leading to the low replicability statistics?
Gilbert and colleagues (2016) present potential reasons for the low replicability of original studies, specifically those conducted in the open science collaboration. They argue that one potential explanation could be an error in the methodologies that replications use rather than the effects of the original studies not being reproducible. OSC had multiple sources of error in their data including research method infidelities that were taken into account when reporting the results (Gilbert, King, Pettigrew, & Wilson, 2016). Moreover they argue that the OSC studies were underpowered. Finally, Gilbert and colleagues (2016) state that the OSC replication studies may have been biased toward failure; that is, they were expecting low replicability of the original studies and their methodologies and results were utilized to confirm that bias.
Conclusion:
So, Is the field of psychology facing replication crisis? Lynch and colleagues (2005) argue that non-replication does not imply falsehood. Given that exact replication of studies is impossible and even unnecessary, replication can fail due to differences in operational variables and the construct (concept) itself.
"Reproducibility is important, hard and improvable. We can nudge the incentives driving our behavior so that researchers are rewarded for more transparent and reproducible research." -Nosek, 2015
While whether or not the current controversy poses a threat to the science of social psychology continues to be debated, it would benefit the filed to at least decrease the bias against publishing null results so that researchers can maintain the integrity for their results. Besides, creating a culture that encourages researchers to replicate studies may benefit to strengthen the results of novel findings before they are accepted as truth.
References:
References:
- Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., ... & Van't Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224.
- Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). More on “Estimating the Reproducibility of Psychological Science”. Available at projects. iq. harvard. edu/files/psychology-replications/files/gkpw_post_publication_response. pdf
- Klein, R., Ratliff, K., Vianello, M., Adams Jr, R., Bahník, S., Bernstein, M., ... & Cemalcilar, Z. (2014). Data from investigating variation in replicability: A “Many Labs” Replication Project. Journal of Open Psychology Data, 2(1).
- Nosek, B. A., & Lakens, D. (2014). Registered reports.
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
RSS Feed
