by: shengtian wu
The field of psychology was rocked by a report that studies could not be replicated. It was viewed as an alarm for the psychology’s crisis. As for this issue, there were debates among researchers. Some of them contend this is the crisis, while others insist it is not. After weighing evidences reported in literature, I think it is a crisis of psychology. This blog will discuss the opinions occurred in the literature and also discuss about what needs to be done to avoid the crisis.
The Issue of Replication Crisis
Reproducibility of research is a very essential feature of science (Psychology, 2015). Results of studies should be replicated when the methods, procedures, and other conditions were implemented as they were designed in the original studies. We are able to utilize the research methods to create same positive results only when research is reproducible. However, the reproducibility of psychological science was questioned by researchers, which also led to another question, “can we trust the psychological science?” Open Science Collaboration (OSC) reported surprising results. There were only 36 to 47% of the original studies were successfully replicated. Even though the same materials, procedures, and statistical methods, effect size of studies were significantly different from original studies.
However, another study attempted to replicate previously published studies found 11 out of 13 classic and contemporary effects among 36 studies were successfully replicated (Richard et al., 2014). Among successfully replicated studies, it was found that some original studies underestimated or overestimated the target effects. The primary goal of the study was to find out what factors among sample sizes, locations of studies, and culture (i.e., U.S. vs International) contribute to the variation of effects. Nonetheless, no factor was identified as the main contributor, while the measurement error was reported to be the main factor of the variation.
Sampling Can Screw Up the Results
Responding to the two studies that examined the reproducibility, Anderson and his colleagues (2016) disagreed with opinion that the studies were not replicable. They mentioned that OSC made several mistakes that led to low reproducibility. OSC’s participants of research did not match well with the original studies. For example, while the original study recruited Americans in order to measure their attitude toward African American, OSC’s study recruited Italians, who might share different stereotype toward the target population.
Replication Method Matters
Second, OSC’s replication method tends to underestimate the actual rate of replication. If OSC used “Many Labs” project’s (MLP) method (i.e., replicate each study 35-36 times and then pool the data), they would have produced a much more higher rate of replication. Sometimes, repeated replication should be attempted in order to observe the effects of the studies. It is possible that the ineffectiveness is produced by chance.
Infidelities in Research Procedures
Third, OSC also assumed infidelities in research (i.e., not follow protocols of studies accurately) do not impact the results of the studies. However, studies endorsed protocols had almost three times higher replication rate. That is, it is difficult to believe OSC’s results, “replication crisis”, while the infidelities were observed. Given three mistakes of OSC, Anderson and his colleagues concluded the OSC’s analysis did not prove the low reproducibility of studies in the field of psychology. Moreover, one study altered the procedures of study by giving older children relatively easy task of locating items on a small computer screen, while the original study provided gave younger children some easier tasks. A slight change in the sampling would result in different outcomes whether the study intentionally selected inappropriate samplings.
Journals’ Bias Toward Studies With Positive Outcomes
It is also well-known that journals target the studies that show positive outcomes (Bower, 2016). For that reason, researchers tend to only try to submit studies that have positive outcomes to journals whereas studies that did not find a significant effect would be disregarded by researchers. Thus, it appears that pressures to publish also leads to some more infidelities among researchers. In order to get publication, researchers might selectively present those results that show significant results.
What Can We Do?
Metascience might be a possible way to decrease current problems (Schooler, 2014). Metascience means using rigorous methods to assess how scientific practices impact if the scientific conclusion is valid. In other words, researchers should be trained in way that they could clearly understand what kinds of factors could impact the reliability of the new findings. Metascience needs to be strengthened for those individuals who work in journals, so they are able to identify problems in replication.
In my opinion, the atmosphere of the journals should be changed to value or even collect the studies that show non-significant results might be also helpful. Publication is directly related to survival as a faculty in a university and researchers in certain research institutions. Journals bias toward only positive outcomes surely impacts researchers’ tendency to hunt for positive outcomes. Meanwhile, more education on research ethic might be needed to decrease unethical behaviors among researchers.
Strengthening statistical techniques for researchers might be a solution for decreasing the probability of producing inaccurate outcomes and misunderstanding methodology of research. For example, in order to see a significant effect in a study, there should be enough statistical power, which in part is influenced by the number of participants. Moreover, other characteristics, such as cultural backgrounds, locations of study, and age of participants, should be well-controlled in order to replicate a study accurately. Missing any part of the statistical method might lead to a deviation from the original study. Researchers should be well-trained to understand all of statistical methods.
Summary
The topic of replication crisis appears in the field of psychology with a heated debate. The failures of replication results from sampling errors, unethical research practices, journals’ bias toward positive outcomes of studies, and inappropriate replication methods. In order to minimize the problems, we might need rigorous methods to scrutinize the studies, alternation of the journals’ bias, enhancement in ethic education and statistical skills.
- Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., & ...
- Zuni, K. (2016). Response to Comment on "Estimating the reproducibility of psychological science". Science (New York, N.Y.), 351(6277),1037. doi:10.1126/science.aad9163
- Bower, B. (2016). Psychology's replication crisis debated. Science News, 189(7), 8.
- Psychology. Estimating the reproducibility of psychological science. (2015). Science, 349(6251), 1-6. doi:10.1126/science.aac4716
- Richard A., K., Kate A, R., Michelangelo, V., Reginald B Adams, J., Stĕpán, B., Michael J, B.,
- & ... Brian A, N. (2014). Data from investigating variation in replicability: A “Many Labs” replication project. Social Psychology,45(3), 142-152. doi:10.5334/jopd.ad
- Schooler, J. W. (2014). Metascience could rescue the 'replication crisis'. Nature, 515(7525), 9.
- doi:10.1038/515009a