BOX INFORMATIONSUPPLEMENTARY FILES FOR
Uri Simonsohn ; Joe Simmons, ' Heterogeneity Is Replicable: Evidence From Maluma, MTurk, and Many Labs', Data Coladahttp://datacolada.org/76BOX PUBLIC SINCE
August 18, 2020 (files may not be changed, deleted, or added
Uri Simonsohn (firstname.lastname@example.org)
Joseph Simmons (email@example.com) ABSTRACT
A number of authors have recently proposed that (i) psychological research is highly unpredictable, with identical studies obtaining surprisingly different results, (ii) the presence of heterogeneity decreases the replicability of psychological findings. In this post we provide evidence that contradicts both propositions.
Consider these quotes:
"heterogeneity persists, and to a reasonable degree, even in […] Many Labs projects […] where rigid, vetted protocols with identical study materials are followed […] heterogeneity […] cannot be avoided in psychological research—even if every effort is taken to eliminate it."
McShane, Tackett, Böckenholt, and Gelman (American Statistician 2019 .htm)
“Heterogeneity […] makes it unlikely that the typical psychological study can be closely replicated”
Stanley, Carter, and Doucouliagos (Psychological Bulletin 2018 .htm)
"Repeated investigations of the same phenomenon [get] effect sizes that vary more than one would expect […] even in exact replication studies. […] In the presence of heterogeneity, […] even large N studies may find a result in the opposite direction from the original study. This makes us question the wisdom of placing a great deal of faith in a single replication study"
Judd and Kenny (Psychological Methods 2019 .htm)
This post is not an evaluation of the totality of these three papers, but rather a specific evaluation of the claims in the quoted text. In the case of the paper by Judd and Kenny, the claims quoted above are secondary, used to motivate the remainder of their paper (e.g., in the opening lines of their abstract).