# 17 | ResearchBox

ResearchBox # 17 - 'DataColada 76 - Heterogeneity is Replicable'


Bingo Table
  Show file names
  Show file IDs
  Show timestamps
Studies 1 & 2


  Colada76.I.Maluma.qualtrics.Survey.pdf



  Colada76.I.Maluma.Qualtrics.Survey.qsf


  


  Data 1 - Colada 76 - 2019 04 26 (masked mturk id).csv


  


  Data 2 - Colada 76 - 2019 04 26 (masked mturk id).csv


  R - Maluma Takiti - Colada 76 - Posted - 2019 04 29.R


Supplement


  Simulation.r



  p-value heterogeneity ebersole.R



  many labs i2 .r


Previewing files
Files can be previewed by clicking on descriptions.
Codebooks can be previewed by clicking on


  

Tell us if something is wrong with this Box



BOX INFORMATION

SUPPLEMENTARY FILES FOR
Uri Simonsohn ; Joe Simmons, '[76] Heterogeneity Is Replicable: Evidence From Maluma, MTurk, and Many Labs', Data Colada
http://datacolada.org/76

LICENSE FOR USE
All content posted to ResearchBox is under a CC By 4.0 License (all use is allowed as long as authorship of the content is attributed). When using content from ResearchBox please cite the original work, and provide a link to the URL for this box (https://researchbox.org/17).

BOX PUBLIC SINCE
August 18, 2020   (files may not be changed, deleted, or added)

BOX CREATORS
Uri Simonsohn (urisohn@gmail.com)
Joseph Simmons (jsimmo@upenn.edu)

ABSTRACT
A number of authors have recently proposed that (i) psychological research is highly unpredictable, with identical studies obtaining surprisingly different results, (ii) the presence of heterogeneity decreases the replicability of psychological findings. In this post we provide evidence that contradicts both propositions. Consider these quotes: "heterogeneity persists, and to a reasonable degree, even in […] Many Labs projects […] where rigid, vetted protocols with identical study materials are followed […] heterogeneity […] cannot be avoided in psychological research—even if every effort is taken to eliminate it." McShane, Tackett, Böckenholt, and Gelman (American Statistician 2019 .htm) “Heterogeneity […] makes it unlikely that the typical psychological study can be closely replicated” Stanley, Carter, and Doucouliagos (Psychological Bulletin 2018 .htm) "Repeated investigations of the same phenomenon [get] effect sizes that vary more than one would expect […] even in exact replication studies. […] In the presence of heterogeneity, […] even large N studies may find a result in the opposite direction from the original study. This makes us question the wisdom of placing a great deal of faith in a single replication study" Judd and Kenny (Psychological Methods 2019 .htm) This post is not an evaluation of the totality of these three papers, but rather a specific evaluation of the claims in the quoted text. In the case of the paper by Judd and Kenny, the claims quoted above are secondary, used to motivate the remainder of their paper (e.g., in the opening lines of their abstract).