Read on Twitter
Speaking only for myself and not my coauthors, this replication project taught me more about the behaviour of researchers and their beliefs than it did about the effect being studied. And what I learned was pretty depressing. https://twitter.com/ianhussey/status/1308428889190993921
First, a brief overview of results:
- Effect replicates only under original awareness exclusion criterion (top left)
- Does not under 3 others
- No sig differences between these 4 criteria
- Effect size < half original & meta of published literature
- Effect replicates only under original awareness exclusion criterion (top left)
- Does not under 3 others
- No sig differences between these 4 criteria
- Effect size < half original & meta of published literature
The journal wasn't open to publishing a response by the authors of the original study. Instead, editor asked us to include their short response verbatim in our paper.
The bits that makes my eyes water is "unqualified replication" and "effect size is of minimal relevance"
The bits that makes my eyes water is "unqualified replication" and "effect size is of minimal relevance"
The preregistered analytic strategy - approved by the original authors - was a meta analysis across data collection sites. Their response not only inspects the individual sites, but considers only the descriptive direction of the effect, not even its significance.
Let's look at the p values for each site under the original exclusion criterion - the only one where the meta effect is significant.
At *each site* N is twice that of the original published study. Would you consider these results indicative of a true effect?
At *each site* N is twice that of the original published study. Would you consider these results indicative of a true effect?
Next, the original authors argue that because there is no evidence of difference in outcomes between the 4 exclusion strategies, we should accept that the effect is real because it was significant under the original one. But the reverse argument would also hold to support null.
The original authors also refer to the effect size as minimally relevant, but this couldn't be further from the truth. The RRR's protocol was approved by the orig authors. Results were an ES half the original / meta of the published lit. This can't be explained without bias.
(The above puts aside the q of whether the exclusion methods are even valid measures, which many coauthors do not believe. We discussed this in both the main manuscript & a separate preprint here.) https://psyarxiv.com/4gzsp w/ @seanhughes102
My real interest in this project was to observe the behaviour of scientists.
I observed there are very few stop signals. Under no condition can we collectively conclude "no further research necessary". Ie we cannot discard incorrect ideas.
If so, how can we possibly progress?
I observed there are very few stop signals. Under no condition can we collectively conclude "no further research necessary". Ie we cannot discard incorrect ideas.
If so, how can we possibly progress?
Many people think of science as a form of behaviour orientated towards generating answers. The behaviour I observed here (& in many other projects I've been involved in) looks more like generating questions - almost for the sake of it, as ends in themselves.
I should add that by “researchers” here I mean our collective group behaviour, as social groups with dynamics under the influence of the system we all work in - not any of the individual collaborators, all of whom who I am grateful they contributed to this project.
