I definitely agree that we shouldn't place too much weight on a single study picked from an otherwise strong literature. As @farid_anvari pointed out (thanks Farid!), my quick tweet simply called for caution and skepticism 1/ https://twitter.com/hansijzerman/status/1249215437533044736

But as @siminevazire noted, one study can sometimes raise concerns that may have implications for that literature, and honestly, although I picked the study mostly randomly (it was recent, had some variables I was interested in, was publicly available), I'm concerned 2/

Disclaimer: I have not delved deeply into this lit. My comments are about the implications of my evaluations of THIS study. I'm open to the possibility that the rest of the lit addresses my concerns. 3/

Here's the study, published in 2017, showing a three-way interaction between dispositional affect, social support, and stress predicting the common cold: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5830120/ 4/

Some context: The study comes from the Common Cold Project, which is a set of five really cool studies that measure a bunch of things, intentionally infect people with viruses, and then follow up in quarantine https://www.cmu.edu/common-cold-project/index.html 5/

There's a lot of really great things about this. First, it looks like they made the data available for reanalysis. That's awesome (though I have not tried to access it) 6/

Second, this is a really difficult, important work, and the design of these studies is in many ways really strong. Sample sizes are decent compared to others in this type of work; many non-self-report measures. Some really important strengths 7/

Third, the main ideas make sense and fit with intuitions: Stress and other hardships might affect our health, and one way may be through the immune system. Having heard the basics of these studies before, I had assumed they were strong 8/

So why am I concerned? 1) The studies were designed and run before the recent changes in norms about research practices. There are many ways the samples can be divided and combined and there are dozens if not hundreds of measures that can be coded many different ways 9/

Second, because most were conducted between 1991-2015, I don't think these studies were preregistered, there were no multiverse analyses, etc., that could increase confidence that the reported results are robust 10/

Third, precisely because this work is so hard to do and so expensive, it's difficult for anyone else to replicate it and I'd imagine researchers who work on projects like this (including me if I was doing it) must feel pressure to find something to satisfy the funders 11/

So this is the type of literature that we should all be asking questions about, even before seeing the results of any particular study. And again, part of the reason we should be critical is because it is important 12/

But what about the specific study I looked at? Well, I think we should probably start with some skepticism of a three way interaction between three or four self-report predictors and an extremely complex outcome like catching a cold. Indeed, the p values were .02 and .04 13/

More importantly, this study combines data from 3 of 5 prior studies used in the authors' prior work. In testing the 3-way interaction, they also had to test the simple effects of the predictors that had received so much attention before: Stress, Social Support, PA, NA 14/

In other words, they did their own larger-sample quasi-replication (but with overlapping samples) of the work that was cited in the thread that @siminevazire tweeted about https://twitter.com/LFeldmanBarrett/status/1249019897843257346 15/

Note: They said in the paper, they first tested these predictors on their own, before adding the interactions, and this is what we want to know about to replicate the prior work 16/

And somebody please correct me if I'm misinterpreting this, but it looks like the authors failed to replicate their prior work in this larger sample: No effect of social support, no effect of stress, results for PA and NA not reported in enough detail to tell 17/

Now again, I want to be clear, there maybe something I'm missing here, so I'd be happy to have anyone point out that I'm misinterpreting what they found. But if I'm interpreting correctly, this seems problematic for the literature 18/

And of course, maybe there is a large literature beyond these five samples that supports these claims; but these are definitely some of the most famous studies showing support, and my guess is that the difficulty of this work would make similar studies very rare 19/

And as a final important point, I wanted to note that in the discussion, the authors emphasize the three-way interactions that they found but do not even mention that they failed to find the basic effects that are supposed to be the bedrock findings of this literature 20/

Anyway, I should probably end this thread here and go wake my kid up to begin the disappointment that is Easter in 2020 https://twitter.com/rlucas11/status/1249308575345803264 21/21