Read on Twitter
Hi friends: @MaartenvSmeden, @edwardsjk, and I have some concerns about the OpenSAFELY preprint (available at https://www.medrxiv.org/content/10.1101/2020.05.06.20092999v1.full.pdf) We’ve summarized our thoughts in the form of an open access “letter to the editor” available here: https://doi.org/10.5281/zenodo.3843619. (THREAD)
Briefly, our concerns are around “risk factors.” As numerous folks (including notably @AndersHuitfeldt) have pointed out, “risk factor” can mean almost anything. Is a risk factor a cause? A predictor? Something else entirely?
This preprint (we argue) is not clear on what exactly “risk factor” means.
There’s language in this preprint suggesting that the authors think of their risk factors as (at least somewhat) causal: e.g., language like “effect,” and “attributable,” and “confounding.”
However, causal effect estimation requires a well-defined parameter of interest, a causal model (a DAG, say), adjustment for the confounders identified in that causal model…
…while avoiding control for mediators / colliders, and consideration of other causal identification conditions (causal consistency, say).
The mutually-adjusted hazard ratios reported in the preprint’s Tables 2 and Figure 3 do not appear to meet this bar, and so in general we do not think they can be interpreted causally.
Unfortunately, people have already started interpreting these results as causal. Such interpretations seem to be clear examples of the Table 2 Fallacy. ( https://pubmed.ncbi.nlm.nih.gov/23371353/ , by myself and @Lester_Domes ).
To be clear, the OpenSAFELY authors are not making any explicit causal claims – again, they are talking about risk factors. There are also methods in this preprint suggesting they are interested in predictive modeling, like the inclusion of a c-statistic.
As @MaartenvSmeden and others have pointed out, if what is desired here is a predictive model for COVID-19-related hospital death, there is more to do ( https://www.bmj.com/content/369/bmj.m1328)
In summary, we think the authors should clarify their intended parameter(s) of interest in this paper, and either: reduce their reliance on causal-adjacent language and presentations (e.g., the presentation of their data in ways that seem likely to encourage Table 2 Fallacies)…
…or take a more formal approach to causal effect estimation if that is what is desired.
Versions of these critiques/concerns very likely apply to other “risk factor” preprints and papers – both about COVID and in general. If you agree with the points made here, please keep an eye out for other examples.
Please see also @EpiEllie’s thread on this paper, touching on many similar topics/themes. https://twitter.com/EpiEllie/status/1258607277357006849
