Read on Twitter
John Ioannidis, of Most Published Research Findings Are False fame has again updated his preprint on infection-fatality rate for COVID-19
Kudos to him for updating, let's again look at what's happened 1/n
Kudos to him for updating, let's again look at what's happened 1/n
2/n Study is here: https://www.medrxiv.org/content/10.1101/2020.05.13.20101253v3
3/n You can find my original thread on the paper here: https://twitter.com/GidMK/status/1262956011872280577?s=20
4/n So, what's changed?
At first glance, not much. The median IFR has gone up from 0.26% to 0.27%, based on the inclusion of a bunch more studies
At first glance, not much. The median IFR has gone up from 0.26% to 0.27%, based on the inclusion of a bunch more studies
5/n Some of the more inappropriate statements have at last been removed, such as the age-related stuff and the incorrect citation of the case-fatality rate of influenza
Good!
Good!
6/n On the downside, we now have this comparison with the death rates from flu, which is also just...weird
If nothing else, the COVID-19 pandemic is STILL GROWING and has ALREADY killed as many people as a 'bad' influenza season
If nothing else, the COVID-19 pandemic is STILL GROWING and has ALREADY killed as many people as a 'bad' influenza season
7/n Also, many of the issues I've raised previously are still there. Many studies have been included that probably aren't reliable estimates of population seroprevalence are lumped in with very good estimates
That's...less than ideal
That's...less than ideal
8/n Let me illustrate with an example
This study has been included to calculate IFRs for 4 regions of China - Hubei (not Wuhan), Chonqing, Sichuan, and Guangdong
This study has been included to calculate IFRs for 4 regions of China - Hubei (not Wuhan), Chonqing, Sichuan, and Guangdong
9/n So the author has taken the percent positive of antibody tests for COVID-19 from each of the regions represented in the study, and used that as a population estimate for the entire region to calculate IFR
But is this reasonable?
But is this reasonable?
10/n The study only tested two groups: healthcare workers and people on dialysis
Now, Ioannidis excludes any testing on healthcare workers, but dialysis patients are...fine?
Now, Ioannidis excludes any testing on healthcare workers, but dialysis patients are...fine?
11/n And these high numbers of seropositive estimates led to inferred IFRs for these four places in China of 0.00%!
12/n If nothing else, the numbers here imply that 99.9996% of all infections in Chongqing were asymptomatic (500 official cases, widespread testing, but seropositivity of 3.8% in the study implying 12 million 'true' cases)
Is this plausible???
Is this plausible???
13/n There are also some numbers in this revised paper that are wrong
This figure should read 44%, not 47%
This figure should read 44%, not 47%
14/n Moreover, in the example highlighted above, the IFR calculated is for Brooklyn, but this was only true for a tiny subset of 240 patients in this 28,523 patient study. The IFR calculation should've been for the whole of NYC, not just Brooklyn!
15/n There are also some worrying inconsistencies in how Ioannidis has split up studies that sampled multiple places within countries
16/n For example, the ENE-COVID and Brazilian studies, which sampled entire countries by region, are only summed up as a single value
17/n On the other hand, several studies that sampled multiple regions (but found MUCH lower IFRs) in other places are split up by area
I cannot see any explanation for this in the paper
I cannot see any explanation for this in the paper
18/n On top of this, we've got another problem - collinearity
The basic issue is that you shouldn't lump multiple samples of the same group of people together into one study
The basic issue is that you shouldn't lump multiple samples of the same group of people together into one study
19/n But now, in the study we have Wuhan (A), Wuhan (B), and Hubei (not Wuhan)
It's very poor statistical practice to lump all these estimates together like this
It's very poor statistical practice to lump all these estimates together like this
20/n Similarly, we have two estimates from Spain. One is the ENE-COVID study, a rigorous randomized seroprevalence study that is the envy of the world
The other is a non-random sample of pregnant women at one place in Barcelona
These are given EQUAL WEIGHTS in the analysis
The other is a non-random sample of pregnant women at one place in Barcelona
These are given EQUAL WEIGHTS in the analysis
21/n The Spain example is even more of a problem because the ENE-COVID (the rigorous study) implies an IFR in Barcelona of ~1%
The survey of pregnant women implies ~.5%
Guess which one is used?
The survey of pregnant women implies ~.5%
Guess which one is used?
22/n Now, all of this collinearity is particularly troubling for that 0.27% estimate that I mentioned way back at the start of the thread
23/n If we get average the collinear results - where we've included the same study or the same sample multiple times - the median jumps immediately to 0.35%
That's quite a bit higher!
That's quite a bit higher!
24/n But there are more corrections to be made. In several places, the IFR that is in this paper does not match the IFR calculated by the study authors
25/n For example, Geneva. The original authors calculated an IFR of 0.64%, but this is downgraded to 0.45% in the paper
26/n And this is not the only example. Another study tested over three weeks and found seroprevalence of 3.85%, then 8.36%, then 1.46%. Overall 3.53%
The 8.36% figure is used, giving 5x more infections than the study itself found, and the lowest IFR possible
The 8.36% figure is used, giving 5x more infections than the study itself found, and the lowest IFR possible
27/n Taking all this into account, let's look at the IFRs for only those studies using representative population samples that were correctly calculated
28/n Here's the revised table. The lowest IFR is, again, Ioannidis' own study, at 0.18%. Nearly half of the estimates are above 1%, and they range all the way up to 1.63% (!)
29/n Somehow, for the third time running, there are innumerable decisions made in the paper that seem to only ever push down the IFR, rather than produce the best estimate
30/n As I've outlined, there are also a number of simple errors that make this very problematic as an estimate of the IFR (or the IFR range) for COVID-19
31/n All that being said, the discussion is now MUCH better, and really engages with some of the things I (and others) discussed in previous threads. Too much to go over here, but well worth a read
32/n Ioannidis has also now included some of the government-conducted studies in the paper, which is good to see
33/n All in all, some definite improvements, but a lot of things still in the paper that are really hard to reconcile with best practice
34/n The one thing I would point out - this from earlier in the thread is a classic example of moving the goalposts. The influenza comparison was clearly wrong, so now we have another comparison which is bad but slightly less wrong https://twitter.com/GidMK/status/1283232032085032961?s=20
35/n imo much better practice would be to acknowledge that COVID-19 is probably substantially more lethal than influenza, but that quantifying this difference is somewhat challenging
36/n Also, another statement that is incorrect and has remained in each version - that disadvantaged populations/settings are uncommon exceptions in the global landscape
This remains simply untrue
This remains simply untrue
37/n Also, you can find my personal best estimate in the paper that @LeaMerone and I authored on IFR here. A reasonable guess for most areas seems to be 0.5-0.8% https://www.medrxiv.org/content/10.1101/2020.05.03.20089854v4
38/n Another addition, this thread goes through some of the headaches with the paper that have remained through every version
TL:DR - it's not systematic! https://twitter.com/AVG_Joseph96/status/1283236273558294528?s=20
TL:DR - it's not systematic! https://twitter.com/AVG_Joseph96/status/1283236273558294528?s=20
