Read on Twitter
Ok, this has been bugging me since yesterday, so I thought I'd actually go through this new webpage that claims Hydroxychloroquine (HCQ) is effective for COVID-19 and review it
Here goes nothing 1/n https://twitter.com/CT_Bergstrom/status/1291837801512595457
Here goes nothing 1/n https://twitter.com/CT_Bergstrom/status/1291837801512595457
2/n So, first things first - what do the authors say they did?
They describe the site as a "country-randomized controlled trial" looking at HCQ vs a control
They describe the site as a "country-randomized controlled trial" looking at HCQ vs a control
3/n The findings, such as they are, say that the 'treatment group' - countries that supposedly used HCQ more - had fewer deaths/million than the 'control group'
4/n Now, this site claims that these countries had 79.1% fewer deaths, which, if true, would make HCQ the most effective medical intervention ever, literally the balm of the gods
This seems somewhat unlikely, given that big RCTs have found no benefit whatsoever for the drug
This seems somewhat unlikely, given that big RCTs have found no benefit whatsoever for the drug
5/n What did the authors actually do?
Well, they use the words "randomized controlled trial", but the site actually reports NONE OF THESE THINGS
The procedure was not randomized, there was no control, and this was not a trial
Well, they use the words "randomized controlled trial", but the site actually reports NONE OF THESE THINGS
The procedure was not randomized, there was no control, and this was not a trial
6/n What the authors ACTUALLY did was retrospectively divide countries into 'HCQ' or 'no HCQ' and compare death rates of COVID-19/million between those groups
This is what's known as an observational ecological trial
This is what's known as an observational ecological trial
7/n Ecological trials are, as you may remember from my blog, prone to all sorts of biases
They are extremely hard to do well, and often can tell you very little about whether a treatment is effective or not https://medium.com/@gidmk/why-you-might-be-wrong-about-covid-19-the-ecological-fallacy-e8a47a030902
They are extremely hard to do well, and often can tell you very little about whether a treatment is effective or not https://medium.com/@gidmk/why-you-might-be-wrong-about-covid-19-the-ecological-fallacy-e8a47a030902
8/n It is hard to know why the authors chose to use the terminology they did. It is entirely incorrect, but I'm not here to litigate intentions, only facts, and the fact is that this website is wrong
Many, many times
Many, many times
9/n Moving on, the authors classified countries into either HCQ or no HCQ
How did they choose these groups?
Well, long story short, it appears almost entirely arbitrary
How did they choose these groups?
Well, long story short, it appears almost entirely arbitrary
10/n Pretty much every single country was assessed in the same way - the authors cite a single (or maybe 2) news articles, and/or a tweet, to decide whether a country was in the intervention or control
This is manifestly inadequate to actually define HCQ usage
This is manifestly inadequate to actually define HCQ usage
11/n There is no attempt to actually assess HCQ usage, despite this being SOMETHING YOU COULD DO
You could look at HCQ doses given/purchases made in countries by date, for example
Instead, the authors reference tweets
You could look at HCQ doses given/purchases made in countries by date, for example
Instead, the authors reference tweets
12/n So right off the bat, any analysis is meaningless. We have no idea whatsoever whether people in the 'intervention' group ACTUALLY RECEIVED HCQ, making the calculations largely a waste of time
But that's just the start of the errors here
But that's just the start of the errors here
13/n Next, we can look at the analysis itself
Or, rather, we can't, because the authors don't describe what they did, only giving relative risks and p-values
Or, rather, we can't, because the authors don't describe what they did, only giving relative risks and p-values
14/n Without any description of the analysis whatsoever, we cannot trust a single number that is portrayed on this site, because the stats could be simply incorrect
Impossible to know!
Impossible to know!
15/n The authors then report using "controls" to adjust the estimates, but these controls are again incredibly misleading
For example, country-level obesity rates
For example, country-level obesity rates
16/n The obesity rates are referenced to the CIA world factbook
Problem is, this data was last entered in 2016, and often is cited to sources up to a decade old
So it's really very out of date!
Problem is, this data was last entered in 2016, and often is cited to sources up to a decade old
So it's really very out of date!
17/n But forgetting that, this is a classic example of the ecological fallacy
You CANNOT assume that country-level obesity rates apply to the people who got COVID-19 - if you don't check that this is true, whatever you produce is basically nonsensical https://medium.com/@gidmk/why-you-might-be-wrong-about-covid-19-the-ecological-fallacy-e8a47a030902
You CANNOT assume that country-level obesity rates apply to the people who got COVID-19 - if you don't check that this is true, whatever you produce is basically nonsensical https://medium.com/@gidmk/why-you-might-be-wrong-about-covid-19-the-ecological-fallacy-e8a47a030902
18/n On top of this, the outcome measure is terrible. There is no attempt to clarify whether the reported deaths from Our World In Data are correct for the country, simply the assumption that Algeria and France have comparable death reporting systems
19/n The ecological fallacy is at play in terms of death rates as well - deaths/million is a meaningless measure if you don't take infections into account!
20/n In short, the website fails in every possible way:
- not a trial
- entirely observational
- countless errors
- numerically meaningless
- arbitrary decisions
- not a trial
- entirely observational
- countless errors
- numerically meaningless
- arbitrary decisions
21/n Again, I am not here to litigate people's intentions - I'll leave that to others - but I will say that this is remarkably similar to the trash science produced by tech bros in the first few weeks of March
No understanding of the underlying issues at all
No understanding of the underlying issues at all
22/n Whatever your opinion on HCQ - whether you are pro or for - you cannot conclude anything of value from this website
It is misleading and wrong
It is misleading and wrong
23/n The discussion section is also, it must be said, a sight to behold
They defend their decision to use the term "country-randomized controlled trial" despite CONTRADICTING THEMSELVES (countries deciding is by definition NOT RANDOM)
They defend their decision to use the term "country-randomized controlled trial" despite CONTRADICTING THEMSELVES (countries deciding is by definition NOT RANDOM)
24/n We already have a term for this kind of research - observational ecological study - so as I said, inventing a new term that is wrong is simply misleading. Just use the correct terminology if you're going to do something like this!
25/n If nothing else, the most basic additions that would make the website slightly less worthless would be
- details of statistical analysis
- measure of HCQ CONSUMPTION by country
- deaths BY INFECTION as the outcome
- control for other govt measures to prevent COVID-19
- details of statistical analysis
- measure of HCQ CONSUMPTION by country
- deaths BY INFECTION as the outcome
- control for other govt measures to prevent COVID-19
26/n Another point that's worth making - the authors say that HCQ is preventive of COVID-19
The methodology is not even vaguely close to what you'd need to know if that's true
The methodology is not even vaguely close to what you'd need to know if that's true
27/n You'd want to see case numbers by day, along with every intervention (i.e. social distancing, school closures etc) and the day they were implemented for every country on the list
Then, number of HCQ doses given by day
AT AN ABSOLUTE MINIMUM
Then, number of HCQ doses given by day
AT AN ABSOLUTE MINIMUM
28/n Even if the authors make the suggested changes up until 25/n, they'd just end up with a meaningless correlation without measures like these, which are an enormous amount of work
The thing about ecological studies is that the good ones take A LOT of time
The thing about ecological studies is that the good ones take A LOT of time
29/n Oh, another thing
The authors keep maintaining on Twitter that this study was "random" because patients didn't choose what treatment they got, countries did
This is absolute nonsense
The authors keep maintaining on Twitter that this study was "random" because patients didn't choose what treatment they got, countries did
This is absolute nonsense
30/n Firstly, it is misdefining random
If ANYONE chooses the treatment, then it's not random BY DEFINITION
Random means no one chooses, it's that simple
If ANYONE chooses the treatment, then it's not random BY DEFINITION
Random means no one chooses, it's that simple
31/n But also - PATIENTS DID CHOOSE
This is where we get back to the ecological fallacy - it is absurd to suggest that individuals within countries didn't choose to take HCQ. Even countries that authorised it had adopters and non-adopters
This is where we get back to the ecological fallacy - it is absurd to suggest that individuals within countries didn't choose to take HCQ. Even countries that authorised it had adopters and non-adopters
32/n This protocol was BY NO POSSIBLE DEFINITION "random"
Using the term is incorrect, and at best ignorant not just of the scientific terminology but also the colloquial meaning of the word
Using the term is incorrect, and at best ignorant not just of the scientific terminology but also the colloquial meaning of the word
