Ok, this has been bugging me since yesterday, so I thought I'd actually go through this new webpage that claims Hydroxychloroquine (HCQ) is effective for COVID-19 and review it

Here goes nothing 1/n https://twitter.com/CT_Bergstrom/status/1291837801512595457

2/n So, first things first - what do the authors say they did?

They describe the site as a "country-randomized controlled trial" looking at HCQ vs a control

3/n The findings, such as they are, say that the 'treatment group' - countries that supposedly used HCQ more - had fewer deaths/million than the 'control group'

4/n Now, this site claims that these countries had 79.1% fewer deaths, which, if true, would make HCQ the most effective medical intervention ever, literally the balm of the gods

This seems somewhat unlikely, given that big RCTs have found no benefit whatsoever for the drug

5/n What did the authors actually do?

Well, they use the words "randomized controlled trial", but the site actually reports NONE OF THESE THINGS

The procedure was not randomized, there was no control, and this was not a trial

6/n What the authors ACTUALLY did was retrospectively divide countries into 'HCQ' or 'no HCQ' and compare death rates of COVID-19/million between those groups

This is what's known as an observational ecological trial

7/n Ecological trials are, as you may remember from my blog, prone to all sorts of biases

They are extremely hard to do well, and often can tell you very little about whether a treatment is effective or not https://medium.com/@gidmk/why-you-might-be-wrong-about-covid-19-the-ecological-fallacy-e8a47a030902

8/n It is hard to know why the authors chose to use the terminology they did. It is entirely incorrect, but I'm not here to litigate intentions, only facts, and the fact is that this website is wrong

Many, many times

9/n Moving on, the authors classified countries into either HCQ or no HCQ

How did they choose these groups?

Well, long story short, it appears almost entirely arbitrary

10/n Pretty much every single country was assessed in the same way - the authors cite a single (or maybe 2) news articles, and/or a tweet, to decide whether a country was in the intervention or control

This is manifestly inadequate to actually define HCQ usage

11/n There is no attempt to actually assess HCQ usage, despite this being SOMETHING YOU COULD DO

You could look at HCQ doses given/purchases made in countries by date, for example

Instead, the authors reference tweets

12/n So right off the bat, any analysis is meaningless. We have no idea whatsoever whether people in the 'intervention' group ACTUALLY RECEIVED HCQ, making the calculations largely a waste of time

But that's just the start of the errors here

13/n Next, we can look at the analysis itself

Or, rather, we can't, because the authors don't describe what they did, only giving relative risks and p-values

14/n Without any description of the analysis whatsoever, we cannot trust a single number that is portrayed on this site, because the stats could be simply incorrect

Impossible to know!

15/n The authors then report using "controls" to adjust the estimates, but these controls are again incredibly misleading

For example, country-level obesity rates

16/n The obesity rates are referenced to the CIA world factbook

Problem is, this data was last entered in 2016, and often is cited to sources up to a decade old

So it's really very out of date!

17/n But forgetting that, this is a classic example of the ecological fallacy

You CANNOT assume that country-level obesity rates apply to the people who got COVID-19 - if you don't check that this is true, whatever you produce is basically nonsensical https://medium.com/@gidmk/why-you-might-be-wrong-about-covid-19-the-ecological-fallacy-e8a47a030902

18/n On top of this, the outcome measure is terrible. There is no attempt to clarify whether the reported deaths from Our World In Data are correct for the country, simply the assumption that Algeria and France have comparable death reporting systems

19/n The ecological fallacy is at play in terms of death rates as well - deaths/million is a meaningless measure if you don't take infections into account!

20/n In short, the website fails in every possible way:

- not a trial
- entirely observational
- countless errors
- numerically meaningless
- arbitrary decisions

21/n Again, I am not here to litigate people's intentions - I'll leave that to others - but I will say that this is remarkably similar to the trash science produced by tech bros in the first few weeks of March

No understanding of the underlying issues at all

22/n Whatever your opinion on HCQ - whether you are pro or for - you cannot conclude anything of value from this website

It is misleading and wrong

23/n The discussion section is also, it must be said, a sight to behold

They defend their decision to use the term "country-randomized controlled trial" despite CONTRADICTING THEMSELVES (countries deciding is by definition NOT RANDOM)

24/n We already have a term for this kind of research - observational ecological study - so as I said, inventing a new term that is wrong is simply misleading. Just use the correct terminology if you're going to do something like this!

25/n If nothing else, the most basic additions that would make the website slightly less worthless would be

- details of statistical analysis
- measure of HCQ CONSUMPTION by country
- deaths BY INFECTION as the outcome
- control for other govt measures to prevent COVID-19

26/n Another point that's worth making - the authors say that HCQ is preventive of COVID-19

The methodology is not even vaguely close to what you'd need to know if that's true

27/n You'd want to see case numbers by day, along with every intervention (i.e. social distancing, school closures etc) and the day they were implemented for every country on the list

Then, number of HCQ doses given by day

AT AN ABSOLUTE MINIMUM

28/n Even if the authors make the suggested changes up until 25/n, they'd just end up with a meaningless correlation without measures like these, which are an enormous amount of work

The thing about ecological studies is that the good ones take A LOT of time

29/n Oh, another thing

The authors keep maintaining on Twitter that this study was "random" because patients didn't choose what treatment they got, countries did

This is absolute nonsense

30/n Firstly, it is misdefining random

If ANYONE chooses the treatment, then it's not random BY DEFINITION

Random means no one chooses, it's that simple

31/n But also - PATIENTS DID CHOOSE

This is where we get back to the ecological fallacy - it is absurd to suggest that individuals within countries didn't choose to take HCQ. Even countries that authorised it had adopters and non-adopters

32/n This protocol was BY NO POSSIBLE DEFINITION "random"

Using the term is incorrect, and at best ignorant not just of the scientific terminology but also the colloquial meaning of the word