A systematic review/meta-analysis of honey vs other cough/cold remedies came out yesterday in BMJ Evidence-Based Medicine

I think this is worth a quick peer review on twitter 1/n

2/n The review is here: https://ebm.bmj.com/content/early/2020/07/28/bmjebm-2020-111336

3/n Basically, the authors searched the literature for all the studies looking at honey being used to treat URTIs (cough), and aggregated them together into one model.

They found that honey works better than usual care! Amazing right?

3.5/n (N.B. URTI = Upper Respiratory Tract Infection)

4/n And if you read the methods section of this review, it's fairly reasonable. Good exclusion criteria, risk of bias assessment, it ticks all the boxes!

But as ever, the devil is in the details

5/n At first glance, the results look rosy. Most studies were assessed as of reasonable quality, and the overall benefits seem accurate if very weak due to the low overall sample size

6/n The thing about systematic reviews/meta-analyses is that the included studies are EVERYTHING

As they say, garbage in = garbage out

Let's take 2 of these studies as an example

7/n These studies are both by the same author group, published close together, and both report finding that honey works better than the alternative to treat cough over 1 week

8/n But it doesn't take long to find issues. There are large tracts of text that are identical between the studies. There are numerical errors in the tables and GRIM inconsistencies ping @sTeamTraen @MicrobiomDigest (rounding done wrong and incorrect totals).

9/n Moreover, the risk of bias assessment in the systematic review is, as far as I can tell, incorrect. The authors of these studies do not report, for example, how the randomization procedure was generated

10/n They also don't describe blinding of clinical staff, and actually from reading I'm fairly sure the doctors could tell which medication the patients were taking

11/n Another issue for the SR/MA is that the data extraction from one of these studies is just wrong

Here, honey has a mean reduction of 2.4 in cough frequency compared to 1.2 for 'usual care'

12/n Now, firstly, that's the reduction not for "honey" but for "honey + coffee", and it's being compared to a control of "coffee alone" which is definitely not what this analysis is looking at

13/n But further to this, the number given in the meta-analysis of -2.4 is wrong, as it should be 3.0-0.4 = -2.6

So it's wrong in two ways, both of them very bad for this analysis

14/n I should also mention, both of these studies were rated as at low/moderate risk of bias in the systematic review. Have a look yourself and see if you think that this is reasonable:
https://www.sid.ir/en/journal/ViewPaper.aspx?ID=196449
https://www.nature.com/articles/pcrj201372#Sec9

15/n I haven't gone through every study yet, but every one that I have is similarly worrying. I can't actually find a study that I'm not worried about in this analysis

16/n If I go through the studies, I really can't find any that I would rate as a "low" risk of bias. Most of them are "high", some of them are really worryingly bad

17/n Given this, I'm not sure the conclusions of the review make much sense at all. How can you interpret studies of this quality as moderate/strong evidence???

18/n I'm also worried about the heterogeneity of the 'usual care' groups across studies. Some of the usual care:

- coffee
- paracetamol
- syrup
- diphenhydramine
- placebo
- prednisolone

These are very much not the same!

19/n That's a big issue because you can't really combine the effect of coffee with diphenhydramine and expect it to make sense, but that's what the authors did

20/n Ultimately, I think the only real conclusion you can draw here is that we have very little idea whether honey reduces symptoms for URTI/cough, and that the research is quite problematic

21/n Not something that'll make headlines, perhaps, but sadly that's often how these things go

22/n N.B. this study has already hit ~800 on Altmetric, been covered internationally, and made huge news

23/n Something else I didn't mention. Every study that I've looked at so far used a per-protocol analysis, which is a huge and worrying issue

All of these should be at a high risk of attrition bias, yet none were rated as such. Most of them were green (low risk)

24/n I've gotta say, for anyone teaching students about how finicky bias can be in systematic reviews, this is a beautiful example of getting it wrong

25/n Somehow there are more issues here. This study was included in the risk of bias, but even though it assessed honey vs placebo/salbutamonl (and found no effect) it is not in any of the meta-analyses

Very weird

26/n This is even weirder when you consider that the authors report excluding a study for not providing data

So they exclude one study and report it, but another just...disappears? So strange

27/n Another included study using a per-protocol analysis. This was at least rated correctly as at a high risk of bias

28/n Another one. This study was rated at low risk of bias for most domains. Here's how they described their randomization and allocation concealment. What do you think?

29/n I'll give you a head-start - if they literally don't report ~how~ patients were randomized, by definition this should be unclear or high risk of bias for the domain of random sequence generation