A new study has come out that hit headlines everywhere arguing that there's no safe level of caffeine intake during pregnancy

I think it's worth a bit of a peer-review on twitter 1/n

2/n I should at the outset say that I have a bit of skin in the game - I've previously looked at the evidence and argued that the harms at low levels of intake are not as worrying as they are often made out to be
https://www.theguardian.com/commentisfree/2019/oct/17/is-drinking-coffee-safe-during-your-pregnancy-get-ready-for-some-nuance

3/n The study itself is here, in BMJ Evidence-Based Medicine: https://ebm.bmj.com/content/early/2020/07/28/bmjebm-2020-111432#DC1

4/n The first problem comes right in the title - this is a 'narrative review'

This is an issue because narrative reviews are generally not very good to rely on as proof

5/n This is because of how they are conducted. Narrative reviews are, by nature, non-systematic reviews of the literature. In other words, they are 'expert opinion' level of data - they can inform, but it's hard to make conclusions based on them because of the methodology

6/n We can see quite clearly why when we read the methodology here. This is the entire methods section - 1 paragraph! - and doesn't tell us NEARLY enough about how the included studies were found

7/n Were studies excluded? If so, why? Could the included studies simply be cherry-picked for the authors conclusions? What's the risk of publication bias, or other biases?

We simply have no idea!

8/n So right off the bat, we can't really say, from this study, much more than that this is one professor's expert opinion based on studies that they have identified

9/n Now, there's nothing wrong with that. Professors are smart, and often their opinions are very important

But this isn't ~evidence~ in a scientific sense. The review says nothing new per se except that Prof James, Psychology, has this opinion based on studies he's read

10/n On to the included studies

There are A LOT of them

Kudos to the author for reading all of this literature

11/n Now, the problem is of course that to properly review a narrative review like this you would have to read every included piece of research, and with >40 studies to go through that's really quite a lot to do

12/n But, even if you look at just a few, some issues emerge. The single biggest study on caffeine and miscarriage, for example

In this review, it's cited as evidence that maternal caffeine consumption increases risk of miscarriage

13/n But actually, Gaskins et al found something a bit different. In their study, decaf coffee was associated with a higher risk and caffeinated tea was not

Therefore, caffeine was NOT the likely culprit!

14/n This gets to the heart of the issue with a narrative review. It is, essentially, the author's opinion of each study and what they mean - no systematizing, no group of authors carefully reading through each point to be careful

15/n One person's evidence of the risk from caffeine is another person's evidence of safety

Case in point - this systematic review

16/n In the narrative review, it's cited as evidence that any amount of caffeine is dangerous, but the authors actually concluded that their results were consistent with the WHO recommendation to limit caffeine consumption to <300mg per day during pregnancy

17/n Similarly, the Greenwood et al meta-analysis is cited as evidence that even small amounts of caffeine are probably harmful, but the authors don't really agree!

18/n All of this is fine, and just the limitations of the narrative review. The entire purpose of such a document is to provide a narrative decided on by the author, which is neither good nor bad - it's just how these things work

But that's not all of the issues

19/ If you get to the last part of the paper, you'll see this table

It contains a fundamental error that makes the numbers in the last column incorrect

20/n This table is calculating the population attributable fraction of various pregnancy outcomes. The last column is the context - for the ~1 million miscarriages in the US each year, 280,000 are caused by caffeine!

Shocking. And untrue

21/n This table has several errors. The first is in column (i)

This is cited as the risk per 100mg. In fact, it is the RELATIVE RISK INCREASE per ADDITIONAL 100mg of coffee consumed

22/n This is then used to calculated the risk AT 200mg of miscarriage - double the risk at 100mg

Except, this is incorrect

23/n The BASELINE risk of miscarriage, for a woman who drinks NO coffee, is around 16%. The RELATIVE increase is 14% per 100mg of coffee. So the ABSOLUTE risk for a 200mg per day drinker is:

0.16*1.28 = .205 = 20.5%

Thus, all the numbers in column (ii) are wrong

24/n The last column calculates the number of caffeine-related events by multiplying the risk in column (ii) with the total number in column (iv)

So, 28% risk and 1 mil miscarriages means 280k miscarriages related to caffeine

25/n Think about this for a second. Even if that number was correct - i.e. 20.5% rather than the wrong 28% - it doesn't make any sense because it assumes that 100% of women who have had miscarriages drank 200mg of caffeine a day

26/n This number - called the Population Attributable Fraction (PAF) - is actually calculable. The formula is pretty simple:

27/n In this case, with a prevalence of ~35% for caffeine intake of 200mg or higher, the number of miscarriages attributable to this is:

0.35*(1-(1/1.28)) = 0.763 = ~76,300 miscarriages

So, about a quarter of what's in this study

28/n This actually makes perfect sense when you think about it. Remember, the ABSOLUTE risk increase of miscarriage for women who drink caffeine is pretty small - 1-2% - so it doesn't really make sense to say that up to a third of miscarriages are caused by caffeine

29/n And yet, these figures were picked up in the media and thrown around as if certainly true, which is very worrying!

30/n More broadly speaking, do we know that caffeine causes bad pregnancy outcomes?

That is a VERY tricky question to answer

31/n This narrative review argues yes, absolutely, and while most people would agree a LOT of caffeine is a problem, much previous research has said that small amounts are probably not so bad

32/n This narrative review argues that all such studies are the product of industry funding, which is not entirely untrue, but also doesn't really tell the whole story given that it disagrees with non-industry studies as well

33/n Ultimately, reviews like this come down to opinions

Do we think that Prof James is correct and caffeine is definitely bad? Well, maybe. It depends on your perspective I think

34/n What we can't say is that this is new evidence, or even that we've proven anything here. It's one well-researched opinion that you may or may not agree with

35/n Anyway, regardless of what you think of the review, it's probably worth correcting the inaccurate table @BMJ_EBM

36/n Also, for the record, I think the conclusions are pretty massively overstated, and also miss out a key part of recommendations which is the question of whether they'll be followed!

37/n I should also say that I thoroughly dislike opinion pieces and media attention like this

The evidence hasn't changed, and quite frankly I think pregnant people have enough to worry about without the fear that a cup of coffee will cause a miscarriage