You may have seen a mini-Twitter storm on COVID tests claiming they falsely tell people they have COVID when they do not have COVID

Other’s have done a good job correcting this

I want to show how we work what the results of a COVID test actually mean

/1

The first things we need to know are how common is COVID-19 and how accurate is the test we are using to identify if someone has or does not have COVID.

Let’s focus on the first question - how common is COVID-19?

/2

When answering this question, we need to consider what group of people are we looking at to determine how common COVID is? Are we talking about how common it is if we look at everyone or a specific group. Everyone would include people who have symptoms and those who don’t.

/3

A specific group might be only those people who display symptoms. For example, in England you are only eligible for a COVID test if you display symptoms. Let’s use this as our group for this exercise.

/4

The latest ONS report estimates 28,200 people during the week 14 to 20 August 2020 had COVID-19, equating to 1 person in every 1,900 tested. This is around 0.05% of this tested population: http://ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/coronaviruscovid19infectionsurveypilot/englandandwales28august2020

The 0.05% value is referred to as the ‘Prevalence’

/5

A key point to make here is that the figure you use for prevalence is possibly the single most important number. When you see someone stating anything on the accuracy of a test/test results, ask yourself if they provide a figure for the prevalence of the disease/condition...

/6

...and if they do, they should give a source for that figure so we can check if it's reliable (ie. based on good evidence). You may not be able to determine the reliability, but you should expect to see a prevalence figure.

/7

Once we have our estimate of prevalence (e.g. 0.05%), we next need to look at the test we are using to tell us if someone has COVID or not. There are a few things we use to determine how useful a test is.

/8

First thing we want to know is the proportion of people who HAVE COVID and correctly have a positive test result. This is what we call the ‘Sensitivity’ of the test (scientists are great at coming up with words that everyone else won’t understand - this is an example).

/9

Sensitivity tells us how good a test is at correctly picking up people who do indeed have the disease (e.g. COVID)

The 2nd thing we want to know is the proportion of people who DO NOT HAVE COVID & correctly have a negative test. This is called ’Specificity’ (I know, sorry!)

/10

Specificity tells us how good a test is at correctly identifying people who do not have the disease.

If you’ve followed so far (well done), you may at this point ask “Hang on! What I want to know is if I have a positive test result do I have COVID?”. That makes sense.

/11

For this we need to determine something called the ‘positive predictive value’ or ‘PPV’ - the proportion of people with a positive test who HAVE COVID. Think of this as telling you the chance you have COVID if you test positive.

/12

We might also want to reassure those who have a negative test that indeed they DO NOT HAVE COVID. This is called, you guessed it, the ’negative predictive value’ or ’NPV’. Think of this as the chance of not having COVID if you test negative.

/13

Now we can bring in some actual numbers to help illustrate how this all works. Quick recap - we have our estimated prevalence of COVID in tested people w symptoms during one week in August in England as 0.05%. We don’t yet have the test to tell us if we have COVID or not.

/14

In the twitter storm example mentioned, the test used was a PCR test (don’t worry about this abbreviation - it’s just the technical method the test uses to detect coronavirus). There'll be estimates for the sensitivity & specificity of the PCR test based on existing studies

/15

I won’t go into this evidence here as I want to teach the principle of how we assess if a test is useful or not. Here we have a second key point when you see someone talking about test accuracy - what are the estimates for sensitivity & specificity and are they reliable?

/16

I have chosen a sensitivity of 90% and a specificity of 99% (which I believe are within the range of estimates for sensitivity and specificity for PCR testing in the existing literature). We can now look at what this means using numbers.

/17

Let's start w 10,000 people randomly drawn from everyone tested (in those with symptoms). With a prevalence of 0.05%, 50 of these 10,000 actually have COVID. 9,950 do not. With a sensitivity of 90%, 45 (90% of 50) of the 50 with COVID will correctly have a positive PCR test

/18

But 5 (10% of 50) of the 50 will incorrectly have a negative test.

We can also add labels to each of the results:
1. Has COVID with +ve test = true +ve (TP)
2. Has COVID with negative test = false -ve (FN);
3. Doesn’t have COVID with -ve test = true -ve (TN);
....

/19

4. Doesn’t have COVID with +ve test = false +ve (FP)

The main factor behind the twitter storm was the value given for the ‘false positives’ - given as 90% to 91% (both are wildly wrong as you will see).

/20

Some how the 90%-91% 'false positives' figure appears to have been wrongly extrapolated from something called the 'false positive rate’ (FPR), which was given as 0.8%.

The FPR is basically the proportion of all negative tests that are wrongly (falsely) classed as positive.

/21

We calculate the FPR by dividing the number of people with a false +ve by the total number of people with negative tests (= FP (FP + TN)). Here we can see with our test the FPR is 1% (which also corresponds to 1 minus the specificity = 1-0.99 = 0.01 (or 1%)).

/22

So I’ve no idea where the 90% - 91% ‘false positives’ comes from.

But remember - we intuitively want to know “I’ve had a positive test, do I have COVID?”. Another way of saying this is what are my chances of having COVID if I have a positive test.

/23

But before we do that, let's have a little test of your predictive powers. See the picture below - give your best guess estimate as to how likely (the probability) a person has the disease given a positive test (complete the poll in next tweet)

/24

How likely is that a person has the disease given a positive test

(prevalence = 30%)
(sensitivity = 50%)
(specificity = 90%)

/25

I'll give the answer at the end of this thread.

So let's get back to our COVID tests. In our same 10,000 people with a prevalence of 0.05%, sensitivity of 90% and specificity of 99%, it turns out your chance of having COVID if you have a positive test is 31%.

/26

The 31% figure is our positive predictive value (PPV) as mentioned earlier. Here is the calculation of the PPV directly.

/27

We also want to know “If I test negative, am I definitely free of COVID?”. Running the numbers tells us you are 99.9% likely to not have COVID if you test negative (under the conditions of the test as mentioned).

/28

Remember I said prevalence was an important factor? Let’s look what happens when we change prevalence, to say 10% (100 in every 10,000 actually have COVID). Let's keep sensitivity (90%) & specificity (99%) the same. Your chance of having COVID if you test positive is now 91%!
/29

Your chance of being COVID free if you have a negative test is now 98.9%.

/30

So the morals of the story are:
1. Understand what the numbers related to test accuracy mean, how they are generated and the key factors that impact on them;

/31

2. If you see someone talking about test accuracy, ask have they given reliable estimates of the prevalence of the disease/condition and the sensitivity/specificity of the test(s);

/32

3. I hope this demonstrates that some 'experts' (also known as dedicated, hard-working folk like you) have important skills/knowledge that can make real-world important contributions
4. Don’t rely on media broadcasters for these skills

/33

Great to hear if you found this useful.
Happy to be corrected on my numbers.
Let me know if you want more like this.
Feel free to use my slides (nice to be acknowledged).

End.

On tweet 29 of this thread I should have included a picture of the calculations for the revised PPV when prevalence is increased to 10%. Here it is

'DO I HAVE COVID'

Updated based on the latest prevalence of COVID at 1 in 500 (0.2%) from @ONS

And

TEST SENSITIVITY = 90%
TEST SPECIFICITY = 99%
FALSE POSITIVE RATE = 1%

"I'VE TESTED POSITIVE - DO I HAVE COVID?"

LATEST - based on lates ONS rate of 1 in 160 with COVID = 0.625% prevalence.

TEST SENSITIVTY = 90%
TEST SPECIFICITY = 99.8%
FALSE POSITIVE RATE = 0.2%