Covid testing is big news right now: with a mass testing pilot starting in Liverpool, Boots announcing rapid test plans, and places like universities testing staff and students.

Here’s a thread on some important statistical aspects of rapid Covid testing. (1)

Judging by the available and published data, there is much uncertainty over how well new rapid tests will perform in practice... (2)

In particular, don’t be misled by claims like "100% accurate" for new Covid-19 tests. Quick testing explainer: Two key statistics for understanding the accuracy of a test are: *sensitivity*, and *specificity*. (3)

Sensitivity measures the chance of getting a positive test if you have Covid-19. A good test has a high sensitivity: you want it to report positive if a person has Covid-19. (4)

For example, if a study of a test reports 98% sensitivity, the test found 98% of cases and missed 2% of cases (there should also be a confidence interval around this result). (5)

Specificity measures the chance that the test turns out negative if you don’t have Covid-19. High specificity is important too: you don’t want a test to report positive if somebody doesn’t have Covid-19. (6)

Sensitivity and specificity measure different aspects of a test, so both are important. To reach a claim like “100% accurate”, reports may have ditched one or other of the statistics. (7)

With imperfect tests, it is clear that a fraction of positive cases will be missed, and that a fraction of negative cases will be wrongly reported positive. (8)

We do not know how many will be affected by this when new, rapid tests are brought into mass use. One reason is because the evidence that is initially made available about tests’ performance is... very limited. (9)

What are some problems we see about evidence for test effectiveness? Well...
(a) a test may have been validated on samples from patients in hospital, who have symptoms. (10)

(b) the diagnosis of the sample may have first been confirmed by at least one other test (generally PCR or molecular tests which perform well at finding the virus, but sometimes also other, less sensitive tests). (11)

(c) the test may have been validated to use only on samples that were taken within a certain number of days of symptoms.
(d) A large quantity of samples may have been taken from a small number of patients. (12)

All of these features *should* be reported by the manufacturer to @MHRAgovuk and the equivalent regulators in other countries... but they may not be made publicly transparent, so it's hard to scrutinise the evidence. (13)

We need more reporting of testing evidence so that the key parameters of manufacturers’ studies are made public, and so that the limitations of devices are known. (14)

Even with the best access evidence however, a big challenge of mass testing will always be that this involves very large population groups, and imperfect tests. (15)

In particular, we need to understand how well testing performs for three sets of people:
(i) those who have symptoms but are not severely unwell
(ii) those with an active infection but no symptoms
(iii) those who have no symptoms at time of testing but develop them later (16)

We would expect all Covid-positive results to be re-tested initially, and that lab PCR tests (= more established, can identify asymptomatic infections, but take 24+ hours) could confirm the diagnosis for a good proportion of those taking part (17)

Phew, that was long! Anyway, the key takeaway is that too often, tests are coming into use without the key facts about them being known. (18)

There are three things that could be done to fix this:
(1) Government should consult experts and publish research and evaluations in advance of new policy announcements abut testing (19)

(2) Companies planning to sell new Covid-19 tests should be clear about the limits of evidence on particular products - and avoid misleading or unproven claims. (20)

(3) @MHRAgovuk should ensure that incorrect communications to the public are corrected, and withdraw tests from use if they are being wrongly used. (21/21)