Good thread on sensitivity & specificity of first emergency use authorization (EUA) #COVID19 serology test, Cellex, means for testing a low-prevalence population: increased risk of false positives

BUT IT'S ACTUALLY WORSE THAN THIS bc these numbers =/= *clinical* sensitivity 1/ https://twitter.com/zbinney_NFLinj/status/1245789672833417217

A huge question (if not THE question) in #SARSCoV2 lab diagnostics is the rate of false negatives and false positives.

This can be calculated IF (and it's a crucial "if") you know the True Disease Status of people and then test them with your test approach. Like this:
2/

BUT we don't really have a gold standard/reference method for determining True Disease Status of COVID-19 patients. But we think PCR tests are the best we have right now.

SO you contrast your NEW test vs. results against your best comparator test.

Results look like this: 3/

"Percent agreement" vs. "sensitivity/specificity": The squares LOOK the same & people may use "sensitivity/specificity" when they mean "percent agreement" (as w/ the above thread).

But it's more than pedantry to distinguish between them (see @JClinMicro's relevant comic): 4/

Let's look at Cellex's point-of-care SARS-CoV-2 IgM/IgG serology test. This test info is on the FDA's COVID19 EUA materials page.

Relevant questions:
1. What samples were tested?
2. What was their comparator test?
3. What are the limitations?

5/n https://www.fda.gov/medical-devices/emergency-situations-medical-devices/emergency-use-authorizations

First, we're validating a serology (antibody, IgG IgM) test.

To determine test characteristics, they need samples from patients who most likely WILL have produced antibodies compared to patients who most likely WON'T have antibodies.

Here's what Cellex did: 6/n

All "positives" were from confirmed PCR cases.

98 were mild/no symptoms (point 1). 30 hospitalized w/ severe symptoms (point 3).

All negatives were banked samples from before Sept 2019 (points 2 & 4). As a novel virus chances of SARS-CoV2 antibodies prior to then was nil

7/n

So what was the outcome of comparing these PCR-confirmed COVID-19 cases vs. the Cellex serology test?

93.8% positive and 96.0% negative agreement.

(Side note: It would be interesting to know the performance in the 98 low/no symptom group to the 30 severe symptom group)

8/n

To answer our above questions about this test evaluation:

1. What samples tested? Serum from known PCR pos patients OR serum from before Sept 2019

2. What was the comparator test? PCR

Last is what might be the most important question:
3. What are the test limitations?

9/n

There's a lot of open questions re: COVID-19 serology.Not to review all of them but they include…

- When do antibodies arise in acute disease? (answer: too late to use it for acute diagnosis)

Here's a relevant pre-print - note sensitivity at d0-d7 https://www.medrxiv.org/content/10.1101/2020.03.23.20041707v1

10/n

Another COVID-19 serology question is when do antibodies arise in course of disease?

From the same preprint above ("Serology characteristics…" from Lou et al), this figure, normalized to date of exposure, showing illness onset.

Answer: it varies 11/n https://www.medrxiv.org/content/10.1101/2020.03.23.20041707v1

Still more COVID-19 serology questions:

-What kind of cross reactivity will there be seasonal Coronaviruses?
-Is anti-SARS-CoV2 IgG protective? (maybe? hopefully!) How long lasting is it?

etc.

Lots of questions remain is my point. 12/n

As it relates to point-of-care serology tests, for these questions we need to make best guesses abt what MIGHT happen, what is PROBABLE to happen and what would be the consequences if it DID happen.

That's lots of words to say we REALLY need to understand test limitations! 13/n

Limitations from Cellex's test materials (here: https://www.fda.gov/medical-devices/emergency-situations-medical-devices/emergency-use-authorizations).

First, and my main focus here: We don't know the clinical sensitivity, specificity or predictive value of these tests. They aren't adequate for #COVID19 diagnosis & aren't equivalent to PCR tests.

14/n

I want to be clear: developing and using serological tests? Crucial!

But this limitation, that antibody tests shouldn't be used to diagnose acute disease, to me is just a matter of biology: people likely don't produce antibodies early enough to diagnose an early infection.

15/n

Cross-reactions causing false positives (eg antibodies vs other Coronaviruses) and false negatives (needing PCR tests to adjudicate) are both addressed in the package insert.

These POC serology limitations are crucial and I'm VERY worried they're underappreciated 16/n

I'll be blunt: I know there is widespread unavailability of #COVID19 tests. And the need for SARS-CoV-2 serology testing for epidemiological tracking and immunity determination is clear.

But I don't see this type of rapid serological test filling either of those needs. 17/n

To end back where we started, EVEN IF the Cellex serological lateral flow test had 93% sensitivity 96% specificity (and to be clear, that's % agreement w/ a comparator test), in a low-prevalence setting like we have, the risks of false test results would be non-negligible. 18/n

That, together with the other inherent limitations to what serology tests can and can't determine, I'm afraid that there's little utility for this or any rapid or point-of-care serology tests and high risk they'll be used and interpreted too broadly.

The End.