Read on Twitter
Antibody tests are coming online. Never before have humans needed to understand Bayes rule more. Let's talk about why it's critical NOT to assume you are immune to covid-19 when you have a positive antibody test. Seriously, people need to understand this to prevent many deaths.
Several companies have tests of varying quality, but Cellex makes a FDA-approved antibody test. The goal is to test for immunity due to prior exposure rather than an active infection. Cellex's test has a "sensitivity" of 93.8% and a "specificity" of 95.6%. Lets interpret this
The Cellex test appears to have higher rates than others. Sensitivity: Probability person tests positive if they are infected. Specificity: Probability person tests negative given that they are negative. 93.8% and 95.6% sound good, they mean that only 6.2% of + people test -, and
only 4.4% of - people test +. But what does it mean when you test positive? Since the sensitivity is high, you'd assume that you're positive and immune to covid-19. But sensitivity measures prob testing + given that you are +. We want prob of BEING + GIVEN that you TEST +
The conditionality is reversed. Probability you are + given that you test positive can be written as Pr(are+|test+). Let's just derive Bayes theorem real quick in this context so you don't have to just trust me. Pr(are+|test+) equals the sensitivity Pr(test+|are+) multiplied by..
the probability you are+ irrespective of how you test, which is the same as the proportion of positives in the population. Let's assume that in the USA this is ~1% or ~3.3 million people are really infected. The final term in the denominator is the probability you test positive.
The probability you test positive must be decomposed into the people who test positive and are positive plus the people who test positive and aren't. We have all the necessary numbers for this probability expression and get a denominator of 0.053. Plugging it all in to Bayes rule
If we plug in sensitivity, Pr(test+|are+), the probability a random person is positive, and the probability a random person tests positive, then a randomly tested person who TESTS positive, only has a 17.7% chance of actually BEING positive. More that 82% of people will be wrong!
If you get a positive antibody test result and think you're immune and quit socially distancing, you are far more likely than not to make a big mistake. When millions of people make the same mistake rather that appreciate conditional probability and Bayes rule, many people die.
But it gets worse. If you never had symptoms but test positive and conclude you're one of the lucky people who had an asymptomatic infection, you're even more likely to make a big mistake. Of course, as a higher proportion of population becomes infected, this mistake diminishes.
The take home is that a high sensitivity, Pr(test+|are+)=0.938, does NOT mean that Pr(are+|test+) is high. 82 of 100 people who get a positive test will still be susceptible to the virus. Once 10% of population has been infected, this drops to 30%, but that is a LOT of people!
I foresee that someone will say that more than 1% of people have now been infected. At a prevalence of 5%, 47 of 100 people who test positive will still be wrong. I found an online tool that lets you explore this for various quantities.
http://vassarstats.net/clin2.html
http://vassarstats.net/clin2.html
Since this tweet is being spread around, some addendums and an attempt at communicating with a general audience. 1. Context is testing of random population. If you had severe symptoms in the past, the probability of false positive is lower because Pr(are+) is higher.
2. Prevalence varies locally, so testing random New Yorkers for antibodies will produce fewer errors than random Oregonians.
3. Point of this tweet was to teach some math and an important qualitative message about being careful with a single + test.
3. Point of this tweet was to teach some math and an important qualitative message about being careful with a single + test.
4. Now an attempt at explaining this idea for general audience. Say you test 1,000,000 random people and prevalence is low so only ~10,000 had been infected. Those 10,000 will be picked up with sensitivity 93.8%, so 9380 will get positive test. Only 4.4% of negative people test +
But that is 4.4%*1,000,000=44,000 false positives. 44,000/(44,000 + 9380) = 0.82. So in this example, 82% of the people with positive tests were never infected and are still at risk. There it is w/out math lesson. If you didn't have severe symptoms, a second + test really helps.
My bad, it's 4.4% of negatives. So 4.4%* (1,000,000-10,000)=43560. 43560/(43560+9380)= 0.82. So still 82% but the math was off slightly. This is how you get in trouble with quick tweets while morning coffee hasn't sunk in!
5. Finally (maybe), RT-PCR, which is used to test for current infections is much less susceptible to this issue because false positives are much more rare. Someone "PCR-splained" me about this (Our genetics lab does many thousands of PCRs every year and designs assays)
6. I lied, one more thing. This is not some great insight that I had or a fancy model. This is one of the first things you learn in Bayesian statistics. Every statistician knows this, but many MDs do not (at least that's what statisticians say). I just took the time to tweet it.
People much better at science communication are explaining this better and far more succinctly. https://mobile.twitter.com/amykatherine/status/1248787138704887808
And this visual summary is great. I like getting into the details since typically there are lots of examples without the math worked out, but somehow I've seen nothing about this false positive problem. https://mobile.twitter.com/LCWheeler9000/status/1248781564852363265
7. This is an educational thread. I'm in no way stating that effective serology to test for prior exposure can't be done. For instance, people testing positive for HIV with an ELISA would then follow up with a Western Blot. I have seen no such proposal. Point is...think carefully
If you enjoy learning math in this context, here is an old thread about the trajectory we were on before distancing and could stll get on. Exponential is a good approximation only while prevalence is low and almost everyone is susceptible. https://mobile.twitter.com/taaltree/status/1239763806433116160
And if you want to learn how basic epidemic models work... https://mobile.twitter.com/taaltree/status/1240039926759952384
It appears WHO wants to restrict these tests to labs for exactly this reason https://www.who.int/news-room/commentaries/detail/advice-on-the-use-of-point-of-care-immunodiagnostic-tests-for-covid-19
My day job is teaching/research and I thought teaching on Twitter could be useful. Here is an older post about the OTHER kind of testing used for active infection. https://mobile.twitter.com/taaltree/status/1240684651649220613
And....Yesterday Pence said the U.S. would create more than 20 million new antibody tests a month. There are going to be a lot of people with a false sense of security.
The FDA recently approved (emergency use) tests from Chembio and Ortho. Haven't found the specs yet. Abbott labs also has a new antibody test. Very possible one will have much higher specificity demonstrated with decent sample size. That'd be great https://www.centerforhealthsecurity.org/resources/COVID-19/serology/Serology-based-tests-for-COVID-19.html
NY doing a good job telling people how not to interpret even their much better test.
https://www1.nyc.gov/assets/doh/downloads/pdf/han/alert/2020/covid-19-status-of-serologic-testing.pdf
https://www1.nyc.gov/assets/doh/downloads/pdf/han/alert/2020/covid-19-status-of-serologic-testing.pdf
