Let’s talk COVID testing. Public needs to know how they work, why they can be wrong, what decisions to make based on imperfect “specificity”, & how to tell when a re-infection maybe isn’t. I’m a virology-trained, diagnostics investor & this explainer involves attack dogs.

Specifically, I’ll tell you about tests that detect whether a person has developed antibodies to SARS2. These are called “serology” tests (blood “serum”). They don’t actually detect presence of the virus in a person but rather they detect evidence that your immune system…

…has seen the virus & reacted to it. Consider viruses to be replicating clones of a bad guy & your immune system consists of police officers and their trusty attack dogs. When our immune systems see a virus for the first time our police basically chop virus clones they catch...

…into lots of pieces & train dogs to spot each piece. I know dogs are better at smelling, but that won’t really work for this analogy, so let’s pretend these are dogs that just use their sight. So with the virus broken up into a jigsaw puzzle, individual dogs (antibodies) are...

...trained to recognize unique puzzle pieces. Maybe one is keen on a puzzle piece of the bad guy’s left eye on it, another focuses on a scar, & another on the right index finger. These antibody-dogs circulate around our bodies (the city) looking for their special...

…part of the bad guy clones. When they come across it, they bite down on that spot (technically, bind to it). That does one of two things: 1) If a dog has bitten the bad guy’s right index finger, it can prevent the bad guy from using his hand…

…to turn a doorknob (ie. the ACE2 protein on surface of our cells) to break into (infect) a house (a cell). So the antibody-dog is said to be “neutralizing”. That’s good. But 2) an antibody-dog might also not be neutralizing yet still grab onto the bad guy, e.g. by his shoe.

That will slow him down but won’t stop him breaking into a house. So that’s not very useful, but it’s not entirely useless. Antibodies don’t work alone. They attract immune cells (police!) which grab onto antibody & arrest the bad guy. If house is infected & spewing out…

…bad guy clones, then dogs attach to bad guys’ legs, hands, & noses as they climb out the windows to try to infect other houses. And so police officers will be attracted to the house & burn it down. Ok, that’s inhumane, but pretend this is a Terminator movie & bad guys…

…are evil robots. This tag team of antibody-dogs & immune cell police is how our immune systems fight off the invading bad guy clone army. This is an expansion of analogy I introduced in another thread that explained how SARS2 is more clever than SARS1. https://twitter.com/PeterKolchinsky/status/1246975275021348865?s=20

I can also expand this analogy to explain what “cytokine storm” is, in case you’ve heard that term & wondered. Imagine that a neighborhood (lungs) gets so infected that every block has a building occupied by bad guys. So the army is called in, bring out big guns, & wages…

…such a battle that the city (the person) is leveled in the process of being saved (think Avengers). That’s why some of the drugs we’re testing now are meant to prevent such escalation to let a more modest police response handle the infection. So after the city is saved…

…by the police, they remain on alert. Knowing that this SARS2 bad guy clone army is always out there, some of the police & their dogs continue to patrol the city forever looking for signs of the bad guy. That’s what we mean by immunity. If the bad guy shows up at the city…

…boundary again, odds are good that a dog will spot him, or really only a part of him. “What’s that? A familiar pinky! Get him!”. Now, we don’t actually need to get infected by an actual bad guy to develop immunity. We can get a vaccine. A vaccine is a picture of the bad guy…

…that the police can chop up into pieces to train themselves & their dogs on what to look out for. Getting & recovering from measles is a terrible way to become immune. Way better to skip the infection & gain immunity from a measles vaccines. We get vaccines…

…against all sorts of bad guys (eg measles) while we’re kids, which saves millions of lives each year. Most bad guys don’t change their appearance so getting the vaccine once (though typically involves 2-3 shots over few months) is enough to last us a long time.

It’s true that the police & their dogs may forget what the bad guy looks like after some time. A vaccine can remind them. That’s what the shingles vaccine does, reminds us of what the chicken pox virus looks like. But in the case of a bad guy like flu that changes its…

…appearance pretty often, we have to keep track & show the immune system updated pictures. But coronaviruses don’t mutate their appearance very quickly, so we’ll be able to just expose our immune systems to the same vaccine periodically to keep them well trained…

…to recognize the same SARS2 threat from year to year. I’ve written extensively about why a SARS2 vaccine won’t need to be updated very often b/c SARS2 mutates much less than flu (another analogy involving copy machines). https://www.city-journal.org/coronavirus-vaccine

So let’s get back to the issue of diagnostic tests that look for those SARS2 antibodies. Here’s how they work. When we take out some blood, it’s like we just sucked up some of the stuff from a neighborhood of their city, incoming any police dogs on the street. If we want…

…to know whether there are any antibody-dogs in there that recognize SARS2, then we just need to put bits of SARS2 on a surface of the diagnostic test plate, pour blood over it, & let antibody-dogs spot they pieces they have been trained to recognize & bite down on them.

Then we wash away blood & any unattached antibodies. It’s not easy to see the antibody-dogs attached to the surface, so, get this, we send in special ”lightning fleas” (cross between glowbug & fleas) that I just made up for this analogy. (real scientists, turn away).

If there are SARS2-obsessed dogs busy biting down on the bits of SARS2 that we stuck to the surface of our test dish, the lightning fleas jump all over them & make the dogs glow. Then we wash away any fleas not stuck to a dog & we look at the diagnostic test dish.

If we see it’s lit up, we know that blood must have had SARS2-targeted antibody in it. Which means that city (person) we just drew the blood from must have previously been infected by SARS2 bad guy. If there is no light shining from the test, there must have been…

…no dogs for flees to attach to, which means there were no antibody-dogs in the city that could recognize & attach to bits of SARS2 stuck to our dish. But here’s the problem. The antibody-dogs only recognize their particular jigsaw piece, not the whole virus.

So it’s possible that a person had COVID, has a trained immune system w/ antibody-dogs that recognize certain puzzle pieces of SARS2, but just not the ones that the diagnostic lab chose to attach to surface of their test. So test would say that person doesn’t…

…have SARS2 antibodies, suggesting that they weren’t infected, when really they were. A test that misses such cases of SARS2 infections is said to have a “false-negative” result. If a test is 97% sensitive for prior SARS2 infection, it means that out of 100 people…

…recovered from SARS2, it will incorrectly tell 3 of them they don’t have antibodies, making them think they were never infected. So what’s the harm from a false-negative result? Such a person would continue to be cautious & try not to become infected. That will keep them alive.

If they need to get back to work but are only allowed if they can prove that they recovered from COVID19 then a “less than 100%“ sensitive test will leave some recovered people unable to prove their status. That’s unfortunate for everyone who can’t work…

…from home & so we should strive for more sensitive tests. But now let’s consider the opposite problem. The test might also tell a person that they have antibodies to SARS2 when they actually don’t and have never been infected. How might that happen? Well, consider again…

…that the antibody-dogs were trained to recognize specific pieces of SARS2, not the whole virus. While two different viruses can look very different overall, they can have pieces that are the same just as two different people could have the same looking elbow.

There are four other coronaviruses that regularly infect humans & merely cause common colds. We’ve all been infected by them and have antibodies against them. So some of the antibody-dogs that we all trained from our past common cold coronavirus infections…

…can recognize and bite down on similar looking parts of SARS2. Why aren’t we protected against SARS2, then? Because there are very few parts in common & they aren’t important parts like the hands. If the common cold antibody-dogs recognized critical pieces…

…like the hands (ie receptor-binding domain of Spike protein), they could neutralize SARS2 by keeping it from turning doorknobs (ACE2) to break into homes in our city (infect cells in our body). If that had been the case, we probably never would have heard…

…of SARS2 b/c it wouldn’t have turned into a pandemic in the first place. The fact that SARS2 has spread so wildly is precisely b/c it has so little in common of importance with other viruses to which we have immunity.

So whatever confused, cross-reacting antibody-dogs trained on common cold viruses some people might have, they tend to be pretty harmless to SARS2, maybe recognizing a shoelace or button on the SARS2 jacket, & don’t do anything to protect a person from infection w/ SARS2.

Still, when diagnostic test looks for SARS2 antibody-dogs in blood sample of a person who’s never been infected, then if there are enough of these confused antibody-dogs attaching to diagnostic test surface, e.g. biting into a familiar looking button, the test lights up positive.

So now we have a person who actually has never been exposed to SARS2 (merely other coronaviruses) but thinks that they have recovered & therefore are now immune. When a diagnostic test makes such a mistake, we call it a false-positive result.

If there are 100 people who definitely have not been infected with SARS2 & therefore definitely shouldn’t considered themselves recovered, & a test correctly says that 95 of them are negative (don’t have antibodies that recognize SARS2 in them) then we say…

…the test ”has a 95% specificity” or ”is 95% specific”. & that means that other 5% were false-positives. So just subtract the specificity of a test from 100 & you’ll know its false-positive rate. Here’s what’s so bad about a specificity less than 100%. A person who has…

…been given a false-positive antibody (serology) test result might think they can go back to work w/o being worried about SARS2 b/c they are immune. They could become lax in their social distancing & hand washing. They could become infected & help spread the virus.

But here’s what else could happen. When a person who falsely thinks they have recovered from COVID19 b/c of a false-positive serology result actually does become infected for the first time, & maybe even dies, media will report that as a “re-infection”, inciting fear.

That could undermine our understanding of just how protective our immune response to a real infection actually is. So it’s critical that the diagnostic tests that are administered to the masses have a very high specificity, even if that means tuning them to be less sensitive.

Indeed, depending on how many of the lightening flees you add to the test, you set the threshold so that it is more likely to miss low levels of SARS2 antibodies (so it’s more likely to give a false-negative result, saying you weren’t previously infected even if you were)…

…but also less likely to light up as positive from the low level of confused, cross-reacting antibodies that SARS2-negative people have simply from their past colds. That’s a general rule of diagnostic tests: there’s a trade-off between…

…sensitivity & specificity, which can also be said to be a trade-off between false-negatives & false-postiives. Better to have a SARS2 person be a false-negatives (be told they weren’t infected) who has to remain cautious and who, if they have symptoms, we’ll still treat…

…as if they are positive, than to have false-positives we falsely think are immune. So whenever you hear about a diagnostic test, look at its specificity. If you see, let’s say, 97% specificity, here’s what that means.

If you live in a region where an estimated 8% of people have been infected (that’s known as the prevalence number) & people want to know if they’re one of them, then testing 10,000 people (of which 800 have truly been infected & 9,200 have not) w/ a Dx test…

… that’s 95% sensitive & 97% specific would correctly identify 95% of the 800 infected as being positive (760) but also tell 3% of the 9,200 uninfected that they are positive (276). So 276 (27%) out of the total 1036 positive would be misled. So if you got a positive result…

…& thought you must have already been infected but just didn’t notice it (after all, many people are asymptomatic) then you would actually have a 27% (276/1036) chance of never having actually been infected. Put another way, if the test says you are positive for…

…SARS2 antibodies, it’s only 73% likely to be right. That’s know as its “positive predictive value”. So let’s say then that all 1036 people given the “congrats, you’ve recovered from COVID19” signal go out into the world & start mingling, going to work & catching up…

…on those beers & handshakes they have been missing (kisses in Europe). Let’s say that SARS2 then infects half of the 276 people who were given a false-positive result & therefore actually have no immunity to SARS2. That would mean that out of 1036 people…

…who the world thought were immune, 138 (13%) would appear to have gotten reinfected, maybe even weeks or a few months after supposedly recovering. And yet, what if very few of the 800 real positives (true-positives) got reinfected? We would have…

…a hard time knowing who was who. That’s how a Dx test w/ a reported 97% specificity (an A+ by some standards, though my mom would say B+) turns into a tool of confusion. Also notice how much depends on the prevalence of infections in your region.

If the prevalence is really high, let’s say 50%, then testing 10,000 people (of whom 5000 are actually infect, i.e. are true-positives) w/ a 95% sensitive, 97% specific test would identify 4750 true-positives but call 150 (3% of the 5000 who weren’t infected) as false-positives.

So the test would have a 4750/4900 (97%) positive predictive value, which is good. If half of those 150 false-positives suffered their first SARS2 infection but thought it was a reinfection (& none of the true-positives got reinfected), then we would think there was…

…a 1.5% reinfection rate instead of 0%. Big difference from 13%. That’s why it’s critical that we determine how widespread SARS2 infection actually is by testing randomly, not just testing people showing up to hospitals with symptoms. These key studies are underway.

We expect new data every day. You might notice that the lower the prevalence of infection is in a region, the more critical it is that we use tests with high specificity. If prevalence is only 3%, then a test with a 97% specificity (3% false-positive rate), would be only…

…50% right if it told you that you had antibodies to SARS2. I’m not suggesting that we don’t use such tests. What I’m trying to do is vaccinate the public against just believing reports of reinfections of people who supposedly already recovered from COVID and supposedly…

…have antibodies against SARS2. We’ll need more information to know if it’s true. For example, be sure to ask “was the initial infection confirmed by a PCR molecular test that actually detected presence of the SARS2 virus?” If yes, then ok, maybe we really do have…

…a case of a reinfection. But if person doesn’t remember being infected or just recalls mild symptoms & never actually got a molecular PCR test (or got one and it was negative), then maybe they never had the infection in the first place. So “re-infection” could…

…just be their first infection. Stay skeptical! Don’t spread reports about “reinfections” unless you have enough information to know it’s not due to this or other misunderstandings, as we saw before w/ molecular PCR
tests. https://twitter.com/peterkolchinsky/status/1244029896453754880?s=21

But there’s also something I must caution you about. While recovering from COVID19 will “most likely” allow your immune system to generate antibodies & make you immune to reinfection, at least for a while (eg many months or a year), not everyone…

…will necessarily become immune. It’s possible that some people will recover from a very mild infection with just enough antibodies to make a test turn positive but not enough to prevent a re-infection. It’s probably not going to be common. But until we have…

…done studies to quantify how many antibodies people have & how those levels correlate w/ protection against re-infection, we can’t be sure that everyone who is simply positive for some antibodies is really immune. Immunity requires having enough antibody-dogs!

But even those who are immune enough not to experience the symptoms of a re-infection might still have low enough immunity that the virus could infect & replicate in their noses and throats, spreading to others, but high enough immunity to keep the virus…

…from spreading to the lungs, keeping the infection mild, even asymptomatic. So if you live or work with older people, it still makes sense, after you have recovered & get a positive serology/antibody test that more or less proves it, for you to act as before.

Practice social distancing whenever possible & wear a mask or nose/mouth cover when around vulnerable people in case you might be asymptomatically carrying the virus. Once we have vaccines, they will help boost antibody levels higher, even in people who have…

…previously been infected. And we’ll have a sense of how high antibody levels need to be to protect against infection. Currently, the serology tests that are coming to the market don’t tell you how much antibody you have, just if they are above a certain threshold.

Tests that can quantify antibodies are being used in studies to determine how much antibody is high enough to be protective against re-infection. We’ll then use those results to tune antibody tests that say not just if you have antibodies, but if you have enough to be protected.