Read on Twitter
An epidemiologist& #39;s primer to DIAGNOSTIC TESTING
- sensitivity & specificity
- positive and negative predictive values
- likelihood ratios
- pre- and post-test probabilities
Sound familiar?
Sound really confusing?
In this
https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread">I& #39;ll try to tackle how these terms fit together.
1/
- sensitivity & specificity
- positive and negative predictive values
- likelihood ratios
- pre- and post-test probabilities
Sound familiar?
Sound really confusing?
In this
1/
AGREEMENT ON TERMS AND ABBREVIATIONS
- sensitivity (SENS)
- specificity (SPEC)
- positive predictive value (PPV)
- negative predictive value (NPV)
- likelihood ratio positive (LRP)
- likelihood ratio negative (LRN)
2/
- sensitivity (SENS)
- specificity (SPEC)
- positive predictive value (PPV)
- negative predictive value (NPV)
- likelihood ratio positive (LRP)
- likelihood ratio negative (LRN)
2/
SENS & SPEC
- when we speak of COVID-19 viral tests like PCR and antigen tests, we want them to be ACCURATEhttps://abs.twimg.com/emoji/v2/... draggable="false" alt="🎯" title="Volltreffer" aria-label="Emoji: Volltreffer">
- SENS and SPEC are the accuracy characteristics of your test
- to measure these, we need to know whether people actually have the outcome of interest (disease)
3/
- when we speak of COVID-19 viral tests like PCR and antigen tests, we want them to be ACCURATE
- SENS and SPEC are the accuracy characteristics of your test
- to measure these, we need to know whether people actually have the outcome of interest (disease)
3/
SENS & SPEC (cont& #39;d)
- SENS reflects how good the test does in returning a POSITIVE result in people who actually have the "disease"
- SPEC reflects how good the test does in returning a NEGATIVE result in people who actually DO NOT have the "disease"
4/
- SENS reflects how good the test does in returning a POSITIVE result in people who actually have the "disease"
- SPEC reflects how good the test does in returning a NEGATIVE result in people who actually DO NOT have the "disease"
4/
SENS & SPEC (cont& #39;d)
- a SENS = 90% means that of every 100 people WITH the "disease", the test reads + for 90 (it gets 10 wrong)
- a SPEC = 95% means that of every 100 people WITHOUT the "disease", the test reads - for 95 (it gets 5 wrong)
5/
- a SENS = 90% means that of every 100 people WITH the "disease", the test reads + for 90 (it gets 10 wrong)
- a SPEC = 95% means that of every 100 people WITHOUT the "disease", the test reads - for 95 (it gets 5 wrong)
5/
For SARS-CoV-2 viral tests, you might see metrics like:
PCR
- 98% sensitivity
- 99.9% specificity
Point-of-care antigen (best-case)
- 96% sensitivity
- 97% specificity
These metrics are used to say that PCR is a "more accurate" test than currently available antigen tests.
6/
PCR
- 98% sensitivity
- 99.9% specificity
Point-of-care antigen (best-case)
- 96% sensitivity
- 97% specificity
These metrics are used to say that PCR is a "more accurate" test than currently available antigen tests.
6/
DIAGNOSTIC ABILITY IN THE "REAL WORLD"
- What is most valuable to clinicians/public health is how a test with a certain SENS and SPEC will perform when administered to people whose true "disease" status is unknown.
7/
- What is most valuable to clinicians/public health is how a test with a certain SENS and SPEC will perform when administered to people whose true "disease" status is unknown.
7/
YOU BE THE DOCTOR
- a patient comes in from the community
- prevalence of SARS-CoV-2 is 5% in the community
- seems like an "average person"
Before giving the test, you might guess their chance of having SARS-CoV-2 is 5%
- this is the PRE-TEST PROBABILITY
8/
- a patient comes in from the community
- prevalence of SARS-CoV-2 is 5% in the community
- seems like an "average person"
Before giving the test, you might guess their chance of having SARS-CoV-2 is 5%
- this is the PRE-TEST PROBABILITY
8/
HOW DOES THE TEST HELP?
- the test should help you revise the likelihood that the patient has SARS-CoV-2
- this is the POST-TEST PROBABILITY (PTP)
For a test with 100% SENS & SPEC (perfect accuracy)
- a POS result wouldhttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> PTP to 100%
- a NEG result wouldhttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> PTP to 0%
9/
- the test should help you revise the likelihood that the patient has SARS-CoV-2
- this is the POST-TEST PROBABILITY (PTP)
For a test with 100% SENS & SPEC (perfect accuracy)
- a POS result would
- a NEG result would
9/
NOMOGRAM
- using a schematic to illustrate what a test should do
- let& #39;s say patient has 50% chance of disease BEFORE the test (coin flip)
- you have a test with 90% SENS & 90% SPEC
- a POS resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> chance of disease to 90%
- a NEG resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 10%
10/
- using a schematic to illustrate what a test should do
- let& #39;s say patient has 50% chance of disease BEFORE the test (coin flip)
- you have a test with 90% SENS & 90% SPEC
- a POS result
- a NEG result
10/
chance of disease to 90%- a NEG result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 10%10/" title="NOMOGRAM- using a schematic to illustrate what a test should do- let& #39;s say patient has 50% chance of disease BEFORE the test (coin flip) - you have a test with 90% SENS & 90% SPEC - a POS result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> chance of disease to 90%- a NEG result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 10%10/" class="img-responsive" style="max-width:100%;"/>
NOMOGRAM (cont& #39;d)
- a better test would give you more confidence
- Increase SENS & SPEC from 90% to 99% each
- a POS resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> chance of disease to 99%
- a NEG resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 1%
Not perfect confidence, but MUCH better because test is more accurate.
11/
- a better test would give you more confidence
- Increase SENS & SPEC from 90% to 99% each
- a POS result
- a NEG result
Not perfect confidence, but MUCH better because test is more accurate.
11/
chance of disease to 99%- a NEG result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 1%Not perfect confidence, but MUCH better because test is more accurate.11/" title="NOMOGRAM (cont& #39;d)- a better test would give you more confidence- Increase SENS & SPEC from 90% to 99% each - a POS result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> chance of disease to 99%- a NEG result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 1%Not perfect confidence, but MUCH better because test is more accurate.11/" class="img-responsive" style="max-width:100%;"/>
IMPACT OF PREVALENCE (PRE-TEST PROBABILITY) ON TEST PERFORMANCE
- take the same scenario from above (99% SENS & SPEC), but instead of a 50% pre-test probability, it& #39;s 5%
- a POS resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> chance of disease to 84%
- a NEG resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 1 in 2000
12/
- take the same scenario from above (99% SENS & SPEC), but instead of a 50% pre-test probability, it& #39;s 5%
- a POS result
- a NEG result
12/
chance of disease to 84%- a NEG result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 1 in 200012/" title="IMPACT OF PREVALENCE (PRE-TEST PROBABILITY) ON TEST PERFORMANCE- take the same scenario from above (99% SENS & SPEC), but instead of a 50% pre-test probability, it& #39;s 5%- a POS result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> chance of disease to 84%- a NEG result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> chance of disease to 1 in 200012/" class="img-responsive" style="max-width:100%;"/>
Pre-test prob is also based on patient factors, but let& #39;s assume it& #39;s an avg person (so we use prevalence)
- Higher prev improves how much a + test result helps (https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten">FALSE +)
- In the prev example, the low prev (5%) meant that 16% of people with + test were actually NEG!
13/
- Higher prev improves how much a + test result helps (
- In the prev example, the low prev (5%) meant that 16% of people with + test were actually NEG!
13/
WHAT IS THE POSITIVE PREDICTIVE VALUE?
- PPV is the % of people with a + test result who actually have the disease
- PPV is the same as the post-test probability after a + test result
- The false POS rate is (100% - PPV)
- Lower prevalence ->https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> PPV and https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> false + rates
14/
- PPV is the % of people with a + test result who actually have the disease
- PPV is the same as the post-test probability after a + test result
- The false POS rate is (100% - PPV)
- Lower prevalence ->
14/
WHAT IS THE NEGATIVE PREDICTIVE VALUE?
- NPV is the % of people with a - test result who actually do not have the disease
- NPV is the same as 100% minus the post-test probability of disease after a - test result
- The false NEG rate is (100% - NPV)
15/
- NPV is the % of people with a - test result who actually do not have the disease
- NPV is the same as 100% minus the post-test probability of disease after a - test result
- The false NEG rate is (100% - NPV)
15/
WHAT IS LIKELIHOOD RATIO POSITIVE (LRP)?
- You have a test with SENS=90% & SPEC=97%
- Patient& #39;s pre-test probability of disease = 5%
- The LRP=30, which means a + test resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> the odds the patient actually has the disease by a factor of 30!
- The higher the better!
16/
- You have a test with SENS=90% & SPEC=97%
- Patient& #39;s pre-test probability of disease = 5%
- The LRP=30, which means a + test result
- The higher the better!
16/
the odds the patient actually has the disease by a factor of 30!- The higher the better!16/" title="WHAT IS LIKELIHOOD RATIO POSITIVE (LRP)?- You have a test with SENS=90% & SPEC=97%- Patient& #39;s pre-test probability of disease = 5%- The LRP=30, which means a + test result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> the odds the patient actually has the disease by a factor of 30!- The higher the better!16/" class="img-responsive" style="max-width:100%;"/>
WHAT IS LIKELIHOOD RATIO NEGATIVE (LRN)?
- Same ex ashttps://abs.twimg.com/emoji/v2/... draggable="false" alt="👆" title="Rückhand Zeigefinger nach oben" aria-label="Emoji: Rückhand Zeigefinger nach oben">
Prob of disease
- Pre-test: 5%
- Post-test: 0.5%
- The LRN=0.1, which means a NEG test resulthttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten">the odds the patient actually has the disease by a factor of 10 (1/0.1)!
- The lower (closer to 0) the better!
16/
- Same ex as
Prob of disease
- Pre-test: 5%
- Post-test: 0.5%
- The LRN=0.1, which means a NEG test result
- The lower (closer to 0) the better!
16/
Prob of disease- Pre-test: 5%- Post-test: 0.5%- The LRN=0.1, which means a NEG test result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten">the odds the patient actually has the disease by a factor of 10 (1/0.1)!- The lower (closer to 0) the better!16/" title="WHAT IS LIKELIHOOD RATIO NEGATIVE (LRN)?- Same ex as https://abs.twimg.com/emoji/v2/... draggable="false" alt="👆" title="Rückhand Zeigefinger nach oben" aria-label="Emoji: Rückhand Zeigefinger nach oben">Prob of disease- Pre-test: 5%- Post-test: 0.5%- The LRN=0.1, which means a NEG test result https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten">the odds the patient actually has the disease by a factor of 10 (1/0.1)!- The lower (closer to 0) the better!16/" class="img-responsive" style="max-width:100%;"/>
WHY DO WE HEAR THAT https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> SPECIFICITY "RULES IN"?
-https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> SPEC means fewer false +
- for the same prevalence,https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> SPEC results in higher https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> PPV
- so, a + result gives strong evidence the patient has the disease (rules in)
https://abs.twimg.com/emoji/v2/... draggable="false" alt="📌" title="Reißzwecke" aria-label="Emoji: Reißzwecke">SPEC more important for minimizing FALSE POSITIVES
17/
-
- for the same prevalence,
- so, a + result gives strong evidence the patient has the disease (rules in)
17/
WHY DO WE HEAR THAT https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> SENSITIVITY "RULES OUT"?
-https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> SENS means fewer false -
- for the same prevalence,https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> SENS results in higher https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> NPV
- so, a - result gives strong evidence the patient does not have the disease (rules out)
https://abs.twimg.com/emoji/v2/... draggable="false" alt="📌" title="Reißzwecke" aria-label="Emoji: Reißzwecke">SENS more important for minimizing FALSE NEG
18/
-
- for the same prevalence,
- so, a - result gives strong evidence the patient does not have the disease (rules out)
18/
EXAMPLE: DIFFERENCE IN PCR AND ANTIGEN (1)
Assume:
- PCR: 98% SENS, 99.9% SPEC
Prevalence = 5%
Administer to 100,000 people
4,995 will test + (95 will be wrong, a false +)
- PPV 98%
95,005 will test - (100 will be wrong, a false -)
- NPV 99.9%
19/
Assume:
- PCR: 98% SENS, 99.9% SPEC
Prevalence = 5%
Administer to 100,000 people
4,995 will test + (95 will be wrong, a false +)
- PPV 98%
95,005 will test - (100 will be wrong, a false -)
- NPV 99.9%
19/
EXAMPLE: DIFFERENCE IN PCR AND ANTIGEN (2)
Assume:
- Antigen: 96% SENS, 97% SPEC
Prevalence = 5%
Administer to 100,000 people
7,650 will test + (2,850 will be wrong, a false +)
- PPV 63%
92,350 will test - (200 will be wrong, a false -)
- NPV 99.8%
See the diffhttps://abs.twimg.com/emoji/v2/... draggable="false" alt="👀" title="Augen" aria-label="Emoji: Augen">
20/
Assume:
- Antigen: 96% SENS, 97% SPEC
Prevalence = 5%
Administer to 100,000 people
7,650 will test + (2,850 will be wrong, a false +)
- PPV 63%
92,350 will test - (200 will be wrong, a false -)
- NPV 99.8%
See the diff
20/
AGAIN: PREVALENCE MAKES A DIFFERENCE
At 5% prev:
- PCR: 98% PPV, 99.9% NPV
- Antigen: 63% PPV, 99.8% NPV
Increase true prev to 15%:
- PCR: 99.4% PPV, 99.6% NPV
- Antigen: 85% PPV, 99.3% NPV
This is why, for many tests, we use in a "high-risk" group.https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> prev -> https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> PPV
21/
At 5% prev:
- PCR: 98% PPV, 99.9% NPV
- Antigen: 63% PPV, 99.8% NPV
Increase true prev to 15%:
- PCR: 99.4% PPV, 99.6% NPV
- Antigen: 85% PPV, 99.3% NPV
This is why, for many tests, we use in a "high-risk" group.
21/
Often, although we want the highest possible SENS & SPEC, there may be a trade-off.
Do you want to minimize false + or false - MORE?
Depends on situation. Think about HIV tests.
- the stress of a false +
- the ramifications of a false -
22/
Do you want to minimize false + or false - MORE?
Depends on situation. Think about HIV tests.
- the stress of a false +
- the ramifications of a false -
22/
Just scratches the surface, but covers key terms people have been asking me about.
TL;DR
- Tests withhttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> SPEC better to https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> FALSE POSITIVES
- Tests withhttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> SENS better to https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> FALSE NEGATIVES
- The % of false + can gethttps://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> when prevalence is https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten">, even for test w/ great accuracy
23/
TL;DR
- Tests with
- Tests with
- The % of false + can get
23/
Those who MAY care to share the diagnostic testing primer https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread"> https://abs.twimg.com/emoji/v2/... draggable="false" alt="👆" title="Rückhand Zeigefinger nach oben" aria-label="Emoji: Rückhand Zeigefinger nach oben">
@nataliexdean @EpiEllie @zorinaq @_stah @aetiology @meganranney @EricTopol @berthahidalgo @jessicamalaty @bethlinas @jaimiegradus @michaelmina_lab @DrTomFrieden @angie_rasmussen
24/https://abs.twimg.com/emoji/v2/... draggable="false" alt="🏁" title="Karierte Flagge" aria-label="Emoji: Karierte Flagge">
@nataliexdean @EpiEllie @zorinaq @_stah @aetiology @meganranney @EricTopol @berthahidalgo @jessicamalaty @bethlinas @jaimiegradus @michaelmina_lab @DrTomFrieden @angie_rasmussen
24/
