Read on Twitter
An epidemiologist's primer to DIAGNOSTIC TESTING
- sensitivity & specificity
- positive and negative predictive values
- likelihood ratios
- pre- and post-test probabilities
Sound familiar?
Sound really confusing?
In this
I'll try to tackle how these terms fit together.
1/
- sensitivity & specificity
- positive and negative predictive values
- likelihood ratios
- pre- and post-test probabilities
Sound familiar?
Sound really confusing?
In this
I'll try to tackle how these terms fit together.1/
AGREEMENT ON TERMS AND ABBREVIATIONS
- sensitivity (SENS)
- specificity (SPEC)
- positive predictive value (PPV)
- negative predictive value (NPV)
- likelihood ratio positive (LRP)
- likelihood ratio negative (LRN)
2/
- sensitivity (SENS)
- specificity (SPEC)
- positive predictive value (PPV)
- negative predictive value (NPV)
- likelihood ratio positive (LRP)
- likelihood ratio negative (LRN)
2/
SENS & SPEC
- when we speak of COVID-19 viral tests like PCR and antigen tests, we want them to be ACCURATE
- SENS and SPEC are the accuracy characteristics of your test
- to measure these, we need to know whether people actually have the outcome of interest (disease)
3/
- when we speak of COVID-19 viral tests like PCR and antigen tests, we want them to be ACCURATE
- SENS and SPEC are the accuracy characteristics of your test
- to measure these, we need to know whether people actually have the outcome of interest (disease)
3/
SENS & SPEC (cont'd)
- SENS reflects how good the test does in returning a POSITIVE result in people who actually have the "disease"
- SPEC reflects how good the test does in returning a NEGATIVE result in people who actually DO NOT have the "disease"
4/
- SENS reflects how good the test does in returning a POSITIVE result in people who actually have the "disease"
- SPEC reflects how good the test does in returning a NEGATIVE result in people who actually DO NOT have the "disease"
4/
SENS & SPEC (cont'd)
- a SENS = 90% means that of every 100 people WITH the "disease", the test reads + for 90 (it gets 10 wrong)
- a SPEC = 95% means that of every 100 people WITHOUT the "disease", the test reads - for 95 (it gets 5 wrong)
5/
- a SENS = 90% means that of every 100 people WITH the "disease", the test reads + for 90 (it gets 10 wrong)
- a SPEC = 95% means that of every 100 people WITHOUT the "disease", the test reads - for 95 (it gets 5 wrong)
5/
For SARS-CoV-2 viral tests, you might see metrics like:
PCR
- 98% sensitivity
- 99.9% specificity
Point-of-care antigen (best-case)
- 96% sensitivity
- 97% specificity
These metrics are used to say that PCR is a "more accurate" test than currently available antigen tests.
6/
PCR
- 98% sensitivity
- 99.9% specificity
Point-of-care antigen (best-case)
- 96% sensitivity
- 97% specificity
These metrics are used to say that PCR is a "more accurate" test than currently available antigen tests.
6/
DIAGNOSTIC ABILITY IN THE "REAL WORLD"
- What is most valuable to clinicians/public health is how a test with a certain SENS and SPEC will perform when administered to people whose true "disease" status is unknown.
7/
- What is most valuable to clinicians/public health is how a test with a certain SENS and SPEC will perform when administered to people whose true "disease" status is unknown.
7/
YOU BE THE DOCTOR
- a patient comes in from the community
- prevalence of SARS-CoV-2 is 5% in the community
- seems like an "average person"
Before giving the test, you might guess their chance of having SARS-CoV-2 is 5%
- this is the PRE-TEST PROBABILITY
8/
- a patient comes in from the community
- prevalence of SARS-CoV-2 is 5% in the community
- seems like an "average person"
Before giving the test, you might guess their chance of having SARS-CoV-2 is 5%
- this is the PRE-TEST PROBABILITY
8/
HOW DOES THE TEST HELP?
- the test should help you revise the likelihood that the patient has SARS-CoV-2
- this is the POST-TEST PROBABILITY (PTP)
For a test with 100% SENS & SPEC (perfect accuracy)
- a POS result would
PTP to 100%
- a NEG result would
PTP to 0%
9/
- the test should help you revise the likelihood that the patient has SARS-CoV-2
- this is the POST-TEST PROBABILITY (PTP)
For a test with 100% SENS & SPEC (perfect accuracy)
- a POS result would
PTP to 100%- a NEG result would
PTP to 0%9/
NOMOGRAM
- using a schematic to illustrate what a test should do
- let's say patient has 50% chance of disease BEFORE the test (coin flip)
- you have a test with 90% SENS & 90% SPEC
- a POS result chance of disease to 90%
- a NEG result chance of disease to 10%
10/
- using a schematic to illustrate what a test should do
- let's say patient has 50% chance of disease BEFORE the test (coin flip)
- you have a test with 90% SENS & 90% SPEC
- a POS result
chance of disease to 90%- a NEG result
chance of disease to 10%10/
NOMOGRAM (cont'd)
- a better test would give you more confidence
- Increase SENS & SPEC from 90% to 99% each
- a POS result chance of disease to 99%
- a NEG result chance of disease to 1%
Not perfect confidence, but MUCH better because test is more accurate.
11/
- a better test would give you more confidence
- Increase SENS & SPEC from 90% to 99% each
- a POS result
chance of disease to 99%- a NEG result
chance of disease to 1%Not perfect confidence, but MUCH better because test is more accurate.
11/
IMPACT OF PREVALENCE (PRE-TEST PROBABILITY) ON TEST PERFORMANCE
- take the same scenario from above (99% SENS & SPEC), but instead of a 50% pre-test probability, it's 5%
- a POS result chance of disease to 84%
- a NEG result chance of disease to 1 in 2000
12/
- take the same scenario from above (99% SENS & SPEC), but instead of a 50% pre-test probability, it's 5%
- a POS result
chance of disease to 84%- a NEG result
chance of disease to 1 in 200012/
Pre-test prob is also based on patient factors, but let's assume it's an avg person (so we use prevalence)
- Higher prev improves how much a + test result helps (FALSE +)
- In the prev example, the low prev (5%) meant that 16% of people with + test were actually NEG!
13/
- Higher prev improves how much a + test result helps (
FALSE +)- In the prev example, the low prev (5%) meant that 16% of people with + test were actually NEG!
13/
WHAT IS THE POSITIVE PREDICTIVE VALUE?
- PPV is the % of people with a + test result who actually have the disease
- PPV is the same as the post-test probability after a + test result
- The false POS rate is (100% - PPV)
- Lower prevalence -> PPV and false + rates
14/
- PPV is the % of people with a + test result who actually have the disease
- PPV is the same as the post-test probability after a + test result
- The false POS rate is (100% - PPV)
- Lower prevalence ->
PPV and false + rates14/
WHAT IS THE NEGATIVE PREDICTIVE VALUE?
- NPV is the % of people with a - test result who actually do not have the disease
- NPV is the same as 100% minus the post-test probability of disease after a - test result
- The false NEG rate is (100% - NPV)
15/
- NPV is the % of people with a - test result who actually do not have the disease
- NPV is the same as 100% minus the post-test probability of disease after a - test result
- The false NEG rate is (100% - NPV)
15/
WHAT IS LIKELIHOOD RATIO POSITIVE (LRP)?
- You have a test with SENS=90% & SPEC=97%
- Patient's pre-test probability of disease = 5%
- The LRP=30, which means a + test result the odds the patient actually has the disease by a factor of 30!
- The higher the better!
16/
- You have a test with SENS=90% & SPEC=97%
- Patient's pre-test probability of disease = 5%
- The LRP=30, which means a + test result
the odds the patient actually has the disease by a factor of 30!- The higher the better!
16/
WHAT IS LIKELIHOOD RATIO NEGATIVE (LRN)?
- Same ex as
Prob of disease
- Pre-test: 5%
- Post-test: 0.5%
- The LRN=0.1, which means a NEG test resultthe odds the patient actually has the disease by a factor of 10 (1/0.1)!
- The lower (closer to 0) the better!
16/
- Same ex as
Prob of disease
- Pre-test: 5%
- Post-test: 0.5%
- The LRN=0.1, which means a NEG test result
the odds the patient actually has the disease by a factor of 10 (1/0.1)!- The lower (closer to 0) the better!
16/
WHY DO WE HEAR THAT SPECIFICITY "RULES IN"?
- SPEC means fewer false +
- for the same prevalence, SPEC results in higher PPV
- so, a + result gives strong evidence the patient has the disease (rules in)
SPEC more important for minimizing FALSE POSITIVES
17/
SPECIFICITY "RULES IN"?-
SPEC means fewer false +- for the same prevalence,
SPEC results in higher PPV- so, a + result gives strong evidence the patient has the disease (rules in)
SPEC more important for minimizing FALSE POSITIVES17/
WHY DO WE HEAR THAT SENSITIVITY "RULES OUT"?
- SENS means fewer false -
- for the same prevalence, SENS results in higher NPV
- so, a - result gives strong evidence the patient does not have the disease (rules out)
SENS more important for minimizing FALSE NEG
18/
SENSITIVITY "RULES OUT"?-
SENS means fewer false -- for the same prevalence,
SENS results in higher NPV- so, a - result gives strong evidence the patient does not have the disease (rules out)
SENS more important for minimizing FALSE NEG18/
EXAMPLE: DIFFERENCE IN PCR AND ANTIGEN (1)
Assume:
- PCR: 98% SENS, 99.9% SPEC
Prevalence = 5%
Administer to 100,000 people
4,995 will test + (95 will be wrong, a false +)
- PPV 98%
95,005 will test - (100 will be wrong, a false -)
- NPV 99.9%
19/
Assume:
- PCR: 98% SENS, 99.9% SPEC
Prevalence = 5%
Administer to 100,000 people
4,995 will test + (95 will be wrong, a false +)
- PPV 98%
95,005 will test - (100 will be wrong, a false -)
- NPV 99.9%
19/
EXAMPLE: DIFFERENCE IN PCR AND ANTIGEN (2)
Assume:
- Antigen: 96% SENS, 97% SPEC
Prevalence = 5%
Administer to 100,000 people
7,650 will test + (2,850 will be wrong, a false +)
- PPV 63%
92,350 will test - (200 will be wrong, a false -)
- NPV 99.8%
See the diff
20/
Assume:
- Antigen: 96% SENS, 97% SPEC
Prevalence = 5%
Administer to 100,000 people
7,650 will test + (2,850 will be wrong, a false +)
- PPV 63%
92,350 will test - (200 will be wrong, a false -)
- NPV 99.8%
See the diff
20/
AGAIN: PREVALENCE MAKES A DIFFERENCE
At 5% prev:
- PCR: 98% PPV, 99.9% NPV
- Antigen: 63% PPV, 99.8% NPV
Increase true prev to 15%:
- PCR: 99.4% PPV, 99.6% NPV
- Antigen: 85% PPV, 99.3% NPV
This is why, for many tests, we use in a "high-risk" group. prev -> PPV
21/
At 5% prev:
- PCR: 98% PPV, 99.9% NPV
- Antigen: 63% PPV, 99.8% NPV
Increase true prev to 15%:
- PCR: 99.4% PPV, 99.6% NPV
- Antigen: 85% PPV, 99.3% NPV
This is why, for many tests, we use in a "high-risk" group.
prev -> PPV21/
Often, although we want the highest possible SENS & SPEC, there may be a trade-off.
Do you want to minimize false + or false - MORE?
Depends on situation. Think about HIV tests.
- the stress of a false +
- the ramifications of a false -
22/
Do you want to minimize false + or false - MORE?
Depends on situation. Think about HIV tests.
- the stress of a false +
- the ramifications of a false -
22/
Just scratches the surface, but covers key terms people have been asking me about.
TL;DR
- Tests with SPEC better to FALSE POSITIVES
- Tests with SENS better to FALSE NEGATIVES
- The % of false + can get when prevalence is , even for test w/ great accuracy
23/
TL;DR
- Tests with
SPEC better to FALSE POSITIVES- Tests with
SENS better to FALSE NEGATIVES- The % of false + can get
when prevalence is , even for test w/ great accuracy23/
Those who MAY care to share the diagnostic testing primer
@nataliexdean @EpiEllie @zorinaq @_stah @aetiology @meganranney @EricTopol @berthahidalgo @jessicamalaty @bethlinas @jaimiegradus @michaelmina_lab @DrTomFrieden @angie_rasmussen
24/
@nataliexdean @EpiEllie @zorinaq @_stah @aetiology @meganranney @EricTopol @berthahidalgo @jessicamalaty @bethlinas @jaimiegradus @michaelmina_lab @DrTomFrieden @angie_rasmussen
24/
