Read on Twitter
2 weeks ago, Twitter predicted today’s TX COVID hospitalizations far more accurately than biased “experts.”
How? Bayesian logic + data + science from *unbiased* experts, esp Dr. Gabriela Gomes. @mgmgomes1
THREAD: what we should learn. Please share widely. 1/
#rationalground
How? Bayesian logic + data + science from *unbiased* experts, esp Dr. Gabriela Gomes. @mgmgomes1
THREAD: what we should learn. Please share widely. 1/
#rationalground
2/ In "Superforecasting," Philip Tetlock observes experts’ predictions are frequently disastrous - yet the media trots ‘em out again for more "expertise."
His research finds forecasts by thoughtful, data-driven generalists often outperform experts. http://www.askeladdencapital.com/philip-tetlocks-superforecasting-book-review-notes-analysis/
His research finds forecasts by thoughtful, data-driven generalists often outperform experts. http://www.askeladdencapital.com/philip-tetlocks-superforecasting-book-review-notes-analysis/
3/ The COVID pandemic? No exception.
“expert” P. Hotez predicted in late June that only an aggressive full lockdown could curb TX COVID spread.
In fact, with no lockdown, “R” had already peaked. Hospitalizations soon peaked and rapidly declined.
“expert” P. Hotez predicted in late June that only an aggressive full lockdown could curb TX COVID spread.
In fact, with no lockdown, “R” had already peaked. Hospitalizations soon peaked and rapidly declined.
4/ Even as hospitalizations declined, several other “experts” painted a grim picture for Texas, with the standard “wait two weeks” warning.
They claimed Texas was “in trouble” and needed to “act now” to “turn things around” - or else!
They claimed Texas was “in trouble” and needed to “act now” to “turn things around” - or else!
5/ We waited two weeks. On 8/11, Texas COVID hosps were ~7,200. 283 contestants made predictions for 8/25.
Choices {vote share}:
- flat/up {6%, + biased “experts”}
- ↓ 0-10% {4%}
- ↓10-20% {22%}
- ↓down >20%. {69%, + unbiased experts} https://twitter.com/AskeladdenTX/status/1293400485169422337
Choices {vote share}:
- flat/up {6%, + biased “experts”}
- ↓ 0-10% {4%}
- ↓10-20% {22%}
- ↓down >20%. {69%, + unbiased experts} https://twitter.com/AskeladdenTX/status/1293400485169422337
6/ Outcome: TX COVID hospitalizations declined 32% from 8/11 to 8/25, reaching late-June levels.
Every major metro saw a sharp decrease.
For good measure, the same happened in Georgia, Florida, and Arizona.
Every major metro saw a sharp decrease.
For good measure, the same happened in Georgia, Florida, and Arizona.
7/ All the “experts” cited by major media like the NYT predicted something very different.
Osterholm: “I don’t see this slowing down.”
Jha: “Not fading out - this will be with us for at least another 12 months.”
Osterholm: “I don’t see this slowing down.”
Jha: “Not fading out - this will be with us for at least another 12 months.”
8/ Unbiased experts took a different view.
Dr. Gabriela Gomes ( @mgmgomes1) has spent decades studying real-world transmission dynamics of diseases like COVID - and she believed herd immunity was <20%.
Dr. Gabriela Gomes ( @mgmgomes1) has spent decades studying real-world transmission dynamics of diseases like COVID - and she believed herd immunity was <20%.
9/ This is based on a concept called “heterogeneity” that is self-obviously true: we are not all identical dots randomly bouncing around in a box.
We’re each differentially connected and susceptible.
Yet biased “experts” censored and downplayed Dr. Gomes’s work.
We’re each differentially connected and susceptible.
Yet biased “experts” censored and downplayed Dr. Gomes’s work.
10/ What did Dr. Gomes do that many other experts did not?
*Update her beliefs in response to new data.*
This turns out to be key, key, key.
*Update her beliefs in response to new data.*
This turns out to be key, key, key.
11/ In her team’s initial preprint, they did not have enough data to estimate the herd immunity threshold for COVID-19 - so they simply presented different scenarios based on behavior of other diseases like TB. https://www.medrxiv.org/content/10.1101/2020.04.27.20081893v3
12/ However, reopening provided her with *new data* - if homogeneous models were correct, infections would rise exponentially even in hard-hit areas.
In reality, they only rose in areas without significant immunity.
In reality, they only rose in areas without significant immunity.
13/ So in her next paper, Dr. Gomes used the new data to make country-by-country estimates for the COVID herd immunity threshold and concluded HIT was likely <20% in most places. https://www.medrxiv.org/content/10.1101/2020.07.23.20160762v1
14/ Rather than accepting the new contribution to science, this paper was censored by journals and rejected/minimized by biased “experts.”
Not because the math was wrong, but because it *didn’t match their existing beliefs.* https://twitter.com/mgmgomes1/status/1291162358962937857
Not because the math was wrong, but because it *didn’t match their existing beliefs.* https://twitter.com/mgmgomes1/status/1291162358962937857
15/ In other words, most of the people claiming to “follow the science” and “look at the data” are *actively ignoring* BOTH the science and the data.
Mathematician Wes Pegden observed pithily: https://twitter.com/WesPegden/status/1297720206618501121
Mathematician Wes Pegden observed pithily: https://twitter.com/WesPegden/status/1297720206618501121
16/ Returning to Tetlock, let’s understand a concept called “Bayesian reasoning.”
Summarily: updating existing beliefs (“priors”) in response to new data, in a probabilistic fashion. http://www.askeladdencapital.com/priors-bayesian-reasoning-conditional-probabilities-mental-model-be-a-filter-not-a-sponge/
Summarily: updating existing beliefs (“priors”) in response to new data, in a probabilistic fashion. http://www.askeladdencapital.com/priors-bayesian-reasoning-conditional-probabilities-mental-model-be-a-filter-not-a-sponge/
17/ Distinction: priors aren’t facts.
They’re beliefs.
COVID hospitalizations from March until now are facts.
COVID hospitalizations from now until next March obviously aren’t facts yet - they’re beliefs.
They’re beliefs.
COVID hospitalizations from March until now are facts.
COVID hospitalizations from now until next March obviously aren’t facts yet - they’re beliefs.
18/ Unfortunately, media and many ”experts” confused priors (“beliefs”) and facts.
Example: we obviously could not know *for a fact* if herd immunity for COVID is 20% or 60% until we got there.
But 60% was treated like a “fact” - when it was really a belief/prior.
Example: we obviously could not know *for a fact* if herd immunity for COVID is 20% or 60% until we got there.
But 60% was treated like a “fact” - when it was really a belief/prior.
19/ Reasonable scientists and analysts have updated priors based on new data.
For example, @natesilver538 and @trvrb - generally COVID pessimists - have acknowledged immunity is slowing pandemic spread. https://www.technologyreview.com/2020/08/11/1006366/immunity-slowing-down-coronavirus-parts-us/
For example, @natesilver538 and @trvrb - generally COVID pessimists - have acknowledged immunity is slowing pandemic spread. https://www.technologyreview.com/2020/08/11/1006366/immunity-slowing-down-coronavirus-parts-us/
20/ As discussed in @accad_koka Gomes podcast, the “heterogeneity” 20% herd immunity model fits real-world data better than wildly unrealistic “homogeneous” 60% model.
Yet agenda-driven hacks like Bergstrom, Marm Kilpatrick ignore data, maintain priors. https://twitter.com/AskeladdenTX/status/1295786867347140614
Yet agenda-driven hacks like Bergstrom, Marm Kilpatrick ignore data, maintain priors. https://twitter.com/AskeladdenTX/status/1295786867347140614
21/ See this in action: Bergstrom admits that he’s scared of playground equipment despite *no evidence.* “Experts” follow their *feelings* over data. Data in fact clearly shows minimal outdoor transmission, sunlight rapidly inactivates virus on surfaces. https://academic.oup.com/jid/article/222/2/214/5841129
22/ It’s ok to be wrong; not ok to stay wrong.
MIT data scientist @youyanggu started building his own models in spring after failure of conventional (Ferguson) models.
He adjusted these based on new data, such as reducing IFR over time. He’s consistently ranked most accurate.
MIT data scientist @youyanggu started building his own models in spring after failure of conventional (Ferguson) models.
He adjusted these based on new data, such as reducing IFR over time. He’s consistently ranked most accurate.
23/ What did his analysis show? Interventions such as mask mandates or bar closures did not primarily drive reduction in spread. https://twitter.com/youyanggu/status/1292898685173534722
24/ What did clearly drive reduction in spread: acquired specific immunity, i.e. 15-20% of the population becoming infected with COVID-19, gaining immunity, and thereafter acting as roadblocks to future spread.
25/ Idaho is a particularly elegant example: Blaine County, where >20% of the population had antibodies after a spring spike, saw no material spread in summer. The rest of Idaho, with lower immunity, did. https://twitter.com/youyanggu/status/1294324998677639170
26/ We see the same in many other places like Manaus, Brazil, where despite no lockdown and poor compliance with social-distancing guidelines, the epidemic peaked and then declined at 20% or lower seroprevalence.
27/ Conversely, the same interventions (ex. masks) posited to reduce spread in places like Texas, failed to get R<1 in other places at other times, like Miami, Hawaii, and L.A. - suggesting they can’t be the sole variable. https://twitter.com/ianmSC/status/1296903578045407232
28/ Why does any of this matter? Too many “expert” / politician “priors” are stuck in March - the now-disproven ideas that:
A) COVID will spread exponentially if unchecked
B) Highly deadly to everyone
A) COVID will spread exponentially if unchecked
B) Highly deadly to everyone
29/ Reality: never exponential ad infinitum, very low risk for healthy <40s. Huge risk for >80s. Yet businesses still shut, kids not in school because of priors set in March that have been *proven wrong* between March and August.
30/ The data-driven contestants clearly had priors that predicted real-world outcomes with reasonable accuracy.
The blue-checks foolishly maintained outdated, inaccurate priors, engaging in fact-free hysteria.
Hence: independent number-crunchers beat the pants off experts.
The blue-checks foolishly maintained outdated, inaccurate priors, engaging in fact-free hysteria.
Hence: independent number-crunchers beat the pants off experts.
31/ It’s worth noting this behavior was *selective.* On masks and lockdowns (which the science DID NOT RECOMMEND prior to COVID), these “experts” are suddenly singing a different tune - demonstrating their bias. Fitting the science to their priors, rather than vice versa.
32/ Let’s make one thing clear - this doesn’t mean expertise is irrelevant. Who do you want as your pilot - an Air Force veteran, or a random civilian?
33/ Generalists like me have vastly benefited from expertise of many - biologists’ research on T-cells, models/analysis by mathematicians like @wespegden and @mgmgomes1. They do work that you and I cannot.
34/ But, importantly, their work matches real-world data. Experts *can* be valuable; we should presume they are - but if they keep claiming the emperor has fine new robes when he’s actually butt-naked, then use your eyes.
35/ Pols/media/public health made fatal mistake of deciding “the science” was a conclusive consensus. In reality, even early, there was much dissent.
36/ Nobel prize winner @MLevitt_NP2013 said early COVID was never exponential. Smart doctors/epis like @Sunetragupta, Anders Tegnell, Karl Friston quickly observed COVID data was not matching accepted view.
37/ Yet these EXPERTS were ignored, derided, or marginalized. Friston was ridiculed for his prescient “immunological dark matter” comments, which foreshadowed now quite robust data on high prevalence of cross-reactive T-cells.
https://www.researchsquare.com/article/rs-35331/v1
https://www.researchsquare.com/article/rs-35331/v1
38/ Again, scientific journals attempted to censor Dr. Gomes’s paper suggesting COVID herd immunity threshold <20%, because it wasn’t sufficiently alarmist and didn’t agree with previous 60% estimate. https://twitter.com/AskeladdenTX/status/1291171763401916417
39/ Do you see the problem here? Circular reasoning.
“Science says X” - if you ignore or censor/suppress all the science that says “not X.”
So of course, “the subset of science that says X” says X. Duh.
“Science says X” - if you ignore or censor/suppress all the science that says “not X.”
So of course, “the subset of science that says X” says X. Duh.
40/ In the interest of transparency, several of my priors have been proven wrong and needed updating. Two big examples.
41/ First, I believed summer would suppress transmission of COVID-19, because most existing coronaviruses are *highly* seasonal, disappearing in the summer.
I was wrong. https://academic.oup.com/jid/article/222/1/9/5815743?guestAccessKey=b19eb499-007a-4ebf-a4a8-bb6a7481ec0c
I was wrong. https://academic.oup.com/jid/article/222/1/9/5815743?guestAccessKey=b19eb499-007a-4ebf-a4a8-bb6a7481ec0c
42/ Second, I thought that the rise in cases/hospitalizations in the Sun Belt in June would not translate to deaths, because deaths weren’t showing up.
I was wrong - I underestimated the reporting lag (deaths reported way after the fact).
I was wrong - I underestimated the reporting lag (deaths reported way after the fact).
43/ Quickly acknowledging both of those allowed me to reshape my beliefs and make more accurate predictions since that time. Undoubtedly, some of my current priors on COVID will need adjusting, based on data that will emerge over time.
44/ What do we learn?
First, experts who *don’t* update beliefs based on data are likely to underperform *non-experts* who *do* update beliefs based on data.
Experts aren’t infallible. Even Einstein got things wrong!
First, experts who *don’t* update beliefs based on data are likely to underperform *non-experts* who *do* update beliefs based on data.
Experts aren’t infallible. Even Einstein got things wrong!
45/ Second, “SCIENCE” isn’t a monolith, whatever media/politicians may say.
Scientists disagree all the time!
Unfortunately, creating an echo chamber and censoring dissenting views hinders scientific advancement, rather than vice versa.
Scientists disagree all the time!
Unfortunately, creating an echo chamber and censoring dissenting views hinders scientific advancement, rather than vice versa.
46/ Third/finally, the Gomes heterogeneity model provides our most accurate understanding of COVID behavior
Yet she is being treated as a curiosity, rather than viewed as *the* leading expert, like she deserves to be.
Please follow her ( @mgmgomes1) and share her work widely.
Yet she is being treated as a curiosity, rather than viewed as *the* leading expert, like she deserves to be.
Please follow her ( @mgmgomes1) and share her work widely.
