Read on Twitter
1. Models have their strength and weaknesses. It's valuable to understand both. While the @IHME_UW model has certain advantages over other approaches, I want to focus here on a disadvantage, namely the absence of an underlying mechanistic / bottom up / process-based framework.
2. The @IHME_UW makes its projections based on observed deaths. This is sensible in one respect. Because of massive differences in testing availability and approach, deaths are probably more reliable and probably less heterogeneous across states than observed infections.
3. But you also get some implausible results. Today's run of the IHME model projects that the peak in Washington state occurred 11 days ago.
4. Below, I (loosely, by hand) aligned the @IHME_UW projection for Washington State with the data on new cases in Washington state from https://www.nytimes.com/interactive/2020/us/washington-coronavirus-cases.html.
The blue line indicates the timing of the supposed Washington state peak.
The blue line indicates the timing of the supposed Washington state peak.
5. This puts the Washington state peak (for deaths) on March 27th, at which point there were 3768 known cases.
Between then two days, there have been an additional 4616 cases diagnosed in WA.
(Data from https://wsha.org/for-patients/coronavirus/coronavirus-tracker/)
Between then two days, there have been an additional 4616 cases diagnosed in WA.
(Data from https://wsha.org/for-patients/coronavirus/coronavirus-tracker/)
6. Remember that daily deaths is a lagging indicator: we expect deaths to peak substantially *after* positive tests do, because of the long period (very approximately, two weeks) between symptoms and death.
7. Even ignoring that lag, for the IHME projection to be correct it would require that that the cases in blue (diagnosed before March 27) contribute to death peak which should decline as the cases in yellow (diagnosed after the March 27) move through the healthcare system.
8. Of course, diagnosed case is an imperfect measur. We could see something like the IHME model if e.g. testing ramped up substantially over the time period indicated. I don't have WA testing data, but I do have King County. https://www.kingcounty.gov/depts/health/communicable-diseases/disease-control/novel-coronavirus/data-dashboard.aspxhave. I don't see a big ramp-up.
9. Meanwhile data on WA state deaths continue to come in. Here are the data from today's @NYTimes dashboard. I've indicated the @IHME_UW projected peak with a blue arrow.
10. I mentioned at the start of the thread that models have strengths and weaknesses, and said I was going to focus on weaknesses.
What I want to highlight here is not that the @IHME_UW model is necessarily far off (at least for the case that we manage to control the virus).
What I want to highlight here is not that the @IHME_UW model is necessarily far off (at least for the case that we manage to control the virus).
11. The key thing I want to focus on is that estimate of the peak timing in the model is extremely sensitive to small perturbations, and thus in my opinion unreliable.
Below is a loose hand-alignment of the IHME projections on 3/6 with those from one day earlier, on 3/5.
Below is a loose hand-alignment of the IHME projections on 3/6 with those from one day earlier, on 3/5.
12. You'll see that basically nothing changes. We get one more day of data, and the uncertainty ranges do not change much going forward.
But the estimate of the timing of the peak leaps backward by 10 days—in what almost certainly is the wrong direction.
But the estimate of the timing of the peak leaps backward by 10 days—in what almost certainly is the wrong direction.
13. This highlights the importance of understanding a model's purpose, how it is constructed, and how it performs. It also shows the risk of relying on point estimates. (One virtue of the IHME model is that it gives ranges too for other stats.)
14. This thread is not intended to undermine the entirity of the @IHME_UW model. My aim is just to illustrate that what you put in (deaths vs. other sources of info) matters, and that before relying on estimates it's worth understanding their sensitivity to small perturbations
