Today I'm going to be live tweeting:
"Predictive performance of international COVID-19
mortality forecasting models"
(Friedman et al. August 26, 2020)

What looks like a great evaluation of covid model performance over long periods. https://www.medrxiv.org/content/10.1101/2020.07.13.20151233v4

I picked this bc one of the problems we found in our review was that modelers were constantly tweaking their models to overfit recent data making it impossible to tell if they were going to generalize to future data. Need the whole history of predictions. https://twitter.com/RexDouglass/status/1295391498477826053

There are a shocking >383 covid forecasting models now. But despite that only 7 of them provide sufficient coverage and histories to make judgements about how they'll perform over the long haul/how good their teams are.

The left hand side is count of confirmed deaths at the country level (country-day) and NYT for U.S. states (state-day)

Because this is an evaluation project it has to think really hard about things that most forecasts/papers just gloss over entirely. Getting into the weeds on that is a big deal.

For instance because day to day reporting fluctuates a lot from reporting mechanisms alone, the focus on cumulative and weekly total mortality. They also exclude excess mortality which has its own modeling issues.