Read on Twitter
1. This thread about an important paper from @jonassjuul, K. Græsbøll, L. Christiansen, and @suneman will be more technical than usual.
The paper addresses how to depict ranges of outcomes in simulations of COVID outbreaks (or other stochastic processes). https://arxiv.org/abs/2007.05035
The paper addresses how to depict ranges of outcomes in simulations of COVID outbreaks (or other stochastic processes). https://arxiv.org/abs/2007.05035
2. As such it is of critical importance to people modeling the pandemic. But it's also important for anyone who wants to read and interpret models that others have created, because it warns against a trap that would be easy to fall into.
3. Disease outbreaks are fundamentally stochastic processes. The same disease, introduced into the same population, might infect a large number of people one time, and disappear quickly another time, based on the luck of the draw.
4. We try to account for this in simulations by running multiple iterations and showing the range of possibilities that can occur given a particular set of parameters. One way to do that is to simply show an ensemble of epidemic curves each from a different simulation run, e.g:
5. That's actually not a bad approach, but these plots can get pretty messy. It would be nice to have a way to plot something like confidence ranges on how bad the pandemic will be, according to the model.
For example, suppose you want a 25%-75% central range.
For example, suppose you want a 25%-75% central range.
6. One tempting approach is, at every time point, to find the simulation run is above 75% of the others, and link these points together to form an upper envelope. Then you'd do the same for the 25th percentile to form a lower envelope.
I see this all too often.
Don't do it.
I see this all too often.
Don't do it.
7. Or if you see it, don't trust it.
Juul et al. show that this approach systematically underestimates the severity of a pandemic.
They also provide a lovely intuitive explanation for how such an approach goes wrong.
Juul et al. show that this approach systematically underestimates the severity of a pandemic.
They also provide a lovely intuitive explanation for how such an approach goes wrong.
8. Imagine a scenario where you know exactly what the epidemic curve will look like once the epidemic begins—but you don't know when it will start. It might be on day one, or it might not happen until several weeks later. If you simulate this, you'd get the blue curves below:
9. The gray shaded region is the 25-75% envelope. It never reaches anywhere near the peak height because only a few simulation runs will reach their peaks around any particular time point.
Every single epidemic you simulated is substantially worse than the 25-75% envelope!
Every single epidemic you simulated is substantially worse than the 25-75% envelope!
10. The same thing happrens if you try to trace a median curve by finding—for every given time point—the simulation that is larger than half and smaller than half of the other runs. This fake "median" epidemic curve is shown in black and looks nothing like any individual curve.
11. In general, this sort of thing can be a problem whenever the epidemic curves from different runs of your simulation cross one another. That can happen in many ways, but it becomes particularly likely when the *timing* of events, such as introductions, varies stochastically.
12. So what to do? You could just plot the trajectories themselves and let the reader try to interpret.
Another approach that Juul and colleagues describe is known as curve-based description statistics. I won't go into it here.
Another approach that Juul and colleagues describe is known as curve-based description statistics. I won't go into it here.
13. A third approach is even simpler to conduct and to interpret: plot the probabilities of reaching some threshold. E.g., in what fraction of runs are 50% infected by the end of the epidemic, or in what fraction of runs do 100 people need hospitalization at the same time?
14. Doing so you lose some shape information about the trajectories, but you can still tell complex stories, as illustrated by the figure below. This shows the probability of having a given number of people infected for a specific number of consecutive days.
15. I found this paper very useful as a cautionary tale about visualizing ensembles of simulations. It would be easy to make the mistakes that the authors caution against. So the take home message seems to be twofold.
16.
i) For modelers, don't make these mistakes!
ii) For readers of simulation papers, if you see median trajectories or percentile ranges, make sure they have not been computed incorrectly, lest you underestimate the severity that the models predict.
/fin
i) For modelers, don't make these mistakes!
ii) For readers of simulation papers, if you see median trajectories or percentile ranges, make sure they have not been computed incorrectly, lest you underestimate the severity that the models predict.
/fin
Adding a pointer to another approach from @vnminin: https://twitter.com/vnminin/status/1283264115062919168
