Thought experiment: If a bicycle cannot fly, is it a defective bicycle?

I only ask because virus prediction models are being called defective based on the fact they do not accurately predict the future.

The public generally doesn't understand the point of prediction models. They are not intended to predict the future. No model can do that. No human can do that. It is not a thing.

When you say, "That model is wrong because it did not accurately predict the future," you are talking nonsense. You are saying a bicycle is defective because it can't fly. Models are not intended to show you the future. Bikes are not intended to fly.

My critics will say, "Hey, you always say analogies can't persuade! What's up with that?" And then I say, "Did it persuade you?" Then they say no. Then I say, "See?"

Analogies can sometimes teach a point. The point here is that you can't rationally judge the effectiveness of a thing if you don't know what it was designed to do in the first place. And models are not designed to be "right."

If anyone could create a complex model with lots of sensitive variables that could predict the future -- of anything -- during the fog of war period of a new challenge, that person would be a trillionaire by now. No one can do that. No model can do that. It isn't a thing.

What models CAN do for you is show the potential size of a problem and persuade non-experts that it is worth fixing. The #coronavirus models did just that. If there was an award for "best prediction model of all time," the #coronavirus models would be contenders.

The lower bounds of the models said 100K deaths but now have been revised to 60K. That's a 40% change! Terrible models! Fire everyone involved! Right?

Keep in mind the models had a range from 100K at the low end, with mitigation, to over 1 million without. If the actual death toll comes in at 140K with mitigation, would you say the models were wrong or right? I think you would say they were damned good.

But if we miss the bottom estimate by exactly the same amount on the low side, and the final tally is 60K dead, you call that a gross miss. Yet it would be a miss by exactly as much as the 140K, only in the other direction.

I contend there is an illusion here that is skewing opinions. If you compare the revised 60K estimate to 100K, it looks like a big difference. But if you consider the range of uncertainty was 900K, depending on mitigation, then 60K and 100K are in the same ballpark.