Read on Twitter
Quick thread highlighting a potentially hot take for fellow data nerds about the @FiveThirtyEight NFL Prediction Model (yes, they do much more than just electoral politics):
I& #39;ve long suspected that the model& #39;s pre-season ELO ratings imply too much parity amongst teams.
(1/n)
I& #39;ve long suspected that the model& #39;s pre-season ELO ratings imply too much parity amongst teams.
(1/n)
Let& #39;s start with how a team& #39;s pre-season ELO is determined.
In short, it& #39;s majority (two thirds) using Vegas odds for the upcoming season, and minority (one third) using last season& #39;s ending ELO mean-reverted by 33%.
Full explanation by the @FiveThirtyEight team pictured
(2/n)
In short, it& #39;s majority (two thirds) using Vegas odds for the upcoming season, and minority (one third) using last season& #39;s ending ELO mean-reverted by 33%.
Full explanation by the @FiveThirtyEight team pictured
(2/n)
My suspicion is that the decision to revert to the system& #39;s mean ELO by specifically 33% is the issue... some interesting questions: Why 33%? Was that the optimal back tested value? If so, should we be concerned about data dredging?
(3/n)
(3/n)
This only matters (in the context of a sports model 
https://abs.twimg.com/emoji/v2/... draggable="false" alt="đ" title="Winking face with tongue" aria-label="Emoji: Winking face with tongue">) if it manifests in the predictions.
Best might be to consider the forecasted full season records of teams immediately prior to the start of the season, when the above described ELOs are running through the model.
(4/n)
Best might be to consider the forecasted full season records of teams immediately prior to the start of the season, when the above described ELOs are running through the model.
(4/n)
As pictured, the model forecasted that no team would *on average* (across tens of thousands of simulations) finish the 2020 season with a record worse than 5 wins and 11 losses. How does that compare with how the worst teams in the league have fared historically?
(5/n)
(5/n)
Since the NFL moved to a 16 game season in 1978, the number of full seasons in which no team finished with a record worse than 5 wins and 11 losses is... zero.
Worse than 4 wins and 12 losses? Just once (2003).
78% of seasons have a team that wins 2 games or less (!!!).
(6/n)
Worse than 4 wins and 12 losses? Just once (2003).
78% of seasons have a team that wins 2 games or less (!!!).
(6/n)
Importantly, this *is not* damning in and of itself. After all, mean reversion is real (and, like, the whole point of this thread...)
That is, *most* teams that finish with a 2-14 record are "better than 2-14" across tens of thousands of simulations. But by how much?
(7/n)
That is, *most* teams that finish with a 2-14 record are "better than 2-14" across tens of thousands of simulations. But by how much?
(7/n)
An efficient approach to learn whether this is worth exploring further (or whether my hypothesis is blatantly wrong) could be to run the current model on each historical season (n=41), and determine the probability that *any team* finishes with 2 wins or less.
(8/n)
(8/n)
Using these 41 data points, we could calculate the value of & #39;x& #39; that the model would have used to complete the statement: "We expect that in [& #39;x& #39;%] of seasons, there will be a team that finishes with 2 wins or less".
Does the model view it as extremely unlikely?
(9/n)
Does the model view it as extremely unlikely?
(9/n)
Unfortunately, this is where we start losing visibility.
There& #39;s a downloadable CSV of the historical data linked, but as far as I can tell, it isn& #39;t reconcilable to the distribution of potential records for each team across many simulations.
https://github.com/fivethirtyeight/data/tree/master/nfl-elo
(10/n)">https://github.com/fivethirt...
There& #39;s a downloadable CSV of the historical data linked, but as far as I can tell, it isn& #39;t reconcilable to the distribution of potential records for each team across many simulations.
https://github.com/fivethirtyeight/data/tree/master/nfl-elo
(10/n)">https://github.com/fivethirt...
If I& #39;m misinterpreting the data and it actually is reconcilable to the model& #39;s predicted records for each team at the start of every season, hopefully someone reading can point out how to me!
(11/n)
(11/n)
Would also love to hear anyone& #39;s thoughts on this, certainly not least including @NateSilver538, @jayboice, and @Neil_Paine ... especially any insider commentary on what they discussed when thinking about reversion to the mean in this particular model!
(12/n)
(12/n)
