A large number of people have developed models for predicting the outcomes of college basketball games. For those that have made their picks publicly available, ThePredictionTracker does a great service by tracking the live performance of each model over the course of the season. Unfortunately, it's difficult to do a direct comparison of models using the summary page on the Tracker. For one thing, each model has predicted a different subset of games (in many cases this is accidental -- schedules get modified and web scrapers don't pick up the changes -- but some models don't start making picks altogether until weeks or months into the season). Further, there are a few misprinted lines in the Tracker data. For example, the Tracker shows an opening line of -22 and a closing line of +64.5 for UCLA vs. Presbyterian on 11/19/2019 (the line closed at -23 or -23.5 depending on the book).

Throughout the 2018-19 season, I'll try to update this page with some further analysis of the Tracker data. For what follows, I chose a subset of models which have made picks since the very beginning of the season, and I threw out games for which any of those models did not make a pick. In a (somewhat lazy) attempt to address misprinted lines, I filtered out any games for which the opening and closing lines differed by more than 5 points (it's very rare that this really happens).

Results are shown as of 2019-02-14 for a set of 3867 games:
Mean Squared Error (MSE) Model
124.276 Line
125.529 Opening Line
125.941 Erik Forseth
126.722 TeamRankings
127.983 Dokter Entropy
130.353 Sagarin Rating
130.695 Sagarin Predictor
131.170 Sagarin Golden Mean
131.634 ESPN BPI
132.416 Kenneth Massey
135.191 DRatings.com
135.547 Sonny Moore
140.241 StatFox
141.324 ComPughter Ratings
146.920 Sagarin Recent

Accuracy Model
0.747 Opening Line
0.745 Erik Forseth
0.744 Line
0.742 Sagarin Predictor
0.740 ESPN BPI
0.740 Sagarin Rating
0.740 TeamRankings
0.738 Sagarin Golden Mean
0.736 Dokter Entropy
0.734 Kenneth Massey
0.732 DRatings.com
0.730 Sonny Moore
0.729 StatFox
0.728 ComPughter Ratings
0.722 Sagarin Recent


Clearly the line is the best statistical predictor of the outcome. Nevertheless, we can ask how each model would have done against the spread, shown below:
% Against the Spread Model
0.514 ESPN BPI
0.512 Sagarin Golden Mean
0.510 Dokter Entropy
0.505 Sagarin Rating
0.504 Sagarin Predictor
0.503 Sonny Moore
0.502 Erik Forseth
0.499 Kenneth Massey
0.498 TeamRankings
0.497 StatFox
0.495 Opening Line
0.493 DRatings.com
0.491 Sagarin Recent
0.481 ComPughter Ratings


Although no individual model predicts the point spread as well as the line, we might ask whether any linear combination of models can do so. Let's regress the observed margins of victory onto the predictions of each model, but constrain the regression to have nonnegative coefficients. Subject to the nonnegativity constraint, this would give the optimal (backward-looking) mixture of predictors. We find:
Coefficient Model
0.459 Erik Forseth
0.269 Dokter Entropy
0.197 ESPN BPI
0.063 Sonny Moore
0.026 Sagarin Golden Mean
0.003 StatFox
0.000 Kenneth Massey
0.000 Sagarin Rating
0.000 TeamRankings
0.000 Sagarin Recent
0.000 Sagarin Predictor
0.000 DRatings.com
0.000 ComPughter Ratings

The MSE of this hypothetical predictor would be 124.838. Note that this is optimistic, since in addition to being backward-looking, we both fit the model and then computed the MSE using the full dataset.

Out of curiosity, what if we included the line itself in the above regression? Can any of our models add value when combined with the line?
Coefficient Model
0.649 Line
0.156 Erik Forseth
0.113 ESPN BPI
0.092 Dokter Entropy
0.016 Sonny Moore
0.000 Opening Line
0.000 Kenneth Massey
0.000 Sagarin Rating
0.000 TeamRankings
0.000 StatFox
0.000 Sagarin Recent
0.000 Sagarin Predictor
0.000 Sagarin Golden Mean
0.000 DRatings.com
0.000 ComPughter Ratings

Interestingly, it seems that a couple of the models do perhaps capture something the line does not. The hypothetical MSE of this mixture would be 123.841.

At some point I will make my analysis available as a Jupyter notebook.