A question that’s befuddled us for a long time now has been just how accurate are all these preseason polls that every media entity puts out each year are.  Remember last season – all four NCAA #1 seeds made it to the Final Four, but what was equally interesting to us was that those same four teams – Kansas, Memphis, UCLA and UNC – were also the top four ranked teams (in a different order) in both the Preseason AP and ESPN Coaches polls.  With an n=1, we know that the 2007-08 polls were extremely accurate in predicting last year’s F4 teams, but that only tells us part of the story – what we really want to know is how accurate are preseason polls in general?

To try to answer this question, we had to make some concessions.  We believe that, generally speaking, most preseason polls are largely the same, whether AP, ESPN/Coaches, CNNSI, etc.  Take for example, the blogpoll that came out this week.  The top twenty teams that the bloggers chose were mostly consensus picks – no team was left off of more than one ballot, and a total of only thirty-six teams received at least one vote.  That shows a relatively high consistency of thought – groupthink, if you will – about who the best teams in the country will be this season.  So we feel that we can derive some strong basic principles (and save a boatload of time) by examining only one of the major preseason polls – the ESPN/Coaches Poll – because it is the sole major poll that does a postseason version (after the NCAAs) to enable a fair comparison. 

We looked at the last five years where we could find the available pre- and postseason polls (the 2005 postseason poll is incorrect on both the ESPN and USA Today websites), and made some simple comparisons.  Our findings are below the table. 

Findings.

• In a given year, there are between 50-60 teams receiving votes from the preseason pollsters.  This tightens up to approximately 40 teams receiving votes in the postseason poll. 
• So how does a team receiving preseason votes equate to the postseason?  Ehhh, not terrible, but not great either.  Over the last six seasons (excl. 2005), if a team received votes in the preseason poll, there was a slightly better than half (54%) chance that it would also get votes in the postseason poll.  That alone doesn’t tell us a whole lot, though.  What if your team was in the preseason Top 25?  Those teams receive votes in the final poll approximately three-quarters (76%) of the time, which at minimum, means that the takeaway is that a preseason team receiving votes will usually make the NCAA Tournament. 
• Looking at the distribution of the final postseason polls can tell us a little bit about how accurate preseason pollsters are at predicting how good a team will be.  There appears to be a much stronger tendency to overlook teams that turn out later to be good rather than to overrate teams that turn out to not as good as pollsters thought.  Over half of the teams in a given year (~23) in the final postseason poll will have moved up >5 spots in the rankings from their initial selection; but only a handful of teams (~7) will have moved down by >5 spots from the preseason.  Another ~12 teams won’t move much from its initial standing.  This is strong evidence that pollsters generally have an accurate sense of the abilities of about 30% of teams in a given year, but they’re far more likely to underrate teams (usually by not ranking them at all) than to overrate teams (by a 3:1 ratio). 
• Some of the more notable examples of the pollsters being right on the money were in 2004, when they rated UConn/Duke as #1/#2, which is exactly where they ended the season.  Florida rated as preseason #1 in 2007 and Kansas as preseason #2 in 2003 were some other clear winners. 
• The swing-and-a-misses where the pollsters vastly overrated a team were Indiana in 2008 (#9 to #33), Duke in 2007 (#11 to #38), and Michigan St. in both 2006 (#5 to #34) and 2005 (#3 to #41).   
• The biggest misses where pollsters underrated a team was most obvious in 2003 and 2007, when preseason #31 Syracuse and #39 Florida, respectively, vaulted all the way to #1 by season’s end, and in 2004 when preseason unranked Georgia Tech made it to the F4 and #3 at the end of the year.  The only other preseason unranked team to have made the F4 in the last six years was George Mason in 2006. 

What does this mean for the 2008-09 season?  Well, if your team was ranked in the Top 25, you’re more than likely going to make the NCAA Tournament.  And if you’re already highly ranked, you should feel relatively secure in your position at or near the top – most teams simply don’t have huge drops in rankings from beginning to end of the season.  The good news is that if your team was lower ranked or not ranked at all, but you feel like they’re extremely underrated, history shows that an awful lot of teams move significantly up the rankings as the season goes along.  We’ll leave the guesswork as to who those teams might be to the rest of you guys. 

rtmsf (3998 Posts)

This entry was posted on Friday, November 14th, 2008 at 4:08 pm by rtmsf and is filed under rtc analysis. Tagged: ap poll, blogpoll, connecticut, duke, espn coaches poll, florida, george mason, georgia tech, indiana, kansas, memphis, michigan st, preseason polls, syracuse, ucla, unc. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.