College Football Playoff Committee Rankings Lack Transparency

The first ever college football playoff committee rankings were released today, and as expected, the initial standings incited drama and controversy.  While I agree with many of the pundits that there is nothing to be overly concerned with in terms of which particular teams are currently located in the top four as there is plenty of football still to be played, I do take issue with the process and the apparent priority demonstrated by these initial rankings.

The destruction of the BCS and the inception of the playoff system was purported to have three general goals: increased objectivity, increased access and increased transparency.  While I have argued since the prospect of a committee was being discussed that this vastly decreased objectivity, this is not the current point of issue, though hopefully some are beginning to see the light on this topic.  Access has certainly increased to the Championship Game as four teams will now have an opportunity as opposed to two, though access to the major bowls can be seen as more limited for teams outside the power five in the new system.  My main concern on the day of the first rankings release, however, is the third goal, transparency.

If you watched ESPN’s half hour coverage of the rankings and heard Committee Chairperson Jeff Long speak, you likely were as unsatisfied with the results as I was.  The head of the playoff committee didn’t provide any tangible information or detail, avoided most of Rece Davis’ questions and actually spoke about the wrong team on two occasions.  Hopefully with time, his on-air appearances improve, but I am more concerned with the lack of information that the woeful stage presence.  We were given no description of the committee’s voting process, no information on the importance of any of the possible metrics and no information on the particulars of the results.  In short, we put the first nine weeks of the college football season into a black box, and it spit out a list of teams ordered 1 to 25.  Transparency indeed.

After my disappointment with the telecast, I recalled how upsetting it was during the BCS era to see people rage about a systems and its lack of transparency due generally to ignorance of the available information.  I would hate to suffer the same fate to the playoff system, so I headed towards their official website.  After some searching, I did find a general description of the selection method proposed and adopted by the committee on June 20, 2012, as well as another statement by Jeff Long given as a press release along with the first rankings.  The statement is as follows:

The College Football Playoff selection committee met for the past two days and we engaged in a lively and detailed discussion about who the best teams are in college football, as of October 26.  We debated, we reviewed facts and statistics, and we used our judgment.  There are 18 one-loss teams in the FBS and the differences between many of these teams are slight.  The bottom line is it’s early, it’s close and it’s going to change. – Jeff Long

Thanks Jeff, that’s about as insightful as your ESPN appearance and it sounds to me like you’re already making excuses for the rankings.  If the committee isn’t going to put the effort into the first set of rankings that they are the last set, I would rather they not release them at all, and if they did, take ownership of your product and don’t have your spokesperson tell us it doesn’t matter if they’re wrong because they’ll change anyway.

So the press release didn’t help much. 

The selection process description actually was a bit more helpful, and I encourage everyone to read it.  It is too detailed to go into here, but it can be found on the official website under College Football Playoff Selection Committee Voting Process here.  Essentially, the format is such that everyone votes for six teams, they then take the six teams with the most votes and revote to place them in order.  The top three are ranked accordingly and the bottom three are put back into the mix.  They then repeat this process, ranking three more teams on each iteration until they have a set of 25.  The most important part of this process description, however, was the very last line.  It reads, “All votes will be by secret ballot.” Transparency indeed.

While I can understand the desire of privacy for the individual votes of each member of this committee, and let’s face it, privacy in this case is necessary to ensure the personal safety of these committee members from fans crazy enough to take matters into their own hands, I see no reason to keep the number of votes each team received private.  This information would allow us to see, much like in the AP or Coaches’ Polls, the separation between the fourth and fifth ranked teams.  Was it a tight split between Ole Miss and Oregon, or do the Ducks have some work to do?  Are the top ten teams separated by just a few votes, or is there a significant drop off somewhere?  This is the kind of data that I found disturbingly absent from the initial rankings, and unnecessarily so.

In addition to this lack of data, a recap of this process would be extremely beneficial to all the fans.  To my knowledge, this process has never been explained other than the policy being passively posted on the website.  Furthermore, this is simply the policy that was adopted over two years ago.  I have no way of knowing whether this is actually what took place over the past two days in a boardroom in Dallas.  The only ones that know that are the twelve committee members that sat around that boardroom table, and without a better explanation from Jeff Long, we’ll never really know.  Transparency indeed.

Finally, in addition to the utter lack of transparency into the new process that was demonstrated tonight, I, like many others do have concerns with the overall result.  My largest issue is that while three of the four teams placed in the top four slots are playing in the same division, and will thereby suffer additional losses, if the season ended today, this is what we would be left with.  The excuse that Jeff Long provided, and many of the talking heads repeated, that these are not the final rankings and we should therefore, not be upset by them, implies that either these rankings were not given the same scrutiny as the final rankings will, or that this committee is not looking strongly enough at conference alignments. 

The blindness to conference affiliation was actually a point of emphasis by Jeff Long in his short interview, and it is understandable given the result not only of three playoff teams in the same conference, but actually in the same division.  But while conference affiliation should not be dragging second tier teams into the final four, it should be used to diversify the playoff field.  The reason being is that these committee members are generally using statistical information as well as head to head matchups and common opponents to determine the rankings, as evidenced both by their stated metrics and by the resulting rankings.  These metrics, however, are skewed based on the limited conference cross over which occurs in college football.

Any decent college football statistical analyst will tell you that because there are 126 teams in the FBS and only 12 games in a season, the comparative stats do not truly converge to a desirable degree.  This means that because teams in a conference play other teams in their conference the majority of the time, statistics from one conference to another can hold a wide range of error associated with them.  Stats like strength of schedule, strength of record or FPI all rely on generally well connected data, which college football does not have, so comparing these stats between Oregon and Ole Miss for example, is misleading.  While the Rebels outrank the Ducks in these categories, due to the error on the stats, a clear distinction cannot truly be made.  Thus, the importance of giving a team from an unrepresented conference the benefit of the doubt where the playoff field is concerned.  If this was not the case, why wouldn’t we simply use the computer rankings from the BCS system and be done with it? 

Bottom line – irregardless of first or final rankings, don’t give me three teams from the same division and leave the highest ranked team from another power five conference on the bubble. 

And more importantly than that, if the situation does occur, let’s institute the transparency into the system to tell me how it happened, why it happened, and try to fix it for the next go around.  This is a new system that is bound to have some problems along the way, but with a transparent, open process, we can try to make it the best system yet for college football, its players and its fans.