MBA Rankings Denied
In an interesting move that's caused mixed feelings both on and off campus, The Wharton School (and simultaneously Harvard) recently decided to cease giving out email address lists to publications requesting them for the purpose of building rankings of MBA schools. The publications needed these addresses in order to perform the student surveys that are a major input into the ranking process. There are a lot of reasons for such a move, and a few risks involved also, but all in all I think it was a good move, as the rankings had taking on almost mythical status, and the schools were almost beholden to them given the effect they can have on applications and the brand value of the school name.
Of course this is in no way a killing blow to the publication of MBA rankings. The rankings will be published all the same, but without the benefit of a deep and high-response survey of current students and alumni. It is entirely possible that even without the school's help, the publications will be able to solicit sufficient student responses to claim a statistically significant sample (lots of problems with this dealt with below). It's fair to say that many of the statistics that go into the rankings are publically available from the school websites, and the rankings merely transform these into elements in a scoring system.
Why might it be a good thing to stop assisting in the creation of such a ranking system, and why would the Wharton school, which sits at the top of most of these rankings, be one of the schools leading the divorce?
- What does it mean to be the #1 school? Rankings always rank ordinally, and while you can dissect the rankings to see who is best at this or that, generally the overall ranking is what draws all the attention. But schools differ qualitatively in terms of the experience to the student and the qualities of various departments. They also differ in terms of teaching styles, academic vs professional activity tradeoffs, and other dimensions that result in differences in specific scores, reflecting strategic choices by the schools and the differences in their offerings rather than any real differential between the value of studying in one versus the other.
- There's been a fear of late that rankings are defined just as much by editorial considerations than they are by an objective ranking process. Since the ranking/scoring system changes year-on-year, it's clear that comparisons are dubious at best, and manipulative at worst. Quite possibly, Harvard and Wharton will be penalised by their decision to quit aiding the process. A school does not become worse because it chooses to withhold information, and should they drop in the rankings as a consequence of this decision, it will vindicate the decision itself.
- The rankings are not helpful. They seem to be because they are the only information available, but this is misleading. Comparing Wharton to Harvard, for example, is a futile exercise without visiting the schools or speaking to the students. No series of data points, no matter how well analysed (and they are not well analysed in the rankings) is going to give a prospective student the insights they need to come even remotely close to making an informed decision.
- The schools are so hugely affected by their position in the rankings that they end up doing things to improve their position. Since the well-being of the school, the students, the quality of the education, the school brand and the strength of the alumni network are not perfectly correlated with the measurements taken by the rankings, this can result in the schools expending resources in ways that are less than perfect in terms of their alignment with the school's stakeholder's best interests.
- Schools at this level shouldn't need an external validation by a publication. If measures must be taken, let them be audited, properly designed and controlled, and administered by a completely neutral entity with no potential of a conflict of interest. We're talking about the 30-50 best business schools in the world, and having their reputations and relative worths bandied about by external, non-accountable publications is too much of a degree of randomness for comfort. Let a central authority perform the ranking work, and let the publications comment on that rankings. They won't like it because the rankings present a great opportunity to sell lots of copies, but it's better this way.
I have a number of fears regarding the decision, however.
The rankings, as I said above, will be published regardless. This will now be done with incomplete information. A number of scenarios are possible. For example, the publications in question could quite easily decide to hammer reticent schools in their rankings as a form of punishment in return for their disintermediation. They may believe that the corresponding loss of revenue and the drop in the applicant pool for the schools in question would cause them to reconsider their decision and bring them back into the fold. This would be extremely unpleasant, and would vindicate the decision to quit the rankings, but may increase the cost of doing so considerably.
One of the ways the publications in question are trying to get around this is by soliciting feedback from the students without the help of the school administration - through advertising and so on. By doing this they may well get a sample that looks statistically significant, but this would be terribly misleading. This approach would only obtain the feedback of the students that feld they had something to say - the critics always have the most willingness to speak up, and you may get a sample of the least satisfied and most gung-ho of the graduating class. You don't accurately measure a distribution by examining it's tails (as our marketing research professor might say).
All things considered, I think this is both the right thing to do, and a courageous thing to do - it takes someone who's gone through the application process or who's been to one of these schools to understand the unbelievable power wielded by the rankings - we gave them that power by playing their game for so long. It may hurt, but we're taking it back. We'd rather be the masters of our own destinies.
Posted by nlvp at April 16, 2004 10:27 PM