CAUTION AND CONTROVERSY
Many people question the use and usefulness of rankings services such as those found at this site. Without a doubt, knowledge of how rankings are constructed, leavened with a dash of skepticism, can go a long way to ensuring they are used appropriately. In addition to the print articles listed in our rankings bibliography, many interesting articles may be found on the web concerning what one should make of rankings. Among these are "Ignore College Ranking -- Become an Educated Consumer" by Shirley Levin [1]. Glenn Kersten, a research librarian with the Suburban Library System Reference Service, wrote a wonderful article titled "Grading on the Curve: College Ratings and Rankings" [2]. This article provides an excellent overview of the topic and should be mandatory reading for anyone interested in the phenomenon of college ratings and rankings.
Two organizations which do not rank schools, but rather give comparative information about schools, also provide discussion of how to assess the quality of schools and ranking services. The College Board offers "Rankings & Ratings" [3], an interesting discussion of the impact of rankings, while Peterson's presents "Considering College Quality" [4], a discussion of assessing institutional quality. Peterson's also explains why they don't believe in rankings [5].
Just as the general public is concerned about rankings, academicians are increasingly debating the importance and validity of college rankings. Two articles from the Chronicle of Higher Education, the premier news publication for academe, offer excellent perspectives on the subject. "Changes in Annual College Guides Fail to Quell Criticisms on Their Validity" [6] and "A Self-Published Guide Goes Big-Time, and Educators Cry Foul" [7] both provide insight into the ongoing controversy. The latter article focuses on the Gourman Report, a print-only source which has been singled out for heavy criticism by those interested in institutional evaluation. Many observers decry the Gourman Report's lack of identifiable criteria and methodology. As with all other ranking services, we advise anyone consulting the Gourman Report to exercise caution and not overemphasize its seemingly definitive numerical rankings.
Many universities, including highly ranked ones, question both the data and the processes used by some of the ranking services. Of special concern are the aspects of the rankings which deal with the difficult-to-measure concept of institutional reputation. For interesting insights into the topic of university reputations, please read "Building Reputations: How the Game is Played" by William Sharp of Columbia University [8].
The news magazine U.S. News & World Report has long been the target of criticism for its annual ranking of colleges and universities. In 1996, Stanford University President Gerhard Casper kicked off a wave of controversy over the U.S. News rankings with his letter to James Fallows, the magazine's editor [9]. Casper followed this up with "An Alternative to the U.S. News and World Report College Survey," a press release from Stanford University [10].
The online publication Slate ran a story in August of 1999 questioning U.S. News' motives and methods. In an article titled "Cooking the school books: How U.S. News cheats in picking its best American colleges," author Bruce Gottlieb put forth many criticisms of the rankings and accused U.S. News of changing its methodology from year to year in order to shake up the top ranked schools and keep the public's interest [11]. Gottlieb stated, "A successful feature like this [i.e., the U.S. News rankings] requires surprise, which means volatility. Nobody's going to pay much attention if it's Harvard, Yale, and Princeton again and again, year after year." Two U.S. News editors, Peter Cary and Brian Duffy, took to the pages of Slate to respond with "Dissension in the rankings: U.S. News responds to Slate's best colleges story" [12].
"IP Specialty Rankings in U.S. News & World Report" is a fascinating look at rankings from a faculty member at a highly regarded law school [13]. Thomas G. Field, professor in the top ranked intellectual property law program at the Franklin Pierce Law Center, authors the article and comes to the conclusion that the answer to the question "what do the U.S. News rankings mean?," is "not much."
In February 1998, a call for an end to rankings was released by the Association of American Law Schools (AALS) [14]. The AALS called the U.S. News & World Report law school rankings "misleading and dangerous," and released a study questioning the validity of the rankings [15]. Nearly all of the deans of the 180 American Bar Association approved law schools signed a letter sent by the Law School Admissions Council (LSAC) to over 90,000 law school applicants. The letter, titled "LSAC Deans Speak Out," cautioned applicants to not rely on rankings but to make a choice based on what is right for them as individuals [16]. Another law school organization, NALP or the National Association for Law Placement, also provides an excellent cautionary essay titled "Law School Rankings: Through the Education and Employment Looking Glass" [17].
Business school rankings have also come under scrutiny. "What's Wrong With MBA Ranking Surveys?" is the title of an article by Martin Schatz, Ph.D [18]. Dr. Schatz points out some of the flaws of the major business school rankings and notes that no one program can possibly be predetermined as the best for any individual.
In early 2000, the popular Yahoo! Internet Life ranking of the most wired campuses also came under fire as a group of eleven traditionally well-respected universities refused to participate in the survey. According to an article in the Chronicle of Higher Education, many institutions do not feel the Yahoo! ranking accurately reflects how schools are using information technology to meet their educational and research goals [19]. The institutions choosing not to participate in the Yahoo! rankings are Brown, Cornell, Duke, Harvard, Princeton, Stanford, Yale and the Universities of Chicago, California at Berkeley, Michigan, and Washington.
Publishers have been quick to answer with information about their ranking procedures. With the release of the latest version of its undergraduate rankings in August 1999, U.S. News & World Report provides further explanation of its system. The site's "Frequently asked questions about the U.S. News college rankings" offers information on why and how they rank as well as how one should use the rankings [20]. They also offer "Ranking Methodology" [21] for their graduate school site along with "Frequently asked questions about the U.S. News graduate school rankings" [22].
Furthermore, U.S. News presents the article "Universities use rankings too," which points out that colleges often rank students but seem to prefer not to be ranked themselves [23]. As part of its Vision 2020 strategic planning process, Texas A&M University is assessing "Where we are" by using college rankings to develop peer groups for institutional assessment purposes [24].
Efforts are underway by the various concerned parties to introduce some uniformity into the debate over evaluating colleges and universities. A few major publishers are working with the educational community on the Common Data Set, an attempt to standardize institutional data gathering and reporting [25]. Examples of universities with web pages based on Common Data Set information are:
Overall, there is a bit of a "can't live with 'em, can't live without 'em" aspect to college rankings. Colleges routinely disparage rankings but are quick to trumpet their high standing and paste the U.S. News best college graphic on their web site. Schools are even known to tinker with their admissions policies, alumni files, and other "ranking factors" in order to maintain or boost their U.S. News rating. (For more on this, see the Machung and Monks entries on our rankings bibliography.) Like them or leave them, colleges will certainly devote no less attention to rankings in the future. Likewise, students and parents will continue to seek out the best school and continue to consult the wide variety of available college rankings. Being aware of the controversy and "game playing" involved in college rankings can only help interested parties make better use of rankings in their decision making processes.
Bibliography: Please see this page for links to sources for this article.
