What do rankings measure? The U.S. news rankings and student experience at liberal arts colleges
Kim J.; Shim W.-J.
2019
Review of Higher Education
8
10.1353/rhe.2019.0025
The purpose of this study is to evaluate the validity of college rankings, by investigating to what extent measurements employed by rankings are good proxies for evaluating educational practices at liberal arts colleges. Using a hierarchical linear modeling with data from the Wabash National Study of Liberal Arts Education, Integrated Postsecondary Education Data System and the U.S. News and World Report (USNWR) college rankings, we found that overall ranking scores did not predict students’ exposure to good practices, but scores on some measures had a limited yet significant association with students’ experiences of good practices at liberal arts colleges. © 2019 Association for the Study of Higher Education.
College rankings; Liberal arts education; Student experience; U.S. news rankings
Anaya G., College impact on student learning: Comparing the use of selfreported gains, standardized test scores, and college grades, Research in Higher Education, 40, 5, pp. 499-526, (1999); Anderson N., Five colleges misreported data to U.S. News, raising concerns about rankings, reputation, The Washington Post, (2013); Astin A.W., What matters in college? Four critical years revisited, (1993); Astin A.W., How the liberal arts college affects students, Daedalus, 128, 1, pp. 77-100, (1999); (2006); Bowman N.A., Can 1st-year students accurately report their learning and development?, American Educational Research Journal, 47, 2, pp. 466-496, (2010); Bowman N.A., Validity of college self-reported gains at diverse institutions, Educational Researcher, 40, 1, pp. 22-24, (2011); Bowman N.A., Bastedo M.N., Anchoring effects in world university rankings: Exploring biases in reputation scores, Higher Education, 61, 4, pp. 431-444, (2011); Brooks R., Measuring university quality, The Review of Higher Education, 29, 1, pp. 1-21, (2005); Brown R., League tables-Do we have to live with them?, Perspectives: Policy and Practice in Higher Education, 10, 2, pp. 33-38, (2006); Carter T., Rankled by the rankings, ABA Journal, 84, pp. 46-52, (1998); Cech T.R., Science at liberal arts colleges: A better education?, Daedelus, 128, 1, pp. 195-216, (1999); Chickering A.W., Gamson Z.F., Seven principles for good practice in undergraduate education, AAHE Bulletin, 39, 7, pp. 3-7, (1987); Chickering A.W., Gamson Z.F., Applying the seven principles for good practice in higher education, New Directions for Teaching and Learning, 1999, 80, pp. 75-81, (1991); Chickering A.W., Reisser L., Education and identity, (1993); Clarke M., Some guidelines for academic quality rankings, Higher Education in Europe, 27, 4, pp. 443-459, (2002); Clarke M., The impact of higher education rankings on student access, choice, and opportunity, Higher Education in Europe, 32, 1, pp. 59-70, (2007); Clotfelter C.T., Buying the best: Cost escalation in elite higher education, (1996); Cohen J., Statistical power analysis for the behavioral sciences, (1988); Dill D.D., Will market competition assure academic quality? An analysis of the UK and US experience, Quality assurance in higher education: Trends in regulation, translation and transformation, pp. 47-72, (2007); Dill D.D., Soo M., Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems, Higher Education, 49, 4, pp. 495-533, (2005); Ehrenberg R.G., Reaching for the brass ring: The U.S. News & World Report rankings and competition, Review of Higher Education, 26, 2, pp. 145-162, (2003); Ehrenberg R.G., Method or madness? Inside the U.S. News & World Report college rankings, Journal of College Admissions, 189, pp. 29-35, (2005); Ehrenberg R.G., Zhang L., Do tenured and tenure-track faculty matter?, Journal of Human Resources, 40, 3, pp. 647-659, (2005); Espeland W.N., Sauder M., Rankings and reactivity: How public measures recreate social worlds, American Journal of Sociology, 113, 1, pp. 1-40, (2007); Ewell P.T., Jones D.P., Actions matter: The case for indirect measures in assessing higher education’s progress on the national education goals, The Journal of General Education, 42, 2, pp. 123-148, (1993); Fombrun C.J., Reputation: Realizing value from the corporate image, (1996); Grunig S.D., Research, reputation, and resources: The effect of research activity on perceptions of undergraduate education and institutional resource acquisition, The Journal of Higher Education, 68, 1, pp. 17-52, (1997); Harrison W.B., Metchell S.K., Peterson S.P., Alumni donations and colleges’ development expenditures: Does spending matter?, The American Journal of Economics and Sociology, 54, 4, pp. 397-412, (1995); Hazelkorn E., Rankings and the reshaping of higher education: The battle for world-class excellence., (2011); Hayek J.C., A student-centered approach for identifying high-performing colleges and universities (Doctoral dissertation), (2001); Heck R.H., Thomas S.L., An introduction to multilevel modeling techniques, (2008); Hossler D., The problem with college rankings, About Campus, 5, 1, pp. 20-24, (2000); Hu S., Kuh G.D., Being (dis)engaged in educationally purposeful activities: The influences of student and institutional characteristics, Research in Higher Education, 43, 5, pp. 555-575, (2002); Hu S., Kuh G.D., Diversity experiences and college student learning and personal development, Journal of College Student Development, 44, 3, pp. 320-334, (2003); Hughes R., Pace C.R., Using NSSE to study student retention and withdrawal, Assessment Update, 15, 4, pp. 1-2, (2003); Jaschik S., Lederman D., The 2012 Inside Higher Ed Survey of College and University Admissions Directors, Inside Higher Ed, (2012); Kim J., The functions and dysfunctions of college rankings: An analysis of institutional expenditure, Research in Higher Education, 59, 1, pp. 54-87, (2018); Kuh G.D., Hu S., The effects of student-faculty interaction in the 1990s, Review of Higher Education, 24, 3, pp. 309-332, (2001); Kuh G.D., Pascarella E.T., What does institutional selectivity tell us about educational quality?, Change: The Magazine of Higher Learning, 36, 5, pp. 52-59, (2004); Levin D.J., The uses and abuses of the “U.S. News” rankings, (2002); Luca M., Smith J., Salience in quality disclosure: Evidence from the U.S. News college rankings, Journal of Economics and Management Strategy, 22, 1, pp. 58-77, (2013); Martins L.L., A model of the effects of reputational rankings on organizational change, Organization Science, 16, 6, pp. 701-720, (2005); Massy W.F., Zemsky R., Faculty discretionary time: Departments and the “academic ratchet.”, Journal of Higher Education, 65, 1, pp. 1-22, (1994); Meredith M., Why do universities compete in the ratings game? An empirical analysis of the effects of the U.S. News and World Report college rankings, Research in Higher Education, 45, 5, pp. 443-461, (2004); Meyers C.B., Union status and faculty job satisfaction: Contemporary evidence from the 2004 National Study of Postsecondary Faculty, The Review of Higher Education, 34, 4, pp. 657-684, (2011); Morse R., Brooks E., Mason M., How U.S. News calculated the 2017 Best Colleges rankings, The U.S. News and World Report, (2016); Museus S.D., The culturally engaging campus environments (CECE) model: A new theory of college success among racially diverse student populations, Higher Education: Handbook of Theory and Research, pp. 189-227, (2014); Improving the college experience: National benchmarks for effective educational practice., (2001); A review of the methodology for the U.S. News & World Report’s rankings of undergraduate colleges and universities, (1997); Niehaus E., Campbell C.M., Inkelas K.K., HLM behind the curtain: Unveiling decision behind the use and interpretation of HLM in higher education research, Research in Higher Education, 55, 1, pp. 101-122, (2014); Okunade A.A., Logistic regression and probability of business school alumni donations: Micro-data evidence, Education Economics, 1, 3, pp. 243-258, (1993); Pace C.R., The credibility of student self-reports, (1985); Pascarella E.T., Terenzini P.T., How college affects students. Findings and Insights from Twenty Years of Research., (2005); Pike G.R., Measuring quality: A comparison of the U.S. News rankings and NSSE benchmarks, Research in Higher Education, 45, 2, pp. 193-208, (2004); Pike G.R., NSSE benchmarks and institutional outcomes: A note on the importance of considering the intended uses of a measure in validity studies, Research in Higher Education, 54, 2, pp. 149-170, (2013); Pike G.R., Smart J.C., Kuh G.D., Hayek J.C., Educational expenditures and student engagement: When does money matter?, Research in Higher Education, 47, 7, pp. 847-872, (2006); Porter S.R., The robustness of the graduation rate performance indicator used in the U.S. News & World Report college rankings, The Case International Journal of Educational Advancement, 1, 2, pp. 145-164, (2000); Porter S.R., Institutional structures and student engagement, Research in Higher Education, 47, 5, pp. 521-558, (2006); Porter S.R., Do college student surveys have any validity?, The Review of Higher Education, 35, 1, pp. 45-76, (2011); Raudenbush S.W., Bryk A.S., Hierarchical linear models: Applications and data analysis methods, (2002); Ryan J.F., Institutional expenditures and student engagement: A role for financial resources in enhancing student learning and development, Research in Higher Education, 46, 2, pp. 235-249, (2005); Sarraf S., Hayek J., Kandiko C., Padge R., Harris K., The institutional quality debate: U.S. News quality indicators and the National Survey of Student Engagement, (2005); Sauder M., Fine G.A., Arbiters, entrepreneurs, and the shaping of business school reputations, Sociological Forum, 23, 4, pp. 699-723, (2008); Sauder M., Lancaster R., Do rankings matter? The effects of U.S. News & World Report rankings on the admissions process of law schools, Law & Society Review, 40, 1, pp. 105-134, (2006); Sanoff A., Rankings are here to stay: Colleges can improve them, Chronicle of Higher Education, 45, 2, (1998); Smart J.C., Ethington C.A., Riggs R.O., Thompson M.D., Influences of institutional expenditure patterns on the development of students’ leadership competencies, Research in Higher Education, 43, 1, pp. 115-132, (2002); Sponsler B., The role and relevance of rankings in higher education policymaking, (2009); The Princeton Review’s college ranking methodology; Higher education review; How U.S. News Calculated the Best Colleges Rankings; Thompson, Playing with numbers, Washington Monthly, pp. 16-18, (2000); Titus M., An examination of the influence of institutional context on student persistence at 4-year colleges and universities: A multilevel approach, Research in Higher Education, 45, 7, pp. 673-699, (2004); Toutkoushian R.K., Smart J.C., Do institutional characteristics affect student gains from college?, The Review of Higher Education, 25, 1, pp. 39-61, (2001); Tsao J.C., Coll G., To give or not to give: Factors determining alumni intent to make donations as a PR outcome, Journalism & Mass Communication Educator, 59, 4, pp. 381-392, (2004); Umbach P.D., How effective are they? Exploring the impact of contingent faculty on undergraduate education, The Review of Higher Education, 30, 2, pp. 91-123, (2007); Usher A., Savino M., A world of difference: A global survey of university league tables, (2006); Usher A., Savino M., A global survey of university ranking and league tables, Higher Education in Europe, 32, 1, pp. 5-15, (2007); Vaughn J., Accreditation, commercial rankings, and new approaches to assessing the quality of university research and education programmes in the United States, Higher Education in Europe, 27, 4, pp. 433-441, (2002); Webster T.J., A principal component analysis of the U.S. News & World Report tier rankings of colleges and universities, Economics of Education Review, 20, 3, pp. 235-244, (2001); Webster T.J., A principal components analysis of the U.S. News & World Report tier rankings of national liberal arts colleges, The Journal of Applied Business Research, 17, 1, pp. 39-54, (2001); Winston G.C., Zimmerman D.J., Peer effects in higher education, College choices: The economics of where to go, when to go, and how to pay for it, pp. 395-424, (2004)
Johns Hopkins University Press
Article
Scopus