For years, higher education reformers have claimed that transparency and accountability are the keys to creating change in America’s colleges and universities.
Some information about schools—tuition, admissions standards, and student body characteristics—is easy to find, either on colleges’ own websites and in national rankings lists such as Princeton Review or U.S. News and World Report. But information about learning outcomes—such as how much students’ knowledge increased over their four years of schooling—or students’ experiences, such as study habits and engagement in the classroom, is thin.
The Spellings Commission on the Future of Higher Education chastised academia for its failure to report on academic performance. The commission’s 2006 report, A Test of Leadership: Charting the Future of U.S. Higher Education, proposed creating a public database that would make available statistics and other information about colleges and universities. The data would include the cost, price, admissions data, completion rates, and the “learning outcomes of students.” The commission argued that colleges would have a more vested interest in the success of their students if this information were made public to prospective students and their parents.
In response to this federal pressure, in 2007 two websites began to provide information in two key areas: colleges’ learning outcomes and student engagement and attitudes (in addition to providing information about other college characteristics). Two years later, it’s time to consider how effective they are.
The two leading associations of public universities created the Voluntary System of Accountability (VSA) to help their schools demonstrate accountability to the public by measuring educational outcomes and making information accessible, understandable, and comparable. The main way in which they do that is through “The College Portrait.” This website supplies basic information such as cost of attendance and graduation rates, but also includes measures of student experiences on campus and student learning outcomes at 520 public colleges and universities.
The leading organization of private colleges has come up with its own version of “College Portrait,” called the University and College Accountability Network, or U-CAN, which provides information about the nation’s private colleges and universities.
While the two systems are somewhat similar, each has its own peculiarities. In 2007, the VSA recommended testing freshmen and seniors to help institutions better demonstrate students’ learning during their years in college. The VSA listed three tests that universities could use to measure those outcomes: the College Learning Assessment (CLA), the Measure of Academic Proficiency and Progress (MAPP) or the Collegiate Assessment of Academic Proficiency (CAAP). All the tests measure some aspects of critical thinking, analytic reasoning, and written communication.
The VSA also recommended tests to measure student attitudes and engagement. These include the Cooperative Institutional Research Program’s Your First Year College (YFYC) survey and its College Senior Survey (CSS), the National Survey of Student Engagement (NSSE), the College Student Experiences Questionnaire (CSEQ) and the University of California Undergraduate Experience Survey (U-CUES, used only by UC schools). The NSSE is by far the most widely used among these tests; in 2009, 23 North Carolina schools used the NSSE to evaluate student experiences. (Descriptions of each assessment appear at the end of this article.)
Most of these tests weren’t designed to rank schools or to evaluate the success of schools in a public comparison, but for internal university assessments. However, public pressure has forced the tests into new roles, including use by accreditors.
Most of North Carolina’s public schools and many of its private schools have revealed results on at least two of the recommended tests. Some small private colleges, which compete fiercely for tuition dollars, have revealed results for three or more. All UNC system schools are now required to use the College Learning Assessment, although not all have posted their results yet.
The trouble is that it’s unclear whether this new policy of glasnost will affect students’ decisions about colleges—and therefore colleges’ behavior. So far, it seems that education consumers—students and parents—are more interested in a university’s reputation or the SAT scores of incoming freshmen than in these test results. Of the many college rankings and ratings available, including the popular U.S. News and World Report rankings, none use learning outcomes or student engagement as metrics to judge colleges.
Maybe there is a good reason why the leading ranking systems don’t use these measures. With so many tests, it’s difficult for students and parents to compare results between schools—or even to compare a school’s results from year to year.
When schools use the same test, comparing two schools can be valuable for future students. For example, looking at College Learning Assessment scores at both Fayetteville State University and North Carolina A&T shows that A&T students improve their critical thinking skills during their years in college more than do students at FSU. However, comparing outcomes on different tests is virtually useless for measuring learning outcomes, since the tests are scored differently and different learning categories are tested.
Moreover, there is potential for schools to “game” the system. They can switch tests each year so that little comparison is possible. Or they can administer only half of a freshman-and-senior paired test, so that the actual progress of students is not visible. And they can also search for the test on which their students perform best.
The Voluntary System of Accountability, the College Portrait, and U-CAN should follow the UNC system’s lead by focusing on the provision of information that is both accessible and comparable. This could be done by insisting on just one test in each category, or by collating the results to make meaningful comparisons. If they did that, these clearinghouses could give students and parents more of the information they need to make better decisions about college.
Editor’s note: These are the tests used nationally to measure student learning outcomes. The statements below are based on descriptions made on the websites of the assessments.
- College Learning Assessment assesses students’ abilities to think critically, reason analytically, solve problems and communicate clearly and cogently. Scores are aggregated at the institutional level to inform the institution about how their students as a whole are performing.
- Measure of Academic Proficiency and Progress assesses four core skill areas — critical thinking, reading, writing and mathematics — in a single test..
- Collegiate Assessment of Academic Proficiency is a standardized, nationally normed assessment program from ACT, the organization that competes with SAT for high school aptitude tests, that measures students’ achievement levels on a group and individual basis.
These are the tests used nationally to measure student attitudes and engagement:
- Cooperative Institutional Research Program’s Your First Year College: assesses the academic and personal development of students over the first year of college by identifying features of the first year that encourage student learning, involvement, satisfaction, retention and success.
- Cooperative Institutional Research Program’s College Senior Survey: provides information on a broad range of student outcomes, including students’ academic and campus life experiences and students’ post-college plans immediately following graduation.
- National Survey of Student Engagement: measures how undergraduates spend their time and what they gain from attending college. Survey items on The National Survey of Student Engagement represent empirically confirmed “good practices” in undergraduate education. That is, they reflect behaviors by students and institutions that are associated with desired outcomes of college.
- The College Student Experiences Questionnaire: assesses the quality of effort students expend in using institutional resources and opportunities provided for their learning and development. Quality of effort is a key dimension for understanding student satisfaction, persistence, and the effects of attending college. “The more students engage in educational activities, the more they benefit in their learning and development.
- The University of California Undergraduate Experience Survey: solicits student opinions on all aspects of the UC experience. Students evaluate such things as instruction, advising and student services. It also measures student behaviors—their study habits and how they use their time.
Correction: In an earlier version of this article, Educational Testing Service’s MAPP was incorrectly identified. It is the Measure of Academic Proficiency and Progress.