Measuring the performance of colleges—and holding them accountable for their performances—is an unsolved problem. Such measurement was less essential, or at least less urgent, in the past. But today, with rising costs, massive student debt, and a weak job market, taxpayers are starting to demand that public colleges justify their public funding.
But just because performance measures that look adequate are put in place does not mean they are meaningful.
For example, for the North Carolina Community College System (NCCCS), the State Board of Community Colleges has set accountability standards for close to 30 years, but only recently have the measures received serious attention.
This new seriousness has brought an increasing focus on “performance funding.” Performance funding combines the measurement and accountability functions, using a set of goals that public colleges are expected to meet to determine their appropriations.
In 1998, the North Carolina General Assembly required the State Board of Community Colleges to review its performance measures and implement a performance-based funding plan. Under the original plan, colleges that met all of the measures received bonus funding. Colleges that underperformed merely had to submit an action plan to the board, but were not penalized further.
In 2013, the general assembly changed the structure so that colleges would be funded proportionately by how well their students do. Colleges that underperform on a measure receive no funding for that measure, while colleges that exceed expectations receive the maximum amount for the measure.
The legislature set aside $24 million for performance-based funding out of a $1.3 billion community college system budget in 2014-15. The NCCCS now has eight measures:
- Basic Skills Student Progress: measures the progress of unprepared students toward the minimal skills required for “employment and self-sufficiency”
- GED Diploma Passing Rate: the percentage of who pass the test that is an alternative to a high school diploma
- Developmental Student Success Rate in College-Level English Courses: measures the progress of remedial English students
- Developmental Student Success Rate in College-Level Math Courses: measures the progress of remedial math students
- First Year Progression: the percentage of credential-seeking first-year students who pass at least 12 credit hours
- Curriculum Student Completion: records six-year completion rates
- Licensure and Certification Passing Rate: measures the percentage of students who pass licensure and certification tests
- College Transfer Performance: follows the success of transfer students after one year at the transfer institution
These eight appear to be reasonable measures for judging a college’s performance, although an employment measure is conspicuously absent.
But that does not mean an employment measure was always missing. Originally, there were 12 measures. Two of those 12 were directly related to employment: “employer satisfaction with graduates” and “employment status of graduates.”
However, both measures were unreliable, according to Bill Schneider, the community college system’s vice president for research and performance management. He told the Pope Center that both measures had unrealistically high success rates, as they failed to include many people who were not working, used inconsistent survey methods, and cherry-picked which employers to survey.
So they were dropped. But the need remained. Community colleges have multiple missions; one is to prepare students for transfer to four-year schools, another is to train workers for skilled jobs. North Carolina’s community college system defines success for vocational students as “leaving with a job-ready credential in a global economy.”
In response to the increasing focus on whether graduates find employment, the NCCCS is trying to add a new measure to its performance funding model based on graduates’ incomes. A new “earnings gains” measure was approved by the State Board of Community Colleges on August 21, but still needs the legislature’s approval.
There are other problems that are inherent to performance-based funding. For example, it is difficult to tell whether the schools with better-performing students did well because of the education they provided or because their students were better prepared.
While several states utilize some kind of employment metric, most do not. Even fewer have the particular measure North Carolina settled on, an “earnings gains” measure. While NCCCS should be lauded for trying to add a meaningful measure to their performance standards, the specific measure crafted by NCCCS leaves a lot to be desired. The metric, using wage records from the North Carolina Department of Commerce, will measure just a narrow group of students. That group consists only of graduates of vocational programs who have full-time jobs for the two years before enrolling and for the two years after graduation. The average age of students measured was 34, six years older than average for the community college system.
Among the groups excluded from the measure are students in academic programs, and students who are part-time workers, unemployed, or work in another state two years before or after community college.
Stanly Community College president Brenda Kays, who headed the committee that produced the measure, admitted that it is not perfect and may need to be adjusted or added to in the future. Still, she touted the initial results from the cohort that graduated in 2011-12. “Every college had positive gains, and so that tells us that the career and technical experience that these students had increased their earning potential,” she said.
The problem is that the measure does not necessarily do that. How are we to know that the students would not have increased their earning potential without the help of the community college? After all, consistently employed adults in their late 20s or 30s are likely to see significant income gains.
It may be difficult to keep track of all students, especially those who leave the state. But if you confine the group you are measuring to those who are successful by definition, it should come as no surprise that they are successful. What of the students who fail to secure full-time employment in their field upon completion of their program? Removing them from the mix renders the measure inadequate.
Even so, a national higher education metrics expert with which the NCCCS consulted, Nate Johnson of Postsecondary Analytics, told the Pope Center that earnings gains are a good measure and that by using it, North Carolina is “putting itself at the leading edge” of higher education systems seeking to measure workforce data. For example, he said, Florida simply provides average wages without any context or added value. But he also said that measuring workforce outcomes is “not a well-developed science yet in higher education.”
Of course, Johnson could hardly be expected to criticize the new performance measure, since he was one of the consultants who helped NCCCS craft it.
North Carolina’s community college “earnings gains” measure may make the community colleges look good to those who do not look too closely, but it does not tell us much about how good they actually are. The system needs to go back to the drawing board; the new measure is more public relations than a true measure of performance.