Fact-Checking the College Scorecard

An online database was supposed to make college more transparent. Instead, Reed students found, it’s sowing confusion.

How does a high school student decide what college to go to? If you think choosing a school based on earning potential is the best way to decide, students in Reed’s Math 343 class have some data to share with you. As part of their coursework with Prof. Hank Ibser [math], Reed students Wolfgang Brightenburg ’23, Robin Hart ’23, Bhav Khurana ’23, and Tina Qin ’23 found major methodological flaws in College Scorecard, a common online tool breaking down students’ annual income according to where they attended college. Their findings raise complex questions about what a meaningful college experience really is—and how to measure it.

“As current college students ourselves, we recognize the myriad of factors that went into our own respective decisions to pursue higher education at Reed College in specific,” wrote the students in the report on their findings, which concluded that College Scorecard failed to take a comprehensive look at the many ways of assessing student success. “This is different from the motivating factors of our friends from high school, who went to other places, or did not go to college at all. The merit of an education cannot be reduced to a paycheck, nor can a person’s skills.”

And even if all that mattered to you was a paycheck, College Scorecard wouldn’t be the best way to measure it, they found. The website, introduced under President Obama, was intended as a way for prospective students to assess individual colleges’ offerings based on student outcomes, but by focusing on earning potential. There’s just one problem: it doesn’t do that. At least not in the way it’s meant to.

In collaboration with the Center for Life Beyond Reed, the Math 343 students paired College Scorecard with a second dataset known as the HEDS (Higher Education Data Sharing Consortium) alumni survey to fact-check College Scorecard’s breakdown of alumni earnings. In Reed’s case, they immediately identified a $10,000 gap between graduates’ self-reported earnings and those listed on College Scorecard. Additionally, only students receiving federal aid were included in College Scorecard’s data. “That’s a really important population,” says Hart. “I receive federal aid myself. But I know that for Reed that’s about half the student body, and so over half of people are left out of this supposedly really representative database of what college outcomes are.”

That wasn’t all. The numbers might have been more accurate, says Ibser, if they’d included students across all majors at every school surveyed. But they didn’t—until recently. College Scorecard reported alumni earnings by major, and Reed’s valuation included only English majors, said Mike Tamada, director of institutional research. This was due in part to privacy concerns: some majors have so few participants that one could, in theory, find out someone’s salary information based on the data. When, in January of this year, the College Scorecard changed its practices and all majors were included, it reported Reed’s median earnings $33,000 over prior reports. 

Focusing on alumni income five years after graduation was also constraining. The numbers poorly account for one very specific reason students may not have high earning potential after finishing college: many go on to grad school. This is especially true at Reed, which is ranked fourth nationally in producing doctoral degrees across all disciplines among undergraduates. Although College Scorecard excludes earnings of graduate students, this means an important population of highly sucessful Reed graduates is not accounted for. Ibser says, “Reed students tend to catch up later on.” 

And of course, money isn’t everything. Using the HEDS survey, Ibser’s students keyed in on alternative metrics more likely to indicate the quality of a college experience, digging into graduates’ satisfaction and the reasons behind it, and things like ongoing engagement with their colleges in the years after completing their education.

From this analysis, a more holistic picture of college success emerged, one that accounts for varied, qualitative experiences that stretch far beyond tracking income in the five years after graduation. “For Reed alumni in particular, faculty research and professors encouraging critical thinking were the most influential on their satisfaction with the school,” the students wrote in their report, saying that attempts to evaluate students’ experience should factor in school offerings like these.

Money also wasn’t the only thing that students cited when they had a positive college experience. “We found, essentially, that satisfaction doesn’t predict your income, which we thought was really interesting,” said Hart. “Someone could absolutely love the college they went to, but maybe not make a lot of money.”

This might fly in the face of normative definitions of success, but it wouldn’t be the first time a group of statistics-minded Reedies poked holes in the higher-ed status quo. In 2019, Reed students deconstructed the methodology behind U.S. News and World Report’s annual college rankings, and found that Reed, which fell precipitously down the list after refusing to participate in 1996, was inaccurately ranked. Though U.S. News rejected the conclusion, it bumped Reed up the list. The students’ research even attracted the attention of Malcolm Gladwell, who dedicated a podcast episode to the rankings, finding the list was rooted not in sound methodology but social stratification.

Developments like the Varsity Blues college admissions scandal further puncture accepted norms about what makes for a good education, and cultural tendencies to value qualities like prestige over more idiosyncratic experiences tailored to individual student needs. Hart said they’d grown up on the receiving end of messaging suggesting that Ivy League schools were the best, but that old ideas equating quality with elite schooling were shifting.

“I feel like the more I get out into the world and the more I engage at Reed and beyond, I hear people pushing the envelope in terms of what a good school is,” they said. The change even showed up in their own experience applying to graduate school, a process driven not by an end goal like attending a revered institution, but by attending the right one for their specific educational goals. It wasn’t about applying to Harvard, they said. “It’s more like, ‘Well, is there anyone at Harvard that you’d want to work with? What are the faculty members there doing research on?’ And I think that’s a really good frameshift to be thinking about.”

Tags: Academics, Alumni, Cool Projects, Editor's Picks, Institutional, Life Beyond Reed, Students