Where the College Scorecard Has Gained Traction So Far — and Where It Hasn’t

The Department of Education’s College Scorecard is, at its core, a simple college-search tool that highlights key data points with a few clicks. The most-talked-about piece: the average salary of students who attended a college, 10 years after they first enrolled.

Each college’s profile also features its average annual cost, academic offerings, and graduation and retention rates, among other things. (Only students who receive federal financial aid are included in the data.)

Department officials released the revamped tool last September, as a replacement for the Obama administration’s controversial college-ratings plan. When they did so, they said the new Scorecard would ensure that students, particularly those from low-income and first-generation backgrounds, had access to reliable information.

They also made available for download all of the institutional data used in the tool, so that scholars could use the information to conduct research and third-party vendors could develop apps and other products to assist prospective students with their college searches.

Many experts assert that the Scorecard’s mere existence — and the department’s decision to release the data within it for outside use — is a significant step forward for accountability in higher education. This month, the department updated the tool with a new year of data, which signals, they say, that the department is committed to keeping the Scorecard relevant even after President Obama leaves office.

The tool has answered some critical questions in its first year, says Robert Kelchen, an assistant professor of higher education at Seton Hall University. For one, he says, it created a broad snapshot of whether students are able to at least start paying down their loans and whether they’re making a salary that’s above the federal poverty line. Mr. Kelchen has incorporated the earnings figures and student-loan repayment rates into his “Best Bang for the Buck” rankings for Washington Monthly’s College Guide.

Nearly 1.5 million unique users have visited the Scorecard online over the past year, according to the department. But so far, it’s been tough to quantify whether the information in the Scorecard is reaching prospective college applicants, or if they understand what they’re looking at. Experts also caution that the Scorecard’s emphasis on future salaries as the central value of a degree could be framing conversations about college performance too narrowly.

Who’s Using It?

Several college-counseling experts who work with high-school students say most of their students don’t know about the Scorecard. And if they are using the Scorecard, it’s generally with a counselor or parent looking over their shoulder. Most aren’t seeking out the tool on their own.

“There are not students coming to us and saying, ‘Hey, I was checking out the College Scorecard yesterday,’” says Greg Johnson, chief operating officer of the Bottom Line, a nonprofit group that helps low-income students navigate the college-search process.

When students search for colleges online, Mr. Johnson says, they tend to end up on individual institutions’ websites. Bottom Line advisers will often direct students to additional resources on sites they’re already familiar with, like the College Board, where they made accounts before taking the SAT.

While the Scorecard is a “quality tool” that consolidates information about different colleges on one site, Mr. Johnson says, students don’t necessarily understand how to read the data on costs and outcomes without a trained eye to help put them in context.

The Scorecard seems to have gained some traction among college advisers. Kim Cook, executive director of the National College Access Network, says many counselors use the tool to compare colleges when they meet with students. Some assign the Scorecard as homework and ask students to research a handful of institutions before an advising session.

Not only does it spark important conversations about what students should be looking for in a college, Ms. Cook says, but it also helps the adviser understand what about a college appeals to that particular student.

David A. Hawkins, executive director for educational content and policy at the National Association for College Admission Counseling, says the organization plans to survey its members in the near future to get a better idea of how many counselors are using the Scorecard and what they’re finding to be most useful about it.

There’s been at least one notable effort to quantify how the Scorecard might be influencing students’ choices: a College Board study completed earlier this year. It found that the Scorecard made a difference, but only its earnings data, and only for some students. The impact was concentrated among well-resourced students, raising concerns about equity.

That research didn’t try to gauge students’ level of interaction with the Scorecard, says Michael Hurwitz, senior director at the College Board and one of the authors of the study. Perhaps low-income students are accessing the tool as often as other students, but feel constrained in where they can apply, Mr. Hurwitz says. “We don’t know if this story is about access to information, versus the ability to act on information,” he says.

Mr. Johnson, of the Bottom Line, says salary information isn’t a focus for many students who are still in the college-search process. “From our standpoint, we talk about income and earnings when we narrow the conversation — what do you want to major in, what do you want to do when you graduate,” he says. “Oftentimes that doesn’t happen until they’re already enrolled in college,” at which point they’re unlikely to return to the Scorecard.

Hoping for Improvements

The department, for its part, said this month that it plans to incorporate salary data by academic program into the Scorecard.

“That needs to be done with some delicacy,” says Jordan Matsudaira, who helped develop the Scorecard while serving on President Obama’s Council of Economic Advisers. What an engineering student makes is primarily relevant for students who graduate, he notes, and attrition rates in that major are high at many colleges.

But Mr. Matsudaira, an assistant professor in Cornell University’s department of policy analysis and management, hopes the information will help students go into college “with eyes wide open.” “Students who go into teaching will make that choice knowing that they’ll make a lot less than an engineer, but they’ll make the choice happily,” he says.

Jessica Thompson, policy and research director at the Institute for College Access and Success, says she remains concerned that the salary data don’t account for what colleges are actually doing to contribute to the variations in outcomes. Salaries could also be influenced by the economic realities in a particular region and labor-market discrimination based on race or gender, she says.

“Where do you draw the line,” she asks, “where you say, ‘This clearly says something about the institution,’ versus other things that affect earnings?”

Mr. Hurwitz, of the College Board, would like to see what past students are earning at different snapshots in time. He’d also like to see the Scorecard include a range of earnings by percentile — as U.S. News & World Report does with SAT scores in its college rankings — instead of just the median.

Ultimately, Mr. Kelchen says, outside developers will probably be able to create better products and apps with the Scorecard interface and data than the federal government. “That’ll end up being far more important than the College Scorecard’s actual public-facing website,” he says. More than 600 developers have accessed the Scorecard’s interface and data, according to the department.

College Abacus, an online financial-aid comparison tool, is one site that has incorporated the Scorecard data into a search that allows students to look at debt-to-earnings rates across institutions. A mobile app called Schoold, which allows users to calculate their expected salary at more than 3,000 colleges based on their major, also made use of the data.

In research groups that he is involved with, Mr. Matsudaira says, the Scorecard’s information “definitely pervades the conversation.” He hasn’t yet seen any completed studies that used the data, which he attributes at least in part to the lengthy timeline of academic publishing. But he has noticed academics referencing the Scorecard’s metrics to offer broad background about a particular institution.

Also high on student advocates’ the wish lists: a way to integrate more subjective student-learning outcomes into the Scorecard. Labor-market outcomes should not be the only measure of a college degree’s value, they stress. “This was a debate that made working on the Scorecard very difficult,” Mr. Matsudaira says.

It’s tough to quantify things like well-being and life satisfaction, he says, but researchers should try. Such information would bolster the Scorecard’s influence as an accountability tool, he says. “The first step,” he says, “is really agreeing on what the next couple of student outcomes are that should be used to hold institutions accountable.”

Sarah Brown writes about a range of higher-education topics, including sexual assault, race on campus, and Greek life. Follow her on Twitter @Brown_e_Points, or email her at sarah.brown@chronicle.com.