Thumb on the PayScale

In my last post, I took issue with the PayScale college rankings, as well as with how economics reporters framed these rankings, citing their low calculated Return on Investment as evidence that these colleges “make” students poor. Jordan Weissmann has graciously responded to my critique. The Senior Vice President from PayScale itself has also responded in a comment on my post, defending the rankings as useful to high school seniors who later tell him, “I wish I had known this before.” I’m glad for the dialogue, and I think this issue connects to a larger debate on the value of higher education in general, as well as the relative value of different colleges, so I am going to keep the conversation going with a more extensive response.

Weissmann is not alone in his framing of PayScale’s rankings. While his headline at Slate reads “Which College Will Make You Poorest?” Derek Thompson at the Atlantic treads similar ground: “Which College—and Which Major—Will Make You Richest?” and later, “These U.S. Colleges and Majors Are the Biggest Waste of Money.” Walt Hickey goes with a more straightforward, but still uncritical view at 538: “Study Names Colleges with Best Return on Investment.”

My argument was first that salaries from art schools and engineering schools are not directly comparable, since students are not often choosing between Harvey Mudd (#1) and Maryland Institute College of Art (#1301). Chris Chabris mentions this point in his post criticizing 538’s coverage of the rankings. Second, I argued that PayScale and many journalists who wrote about the rankings, are overstating the role of the college in shaping its students salaries and understating other factors in what determines a students future salary, such as the role of selective admissions, constraints in the labor market and choices of the students themselves. Weissmann acknowledges various limitations (including selection bias) of PayScale’s methodology, but he doesn’t see these as reason enough to avoid phrasing like: “Which college will make you poorest?” or “Yes, the art school really is making students poorer if it delivers a negative ROI.”

The subhead for Weissmann’s recent defense of ranking colleges based on graduate salaries reads: “Academics might not like it, but schools should be held accountable.” Since I am the only named academic in the article, and my quote is framed as “Like many in higher education…” I can assume that Weissmann sees my criticism as based on my identity as an academic. True, I am an academic. I do feel solidarity with my fellow providers of higher education, whether they be at the University of Virginia (apparently THE best value! Wahoo!) or at arts schools like Maryland College of Art and Design, or Historically Black Colleges and Universities like Shaw and Fayetteville State University. However, my criticism of this particular set of rankings owes more to my training as a social scientist (and as a psychologist in particular) than to my allegiance with all institutions of higher education. Here’s why:

Claim a Cause? You’ve Got to Earn It

All this writing about salaries reminds me of another kind of earnings. When a scientist such as myself reads a headline with a claim that something causes something else, I ask whether the speaker has earned the right to make that claim.  Can the data cash the check that the causal claim makes? A few examples: “Smoking causes cancer,” “miracle diet causes weight loss,” “stress causes ulcers,” and “vaccines cause autism.”  While many educated people may start with a skeptical “correlation does not equal causation,” most scientists (especially social scientists) know that not all correlations are created equal, some studies are better than others. For a long time, the only good data on the harmful nature of smoking was epidemiological: those who smoked tended to have worse health than those who didn’t. Notice the lack of a causal claim, a study might conclude that a heavy smoker is 50 times more likely to develop lung cancer than a non-smoker, but it is still possible that smokers in general lead more stressful lives, exercise less, and have less healthy diets. In this case, rather than causing cancer, smoking is merely another marker of an unhealthy lifestyle. More recently studies have narrowed in on the biological mechanisms of smoking’s harmful effects on the body. What is it precisely that causes the harm? Is it the nicotine? the tar? What are the cellular mechanisms of the carninogens in cigarettes? These questions now have much more specific answers based on carefully controlled experiments, and even though there still seems to be some randomness to the overall pattern of carcinogens causing cancer (yes, some people can smoke for 60 years and not get cancer), we know a lot more about the biochemical effects of smoking.

But even without research into the cellular mechanisms, the epidemiological, correlational research was able to earn its causal claims. One landmark study compared British male doctors who smoked with …. British male doctors who did not smoke, and followed them for over 40 years. Keeping many other variables constant, this study still finds incredibly harmful effects of smoking.

A similar search has been conducted for a connection between vaccines and autism. Many children exhibit the first signs of autism spectrum disorders at around the same time they are vaccinated. Yet this is a misleading correlation. Epidemiological research would seem to be impossible. How can we compare levels of autism in non-vaccinators and vaccinators if everyone vaccinates? Except now that the fear has spread, we can. Populations that don’t vaccinate their children have the same incidence of autism as those populations who do vaccinate. Amazingly enough, however, populations who choose not to vaccinate DO have higher risk of an outbreak of whooping cough. A long search has concluded that vaccines do not cause autism, through both epidemiological work, as well as the search for biological mechanisms of harm.

Which brings us back to PayScale and the economics reporters. Have PayScale and these reporters earned their causal claims? Are these rankings actually useful to high school seniors deciding where to go to college, as Barnaby Dornfman comments on my previous post, or merely perceived to be?

I remain deeply skeptical.

The Median is not the Message*

*with apologies to Stephen Jay Gould

There are (at least) three classes of factors that could predict a high school student’s future salary. First, a student’s individual characteristics such as preferences, abilities, motivation and even demographic variables such as race and gender affect their future salary. Second, that person’s education and training of course influence what they earn. This is where college fits in, but it might also include high school preparation as well as out of school preparation. Third, students future salaries are dependent on the labor market they enter as they leave college. As many unemployed or underemployed PhD’s are now discovering, someone can have great achievements in the other two factors, but if there are few to no jobs, one’s salary is severely constrained. The PayScale rankings understate these factors in comparing salaries of graduates to the median salary for someone who has only a high school diploma.

The reporters who have covered the rankings, and to a lesser degree, PayScale itself, make two critical errors that undermine their causal claims. First, they gloss over the impact of student and labor market factors on salaries. Second, they also understate how inseparable colleges are from the students who attend them. The rhetorical effect of the apparently straightforward reporting on these rankings are a pair of bold pronouncements “Get thee to a college that doth begin with H-A-R-V, and drink thy fill from the trough of increased economic outcomes” and “But lo! if thy find thyself in Pellville Directional State University, fly fly away, and become a manager at Ye Olde Chipotle, for thy riches will grow more from avoiding useless, expensive learning than from seeking it.” The truth is a lot more complicated. Aggregate salaries depend far more on student characteristics present before they enter college and the labor market that they choose to compete in as they exit college, than what differentiates the colleges that choose to admit them.

There are a few ways to see this in the rankings themselves. I’ll admit that I don’t know much about the Colorado School of Mines, but what is making their students rich is not necessarily the school itself, but the students’ decision to become engineers in Colorado (or miners, maybe Clementine’s dad had it right after all!). When PayScale compares their salaries to what a median high school graduate makes, this isn’t the best comparison, given that these students have already decided to become engineers, it seems with a focus on energy and environment. Likewise, for the number 1 ranked school, (Harvey Mudd), before Harvey Mudd does any educating, it has selected certain students to admit, and some subset (approximately 30%) of those students have made a decision to attend a small engineering school in California. Of these students who choose to attend Harvey Mudd, 14% qualify for Federal Pell Grants and 46% qualify for any federal student loan aid. For these students, the average total federal student loan aid is around $6,400.  I am certain that Harvey Mudd is a fantastic student experience, but I think it is fair to say that they enroll a student body that is privileged, motivated, ambitious, and focused on a high salary occupation from the moment they arrive on campus. These students have decided to become scientists or engineers before they entered Harvey Mudd, and it doesn’t make much sense to compare their return on investment to what the median high school graduate makes.

Perhaps to defenders of the value of these rankings, they are fine with the advice boiling down to: Want to make more money? Become an engineer, not an artist. Here is hard data on how poor you will be as an artist. Going to college won’t help you if you choose a low paying career. As I said before, I am dubious of the value of such advice, although PayScale Senior Vice President Barnaby Dorfman has experienced sending the PayScale rankings to many people who say “I wish I had known!” However, for the I-wish-I-had-known camp to improve economic outcomes by going into engineering, we need enough engineering jobs to accommodate increased labor supply. But this is not the case. There is no STEM skills gap. There is no labor shortage for STEM jobs. Perhaps a better way to improve outcomes would be to forgo college altogether, and, as Weissmann suggests, work in a Ford factory or manage a Chipotle. This assumes that the student choosing these jobs instead of college will make greater than the median salary of high school graduates, which Weissmann acknowledges may not be the case, due in part to “weaker skills.” Even though Weissmann and others briefly consider the other flaws in this model of salary prediction, I think it is worth it to drill down deeper.

The Worst Performers

What if we do this kind of analysis for Derek Thompson’s so-called “biggest wastes of money?” The students entering these colleges quite often already have a deck stacked against them, and not just in college choice. A student entering Shaw University, the oldest historically black college in the South, located in Raleigh, North Carolina, is making a hopeful investment in a college education. But comparing their future salary to the median high school graduate does not take into account the racial disparities in our current labor market. Further, as Raj Chetty and colleagues have shown, economic mobility is constrained not just by demography of the individual, but by geography as well. In his testimony submitted yesterday to the Senate Budget Committee, Chetty describes one pattern with this lack of mobility: “The first pattern we document is that upward income mobility is significantly lower in areas with larger African-American populations.” He also notes that if mobility is measured by a child’s chances of moving from having parents with the lowest quintile in income (poorest 1/5th of population) to themselves having upper quintile income, one of the “commuting zones” lowest in intergenerational economic mobility? You guessed it, Raleigh, North Carolina with 5%, considerably lower than San Francisco, San Jose and San Diego, California,, all with odds more than twice that. See the map he created below:

Chetty map

To connect geographical mobility to higher education, we must understand how far students are likely to travel for college.  Most students stay close to home. Look closely at this data from the ACT.  The median distance to college (for those who took the ACT) is 51 miles. But if you separate groups by test scores, those who score lowest, stay closest to home. If you score between 1 and 15 on the ACT, the median distance to college is 18 miles. The same pattern can be seen depending on parental characteristics. For students whose parents have no college education, the median distance they travel to go to college is 24 miles (parents with graduate degrees? median distance of 95 miles). Poor students are less physically mobile, which is influenced by, and contributes to, their lack of economic mobility.

Take My Advice, Don’t Try it Twice, if You’ve Got but 50 Cents

So the PayScale rankings would have you believe that students should choose higher ranked colleges, instead of those that “make them poor,” like art schools and HBCUs. The advice is not “Don’t be black, because structural racism will still hurt your progress” or “Don’t be poor in the south, because intergenerational economic mobility there is very low” or “don’t find value in art, because your society doesn’t support that,” but rather: “Watch out for these schools, they waste your money!”

Here is where I notice a difference between what I will call the neoliberal economic approach from a conservative one like Charles Murray’s.  Neoliberals are often confident in the power of greater information to improve equality and efficiency, whereas conservatives are openly pessimistic that all students are intellectually capable enough to complete college.  Here’s Charles Murray:

So even though college has been dumbed down, it is still too intellectually demanding for a large majority of students, in an age when about 50 percent of all high school graduates are heading to four-year colleges the next fall. The result is lots of failure.

Whereas Weissmann closes his original reporting of the PayScale rankings with this:

But it does reinforce why we so desperately need high-quality consumer information about higher education. In the past, the higher-ed lobby has stood in the way of allowing the Department of Education to track college graduates over the long term to keep tabs on their lifetime earnings—what’s known as a “unit-record system.” And as a result, we have to rely on less complete government surveys, or less-than-ideal crowdsourced databases like Payscale’s. As a result, some students are going into college financially blind, and they could be ending up poorer for it—literally

Notice another causal claim: the lack of data is what is causing students to go into college financially blind. What goes unsaid here is that if better data existed, then students and their parents would view that data and incorporate it into their model of how college works. Then, armed with better financial vision, they would made better financial decisions. This same confidence underlies the concern about undermatching, in which high achieving poor students choose to attend lower status institutions, when they would apparently have better outcomes at a wealthier, more prestigious college. This is a topic I have written about here and here. Sara Mayeux makes some excellent points about Yale’s tone-deaf and misguided efforts to address undermatching.

A common theme across the educational spectrum emerges. Yes, economic inequality has grown and states continue to disinvest in higher education, but a cheap, easy and effective way of improving students’ economic situation is through educating people who experience low workforce success (i.e. poor people) to help them make more informed decisions. The implication is clear: Low performing schools (whether colleges or urban public schools) don’t need resources, they need to be held accountable. Low performing graduates and their parents don’t need better economic opportunities, they need better data. Graduates of HBCUs don’t need laws and policies to counteract the persistent effects of racist housing programs, racist tax policy, racist hiring practices, racist underfunding of public K-12 schools, they need bourgious norms, invigorating moral culture, and better salary data. Apparently these students should be informed that an HBCU, which may once have been a proud part of familial identity, a cause of pride in black scholarship, a shining beacon of advancement through higher education, well, now big data science shows it offers a low ROI. Prospective students should be advised to go to a better school, or an engineering school, or work at a factory instead.

I know that Weissmann, PayScale, and everyone involved in this debate wants a better system of higher education in these United States of America. We all should want, to paraphrase Lincoln, a higher education system of the students, for the students, and chosen by the students. One that is accountable to the students. But in search of this goal, they are willing to forgive imperfections of their scale, convinced of the desperate need for accountability. But as a psychologist well versed in some of the shameful consequences of imperfect scales in psychology’s past, I am skeptical. Instead of progress and accountability, I see such rankings of colleges based on assumptions of privilege and status, devised by privileged tech boom MBA’s, for the software and energy engineers. I see this scale as one that does not encourage accountability, but merely reflects existing social inequities and blames society’s economic losers, just as existing social policies make their fates increasingly unavoidable. Don’t tell me you are holding colleges accountable, while holding your thumb on the scale.

About Cedar Riener

College psychology professor, husband, father.
This entry was posted in education, higherEd, psychology, science and tagged , , , . Bookmark the permalink.

22 Responses to Thumb on the PayScale

  1. dotkaye says:

    I looked at the ‘worst’ colleges and degrees, notice that quite a few of them are Education. These are probably primary school teachers in poor school districts. Yes, teaching kids is not remunerative, therefore choose STEM instead. We know the price of everything and the value of nothing..

    On Colorado School of Mines: you don’t choose to go there, you aspire to go there. It’s very very competitive to get in. As you note there is no meaningful way in which a poor black kid can be said to choose whether to go there. Their salaries are skewed by the recent fracking boom, so petroleum engineers walk out into six-figure jobs. Once we’ve fracked everything in sight there will be a glut of petroleum engineers and they will need to find something else to do.

    There’s a parallel here with Race to the Top in K12 education. Teachers are now ‘accountable’ for the test scores of their students, which are essentially a measure of the student family income. I don’t see any attempts to improve teacher working conditions, salaries or professional status, only a mean emphasis on punishment for all.

  2. Wow! That’s a lot to digest. I won’t respond to each point, nor will I attempt to represent/defend third parties who have come to their own conclusions about our data. But I do want to address the words you are putting in our mouths here at PayScale. For example, this statement is does not reflect our view: “The straightforward rhetorical effect of these rankings are a pair of bold pronouncements ‘Get thee to a college that doth begin with H-A-R-V, and drink thy fill from the trough of increased economic outcomes” and “But lo! if thy find thyself in Pellville Directional State University, fly fly away, and become a manager at Ye Olde Chipotle, for thy riches will grow more from avoiding useless, expensive learning than from seeking it.’”

    In reality, our data can be used to find great value at hundreds of schools, regardless of major. For example, this sort shows that you can go to a local state school, major in art, AND do well financially. You can even go to a state school outside your own, pay higher tuition, major in art and still do better than working at a high quality Mexican fast serve restaurant (OK…I do have a Chipotle bias):

    BUT, you have to pick among the schools with the right relationship between cost and opportunity creation.

    We’ve asked over 40 million workers about their salaries, where they went to school and scores of other career-related questions, in order to understand what is correlated with pay. We used a subset 1.4 million recent responses, from college graduates, and applied broadly accepted analytic approaches to produce our College ROI reports.

    Finally, I take exception to your last sentence: “Don’t tell me you are holding colleges accountable, while holding your thumb on the scale.” We don’t have a financial interest, like a butcher weighing meat, in promoting certain schools over others.

    • It doesn’t matter if you have a financial interest or not because that’s still what you’re doing. Your thumb on the scale is the extra weight you’re applying (i.e. dropping the full burden of accountability/return on educational institutions) that should be more evenly distributed among the factors Cedar writes about and which Tressie points out in her comments. I guess we could apply this ROI to companies–what return on investment do they get from their workforce? Who scores higher? People who pay less in salary and benefits of course. Obviously, state budget cuts are driving up tuition in state systems, this is well documented, but here, as always, it’s always the school with a ranking.

  3. Cedar Riener says:

    Thanks for reading, and responding, Mr. Dorfman. I can see how you would take exception to the last sentence. I was not thinking of PayScale for that, but Mr. Weissman, who says that this list is part of holding schools accountable (as I mentioned at the beginning of the post). I understand that you do not have a financial interest in promoting some schools over others, and that PayScale certainly doesn’t intend to hold a thumb to the scale.
    However, when you say this in your comment:
    “You can even go to a state school outside your own, pay higher tuition, major in art and still do better than working at a high quality Mexican fast serve restaurant (OK…I do have a Chipotle bias): BUT, you have to pick among the schools with the right relationship between cost and opportunity creation.”
    I think you again are overestimating the causal relationship between school and salary (btw, I love Chipotle too). It is not just the schools that create opportunity, but the students themselves, as well as networks of privilege and support that exist before, during and after the students attend the schools. An art major who goes to Columbia is not the same as an art major who goes to University of Wisconsin at Madison and comparing them as if they are the same students is, in my book, misleading. Many of the students who go to Rhode Island School of Design are interested in being designers, and if I forced them all to go to Wisconsin instead, but they all majored in design and moved back to a big city on the East Coast while they lived at their parents’ house for a few years while they took unpaid internships at big design firms, they might still become talented and well-remunerated graphic designers. It is entirely possible that their talent, particular interests and hard work is what is causing their success, rather than the school. This “opportunity creation” might not be possible for many students at Murray State, through no fault of Murray State.
    I am not saying this to be cynical about the role that schools can play in students lives. Even though I work at a liberal arts college, I am a strong advocate for more thoughtful and sophisticated consideration of student outcomes, both financial and academic. But I am wary of instruments that claim to measure a school’s (or individual teachers) effectiveness without considering the larger social and regional context of that measurement.

    • zekesoule says:

      Couldn’t agree more, I see a lot of students that transfer out of good schools and blaming the school for things that are there own fault. College is what you make it, your own your own and as an adult its up to you to network, meet the right people, figure out what you want to do and go after it. I mean, there’s nothing wrong with transferring schools or even dropping out, but kids shouldn’t blame the school for it (It was probably there decision to attend the school in the first place!) Haha

  4. tressiemc22 says:

    Great post and analysis, Cedar. Here’s the thing, folks with this message about college ROI do not care about the question or the social science for the most part. The surveys and analysis do what they are intended to do: confirmation bias as clickbait. That may sound cynical but at this point methodologists and functionalists (the type of which I’d divide those to whom you are responding) should be well aware of issues like dual labor markets, history, enclosure, and differential mobility. To the extent that they eschew these issues is willful ignorance.

    Having said all of that, I wish firms like Payscale that have the resources would cover the dual pillar of job matching, ie weak private sector demand, depressed wages, and declining corporate investment in human capital development. Alas, that’s what we wacky, untrustworthy academics are stuck doing. The work may not make for equally sexy headlines but it matters.

    • Cedar Riener says:

      Thanks for commenting, and sorry for my delay in replying. I think about your second paragraph too. I started to think that it might be interesting if a set of highered sociologists and scholars like yourself came up with a set of college rankings that took all of this into account. But then I reminded myself that the point of all the research is often that if you take all of these issues into account, you’d be led to the conclusion that a set of rankings are not the best way to depict the patterns in the data. Sometimes things don’t belong on a single ordered list, and colleges might just be one of those things.

  5. Pingback: Yes, Let’s Talk ROI and the Myth of ‘Accountability’ | Sad Iron

  6. Dr. Riener, I emphatically agree with the following statement: “It is not just the schools that create opportunity, but the students themselves, as well as networks of privilege and support that exist before, during and after the students attend the schools. An art major who goes to Columbia is not the same as an art major who goes to University of Wisconsin at Madison and comparing them as if they are the same students is, in my book, misleading.” Many people who have criticized our ROI data focus on the big difference, e.g. top of a list vs. the bottom. However, there are more interesting insights in the marginal differences. Many students do have the opportunity to get into a range of schools, e.g. different state schools, which can lead to choice between in-state tuition locally and out-of-state, which may also be local, but over a close border. This data can help maximize the value of that choice and highlight differences in the relative “network(s) of privilege and support” that a student may choose to join.

    Also, the converse is true, it’s also not just the students themselves. Clearly, there are differences between schools: brand value, teaching quality, facilities, etc. For those who simply reject the notion of measuring schools, this still amazes me. All schools put a price tag on the services they provide. AND, they increasingly price-discriminate based on some calculus of the value of one giving aid to one student vs. another. Almost all schools grade and rate their students and ultimately give them a diploma for meeting certain measured qualifications. Society should also measure those same institutions in as many ways as possible.

    Our data is, admittedly, just a start. In time we’d like to be able to better measure increased economic opportunity against inputs, including tuition, student debt, endowment, and corporate, state and federal funding. And, non-economic benefit, such as job satisfaction, job meaning and other non-economic dimensions. Today we do a bit of this: and

  7. awax1217 says:

    I read your blog with interest. I have often wondered about the pay scale and going to college. I think back on my experience. I have a B.A. and a Masters. Education and History majors. In reality I learned little. I had to have the certification to teach but in the classroom is where I really learned. You can teach to cook but get in the frying pan teaches you what heat is about.

  8. fungalspore says:

    Very interesting piece of writing, thanks! I appreciate the amount of work and research you have put into it. I studied arts. My primary motivation was not to make a lot of money and I am not surprised that I do not. There are good arguments against poor quality schools offering low-grade courses to poor people who spend the little money they have pursuing an evanescent dream that education will improve their lot. But education is big business. There is something structurally dishonest about these league tables, as though education were not gilded and sold by the same people who are shuffling themselves to the top of the heap. Blowing the horn for Harvard is like extolling your local organic delicatessen, when you have invested in a society that allows Walmart to prosper. Maybe I am just a fluffy Romantic avoiding economic realities!

    • Cedar Riener says:

      Thanks for commenting, interesting metaphor. Yes, “league tables” is one way of putting it, it fits into this unfortunate narrative where everything is a competition. I have been known to suffer from occasional fluffy Romanticism myself.

  9. lmarks04 says:

    It is true that salary is not the only indicator of a good college. But it does count for something. That’s for sure. I agree with you that they way they calculate these rankings are suspicious at best.

  10. Great analysis, college is what you make out of it!!

  11. Pingback: Sucking At the Federal Trough (Update) ~ The Body PolitiK Krew | Andy Kaufman's Kavalkade Krew Featuring The Wandering Poet

  12. austinspare says:

    I’m not going to comment on the writing; your graphs make it clear that the “American Dream” is all but dead in the Bible Belt.

  13. Pingback: On Controversial College Speakers, Debating and Critical Thinking | Cedar's Digest

  14. Frankie Bow says:

    Reblogged this on Frankie Bow and commented:
    Here’s a great post on college rankings. The thing is, we already know that universities can’t magically bring back the mid-tier jobs that have been lost to productivity increases and outsourcing. But we keep blaming the universities (and in the case of state run institutions, legislatively micromanaging them) because it’s something we can do.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s