The Absurdity of Ranking Colleges by Graduate Salaries

Jordan Weissman has moved from the Atlantic, and is now covering economics at Slate. He has a post up provocatively titled (it is Slate, after all) “What College Will Leave you Poorest?” which covers the Payscale college salary rankings, in which Payscale.com ranks colleges based on the cost of the college offset by the reported salaries of their graduates. In other words, a return on investment for each college. What a grand idea.

Ok, I think it is a terrible idea for a lot of reasons, but in efforts to keep this blog post under 1000 words, I am going to focus on one seemingly minor issue, which to me invalidates this entire ranking system. To illustrate this, humor me first on this brief tangent into basic statistics and my own education in cognitive psychology.

I was a history of science major, so I only took a few psychology classes as an undergraduate, and these were all in the clinical or neuroscience area. So when I got to graduate school in cognitive psych, I had to catch up to learn some basic concepts. One of these didn’t occur to me until I took a class on “Advanced Research Methods in Cognitive Psychology.” We were discussing the merits of how to graph and analyze different kinds of data, and we were talking about time. We reviewed the different types of measurement scales. When a measurement is on an nominal scale, there is no order to the results of the measurement at all. These as categories, and it would make no sense to say that they are in order, or that the differences between them can be interpreted in some quantifiable way. Gender, race, religion, types of fruit, etc etc.  The next type of scale is ordinal. In this scale, the results are ordered, but the difference between the orders doesn’t necessarily make sense. This could be rankings, like eldest, middle and youngest child. We can put them in order, but the difference between the eldest and the middle is not necessarily the same as the difference between the middle and the youngest. Many likert-style (“neutral, ok fine I guess, agree, strongly agree, DEAR GOD YES”) survey items are on an ordinal scale. We can say that someone who chooses option 5 agrees more than option 3, but is the difference between 5 and 3 the same as between 3 and 1? Probably not, and we should analyze our data as if that is the case. The next scale is interval scale, in which the scores are ordered, but also the differences between ranks are equal intervals. For interval scales, the difference between scores of 8 and 10 is the same as between 2 and 4. Time is often given as an example of an interval scale. Finally, a ratio scale has a true zero point, where a score of zero is real and meaningful, such as wealth or income.

We were talking about these scales, and the issue of measuring reaction time came up. It seems like it could at least be an interval scale, if not a ratio. But as we were talking, the professor reminded us that even if the physical scale of time was an interval scale (the difference between 1 and 2 seconds is the same as the difference between 20 and 21 seconds) this did not mean that the psychological dimension of reaction time was also an interval scale. This struck me as an interesting way of thinking about a central insight in scientific psychology, since its inception. We have to be careful in how we treat our measurements, taking into account that the psychological scale does not always map cleanly onto the more physical or concrete scale (whether it be light, sound, time, or money).

Here’s my contention: while money might be an interval scale, salary is not.

Sure, a dollar is a dollar is a dollar, but saying that a mineral engineer makes twice what an artist does, and therefore this particular art school isn’t worth it, that just seems absurd, but it is exactly the logic that their ROI tanking system encourages, and that Weissman’s article adopts in using language like “to be blunt, these schools make students poorer.” No they don’t. I imagine that most students entering art school are fully informed (by parents, teachers, classmates, strangers in the grocery store) that their choice is disastrous and they will never find a lucrative job doing art. They choose to pay for training anyways. Is the art school making them poorer?

The salaries of graduates from Harvey Mudd (#1), the Colorado School of Mines (#11), UNC-Asheville (#1306) and Maryland Institute College of Art (#1301) are not comparable and should not be compared as if they have anything to do with the schools. Instead, this scale is not just a proxy for which careers graduates choose upon leaving the college, but even what career they were interested in upon entering that particular college.

A given measurement scale supports a certain kind of statistical analysis. If one has an nominal scale, it makes no sense to make ordinal claims of greater than or less than (are oranges less than apples, maybe Honeycrisps). If a measurement is on a ordinal scale you can report frequencies, but it makes no sense to analyze means or standard deviations (what is the mean and standard deviation of mild, medium, spicy and Native Thai?).

Perhaps the payscale people should consider replacing their college rankings with a big simple headline that says “Engineers make more than artists.” and “People who apply and are accepted by Harvard are going to make more money than people who apply and are accepted by Morehead State.”

I could go on and on. No matter how seemingly sophisticated these rankings seem, they have not found a way to disentangle a school from the students who choose to attend. Just because we have found a number (salary) that allows us to compare Williams (#16) with Virginia Tech (#69) doesn’t mean that those 53 spaces mean anything at all.