I had a few thoughts about the big teacher quality and VAM study that came out today that I wanted to share before they float away.
My thoughts are less about the methods of the study itself and more about how it moves so quickly from the econometrics to the policy, and how the journalist presents it in the New York Times. Bruce Baker (@schlfinance101) says that there is a lot of interesting data here, and I look forward to reading his take on it, but I didn’t feel this article fairly presented the context and limitations of any study of this sort.
I haven’t had time to digest the full study, although I do want to read it. But let me start with something that may be a bit surprising to some of my readers. I don’t have a problem acknowledging the following:
1) Good teaching helps students learn, poor teaching does not help, and can even hinder student learning
2) Teaching is to some extent a stable trait, at least it seems so in our current school environment
3) Some teachers are better than other teachers
4) Some of this variation in teaching “ability” is can be reflected in student test scores, and in long term student and adult outcomes
Teachers matter. I wouldn’t be a teacher if I didn’t feel this to be true. As a college teacher, on one hand I (every day) confront evidence of my powerlessness in the face of student circumstances. On the other, I have had a few students tell me how I have changed their lives, and I (sort of) believe them. I certainly think that some of my teachers changed my life.
I am a teacher and I try to improve. But I know that there are some people who start out better than I did, improve faster, and will always be superior. Jeannette Norden is likely a better teacher than I will ever be. But despite that truth, I don’t find competitive teacher rankings by score (such as VAM) motivating, or acknowledge that they are the best way to measure my success.
I am adamant that this approach, as reflected in the quotes in the article, is toxic and corrosive to good teaching and to improving education in our country.
Rant about the NYT article
First: “That test scores help you get more education, and that more education has an earnings effect — that makes sense to a lot of people,” from Robert Meyer, not an author, but another teacher quality researcher.
When scientists talk to the media, they often face a choice between scientific precision and being clear to the layman. It is difficult to convey the uncertainty, precision, context, and humility of a scientific finding to a lay audience, which is why many scientists don’t even try. In explaining my dissertation research, I could say “Hills look steeper when you are sad.” Or I could present it in any of the following ways:
Sad people overestimate hill slant.
Sad people overestimate hill slant relative to happy people (who also overestimate hill slant).
People who listened to sad music, or wrote a sad personal story, reported the hill slant to be steeper than people who listened to happy music, or wrote a happy story.
People who listened to music that had made them sad in past studies, and made them say that they were sad in a post-experiment questionnaire (even though they didn’t realize that the point of listening to Mahler’s Der Kindertotenleider was to make them sad) estimated the slant of an 8 degree hill to be on average 23 degrees (with a standard error of around 1.4 degrees) using a verbal measure of slant as well as a visual matching device…
What? Where are you going? Oh, that won’t fit in the 20 words you have for describing my study? Ok. How about “hills look steeper when you are sad”?
But, back to this quote. I balked at this quote, because I thought, test scores help you get more education? Really? This is treating the measurement (the test score) as the causal agent, instead of the thing the test is measuring. I did well on the SAT (and the CTBS for that matter) but did those test scores help me get more education? No. They reflected the circumstances of my life: educated, curious conscientious parents, good diet, exercise, lots of reading. When I took tests, really what those tests measured were a bit of my ability, and a lot of my own education (whether it came from books, my dad’s clipped New York Review of Books articles, or him answering a question with, “Well, do you want the five minute answer, the ten minute answer, or the hour?”). But if you take away the “test scores” and add the “education” that those test scores reflect in that sentence, it becomes both more accurate, and seemingly nonsensical (although profoundly true) “That education helps you get more education, and more education has an earnings effect.”
Which get to my big beef with this study. Just because variation in one factor (teacher VAM) helps predict variation in another (student earnings, teen pregnancy, etc) does not mean that these factors are as mutable as numbers in a model. I often roll my eyes at the “correlation is not causation” criticism, because not all correlations are equal. For example, controlling for other possible correlations (third variables that could explain mutual correlation) helps a lot in making a greater case for a causal role. But when authors of these studies jump from “relatively better correlation evidence” to “we should be firing teachers tomorrow,” I cry foul. Relatively greater education may have led to greater relative earnings in the past, but does this mean that handing out Ph.D.’s will make us all rich? No. Does it mean that sending everyone to college will make us all rich? No. Somehow these economists recognize that simply giving every child in the state some books (so they have more books in the home!) will not necessarily raise their test scores (thereby lifting them out of poverty!) but they still with a straight face proclaim that a bad teacher costs $2.5 million in lost income and should be fired immediately. A CEO makes his company run more efficiently and lowers prices through outsourcing to lower labor costs (saving his shareholders $2.5 million in payroll costs), and he is a hero. Is anyone else dizzy?
Next, the economists who did this study seem to be willfully ignorant of human psychology.
This section was chilling:
“The message is to fire people sooner rather than later,” Professor Friedman said.
Professor Chetty acknowledged, “Of course there are going to be mistakes — teachers who get fired who do not deserve to get fired.” But he said that using value-added scores would lead to fewer mistakes, not more.
Neither of these economists seem to have any idea of three consequences of these policy recommendations.
First, “Firing people sooner rather than later” could change the labor pool of people ready to replace these people who got “fired sooner.” Are people really rushing to fields that promise to fire those who don’t perform in the first year or two? Could I imagine saying this to my kids’ kindergarten teacher? Could you? Show me some results! Read, kid! Reeeeeaaaad! Maybe Chris Rock is right, we just need more fear in schools. Just tell every teacher that their job depends on every student improving every year . . . What could go wrong?
Second, for those who do enter the profession despite the increased firings, this will drastically change how they teach. They will teach more to the test. You had better hope that your tests are great, and they cover the right material, because teachers will do whatever they can to improve their test scores. Or, you could talk to many of the test makers, who tell you their tests aren’t supposed to do that.
Third, all data points are not equal, nor are they as independent as these economists think. One can use statistical techniques to compare apples to apples, and tweak dimensions to get data to seem independent, but people are social critters, and they notice what happens to other people. ” ‘Of course there are going to be mistakes’ but VAM will lead to fewer mistakes, not more,” said Professor Chetty. Not more than what? Presumably what Chetty means is more mistakes than the current approach of just leaving all these awful union-protected teachers in the classroom until they leave voluntarily. But these mistakes are not equal. Firing a beloved teacher whose VAM dips below some threshhold can send a chill through a school in a way that generic benign neglect does not. My impression of the urban schools I was in (and those that my wife and father have taught in) was that being creative and inspirational must serve as its own reward. My creative and inspirational teachers dealt with their hard work being ignored by most (but savored the imagined long term effects on their students). But when instead of being ignored, you are fired? This changes the dynamic. It wreaks havoc on any dynamic of trust in the school. And it erodes freedom, curiosity, and creativity. Reformers may say, well, “Teachers have had their freedom, their curiosity and creativity, and look what they’ve done: all these bad teachers have cost our students x trillion dollars.” Even if that is true, are more of miraculous good teachers actually willing to sign up for an atmosphere where there is more firing? Less freedom? Less room to grow? Less support for improvement? Where are the studies that ask teachers why they go into teaching? Why do they stay? I doubt it’s the money.
The tone of the final few paragraphs really got me. The reporter gave Jesse Rothstein a quote, as he made my point 3 above:
“We are performing these studies in settings where nobody cares about their ranking — it does not change their pay or job security,” said Jesse Rothstein, an economist at the University of California, Berkeley, whose work criticizing other value-added assessments unions frequently cite. “But if you start to change that, there is going to be a range of responses.”
But look at this, the reporter slips Rothstein the knife, even as he is held up as the lone voice of dissent. Listen: “whose work criticizing other value-added assessments unions frequently cite.” First, I am no master prose stylist, but really? What were the alternatives that Lowery tried on that one? Union-favorite economist Jesse Rothstein . . .” No, no, no . . . whose union-supported and cited work . . . No … I guess that is the least bad way to undermine an academic economist without calling him a union shill (but still insinuating it). I noticed no one else got a “who reformy billionaires frequently cite” moniker.
Then, Lowery follows that up with… “Many other researchers…” of course implying that poor Rothstein is in lonely scientific company (but probably sitting pretty on all his union citations). Those two paragraphs seemed to me to be the reporter’s way of presenting a “there are two equal sides to the debate” but wink wink nod nod, knowhatImean, some are more equal than others.
Ok, if you’ve gotten this far, I thought I would give a little personal background of mine. I went to the DC Public Schools. I had some bad teachers. My high school psychology teacher (yes, psychology!) used textbooks that were from 1948 and were stamped “condemned.” I asked why we used these, and she said “human nature hasn’t changed.” I persisted, “But but but, our understanding of it has, right?” To no avail. I remember a question on an exam was “boys and girls are different in that they _____ and they are the same in that they _______.” There was a lot of talk show psychology presented alongside the (laughingly dated) textbook, and classes consisted of mostly uninformed musing from everybody, myself included. I challenged the teacher for the first quarter and then I gave up in disgust in the second quarter and got a C.
The 18-year-old me would probably have found much to like in these studies, but then again, I also had some great teachers. I had a calculus teacher who stayed in at lunch and afterschool, helping many kids with math, and made me want to be a mathematician (until a college math teacher beat it out of me). My computer science teacher taught me how to program in Pascal, and how to see algorithms in the world. I had a comparative government teacher who taught me that the American political system is not a fact of nature but an accident of man. I learned Spanish from kindergarten through 6th grade in my bilingual elementary school. Each grade had two teachers, one English speaking and one Spanish speaking (but often bilingual). I was given the gift of taking multiculturalism for granted by the time I was ten years old. I had a US History teacher that gave us a scavenger hunt around DC, finding little known monuments to new historical moments, and assigning a book I still haven’t forgotten called After the Fact. I could go on.
Give NCLB another ten years and it will get rid of innovative and creative elementary programs like Oyster’s. Bilingualism is nice, but it can’t be counted on to boost scores on poorly contrived reading and math tests. And those great teachers aren’t at my high school anymore. Not because Jason Kamras didn’t come along and make them rich. Not because they got tired of waiting for savior Michelle Rhee to come along and sweep their lower performing colleagues out of the way. Not because no one came and measured their VAM. No, they got worn down by the bullshit that ALL teachers had to put up with in DCPS. They got tired of being told, in so many ways, big and little, that their effort didn’t matter. I saw them often frustrated, even as I loved to learn from them.
So how do we acknowledge that good teaching matters? How do we retain and celebrate good teachers and get more of them and have fewer bad teachers?
Here’s my modest proposal:
– Provide decent working conditions for all teachers
– Prioritize improving teaching practice, not evaluating teacher quality.
– Pay all teachers a little bit more, but mostly help them do what they came to the profession to do: inspire their students to love learning.
– Respect and support a broad array of inquiry and knowledge, so that teachers can teach a rich and varied curriculum
Do this, and you’ll find bad teachers get better, good teachers want to continue teaching, and more people want to enter teaching.
Or, you could increase class sizes, increase high stakes testing, give a rare $20,000 bonus after making teachers compete with each other for salary gains based on VAM, and get going with that firing project. Let me know how that goes.
It is shameful that our politicians are being bribed for permitting the corporate vultures to prostitute the teaching profession. I left the business world because of the “me” focus and now they (corporate vultures and their paid for politicians) want to drive this selfish attitude into the teaching profession. This is beyond sickening and will only have sickening results. It is interesting that if our country wanted to be #1 on international tests we would be….but I have never heard this being a priority. To our bribed government officials: make the international tests a national priority, soundly support this priority, and we will be number one. Otherwise, you corporate puppets have rigged the game for a corporate takeover…one where profit is the motive and not educating the children – investing in our future.
Thank you, thank you, thank you for your eloquent response to a study that is causing such buzz in the education world. I am a 25 year elementary teacher and I am absolutely dismayed by the trend in ed reform these days. My best gifts to kids hasn’t always been increased test scores, but maybe a little hope and belief that learning is wonderful and important all on its own. Give me a class of well-fed, well-read, and well-parented students and I can rock those scores -but even in the best of neighborhoods those kids make up only a percentage of the class. I hope your blog gets the notice it so deserves – I will send you on to all my very disheartened colleagues. Carry on!
@Vicki Thanks for your kind reply, and your hard work as an elementary school teacher. My kids have been the beneficiaries of some wonderful teachers, despite sometimes struggling with the political circumstances. One of my sons came home last year and said “Dad, I love reading, but I hate reading tests.” I can’t imagine hearing that as a teacher and having to say, “Well, tough kid. Wait until next year when the real tests start..”
@Arizona Joe: thanks for coming by. I agree that the “survival of the fittest” competitive view of the free market isn’t always a great way to run a business either. Thanks for commenting.
I’m grateful to @Diane Ravitch for tweeting the link to your blog. You articulated perfectly my reaction to the NYT article, which had been shared by two Facebook friends, one a school board member and one a former teacher who is a parent activist and reformer. I am not surprised but I am saddened that the public and the media want to simplify the complex problems of education to “just fire all the bad teachers,” as if society’s attitudes and socioeconomic factors have no impact on student success. Please consider figuring out a way this essay can find a wider audience, perhaps by sending it as a proposed Op Ed to the NYT
Very well said, Cedar. Lucky to have come across this brilliant post on the day I am paying “blog calls” to you today (as I am to each @scio12 attendee, eventually) to say “Hi” and give your blog a shoutout on twitter (I’m @sciencegoddess). I look forward to meeting you in January!
Thank you for this post, I have forwarded it to many who may not comment here but really appreciated your thinking. I fully agree with your statement that the approaches cited are toxic and corrosive to the improvement of education in our country. Before this NYT piece came along I was still responding to Sam Dillon’s piece from last week’s NYT on evaluation including the DC teacher evaluation system IMPACT – in my head.
Folks endorsing the “fire them quick” method and elevating evaluation over investment in improving teaching practice with respect and curiosity, should take a hard look at DC. The Sam Dillon article seemed to assume that IMPACT was an accurate measure of a teacher’s quality. IMPACT includes VAM in the grades tested. This is combined with a checklist that is the same for every grade from pre-school through 12th grade. A teacher could conceivably get a highly qualified rating and bonus without assigning and correcting much student writing or even ensuring that students are reading an appropriate amount. The focus is on the student’s engagement during the observed class periods and the teacher’s, questioning, use of a variety of approaches, push for higher understanding, and ability to assess often among other things – writing or work done outside of that class is not included.
I have never seen morale lower in the high schools I have contact with- despite the pay raises and teacher bonuses. The assessment that all we need are good teachers and then the rush to simple measures has come at great cost especially in the urban systems where the need is so great. We have lost a community that feels the freedom to think about how they can do better. Ironically a system driven by test scores and checklists is not one where people think – they work on compliance and controlling what they can. It has not been motivating, one teacher after the first evaluation with a difficult class realized he would not get highly effective, he left at the end of that year. It was not solely because of his evaluation. He went to a charter school where he felt he could be more broadly used and appreciated.
There is even less acknowledgement of the teachers who use their summers to take student’s abroad, foster a debate or robotics team, do a school newspaper or drama club. The lifelong positive effect of expanding student’s horizons in these ways is not reflected in tests or in value added. With even less incentive to engage in student lives this way, it is happening less and less.
The discipline of scientific inquiry and all that you as cognitive scientists are learning has not permeated the education policies currently in use, but I hope you will keep at it- your work and articles like this make a difference.
( I am the director of the Senior High Alliance of Parents, Principals and Educators in Washington DC and also related to Cedar)
Great piece! As we saw in Washington, it requires a huge leap in logic to say, categorically, that better teachers cause higher test scores. When the Washington Post reported that the best teachers were in schools in Upper Northwest based on the higher test scores there, it was obvious that they were not looking at all of the other factors that went into making those test scores higher.
Besides denigrating teachers and teaching, studies like this also denigrate education. For economists, education is just easily measurable input and easily measurable output. Every time I see studies like this or articles praising education reform, I imagine that Paulo Freire is rolling over in his grave. The truth is that economists are not interested in critical, free thinkers. They are interested in ensuring that people believe exactly what they believe and in creating systems that reinforce their dogma.
Thanks for stopping by, Pete!
As an academic social scientist, I think I am somewhat more sympathetic to the plight of the economist than you are, but not by too much. I think rather than being interested in keeping people believing what they believe, I think economists are interested in what they can measure. The things that are harder to measure, they don’t pay as much attention to. I think this is an occupational hazard, but it is a reason why anytime economists are suggesting policy, they should be more measured about the limitations of their measures. I don’t think these tests are worthless, just that we should be realistic about their worth.
Thanks for dropping by. I haven’t yet read Freire. Gotta get to that.
I wish it was true that most economists don’t want us to believe what they believe. However, many are totally and completely driven by ideology and by proving that their ideology is correct and should be the framework through which all decision-making is undertaken. Now, not all economists are free market demagogues, but many are. If you reflect back on our economic history over the last sixty years you will see that we have been at the mercy Hayek and Friedman acolytes with less regulation, labor protections, and certainly no care for the environment the result of their power. If you can prove to me that these two weren’t driven to make their philosophy dominate the world, then I will give you a cookie. The spread of neo-liberal economic thought has even made it into education. We wouldn’t hear the word “competition” otherwise. Why are we talking about removing labor protection for teachers? Worse, this isn’t just about educating children, but profit. Look at our friends at the Washington Post. We know why they publish what they publish.
@Peter Ah well, no cookie for me. I don’t doubt that there are plenty of free market demagogues, and many of them are now “studying” education. And the result is likely the same, but I don’t think it is productive to say that they are totally and completely driven by ideology, as much as many of the assumptions they have to make to simplify the world enough to study it, are wrong. Many paragraphs in that study just seem absurd to me, but for them, they are assumptions they have to make. I guess what I am saying is that saying “driven by ideology” makes it sound as if they already know all the answers, and are plugging away at trying to confirm their biases. I see it more as a kind of blindness, where their world-view constrains the questions they even think about asking. Maybe it is six of one, half a dozen of the other. And I agree that one of those simplifying assumptions often made by certain economists is a little thing called human psychology.
Anyways, thanks for pushing back. Oh, one more thing. I think Hayek would be pretty disappointed with what people are doing in his name. The stuff he was calling fascism was actual fascism, and his world view was quite influenced by a reaction to that. I don’t think he would look around today and call the Civil Rights Act in the same league. In other words, I don’t think he would be voting for Ron Paul.
I guess you have more faith in their honesty than I do. I am quite certain that they believe that they already have the answers and that they will do anything that they can to ensure that any study they undertake will produce results that reinforce their beliefs. Economics requires a world view. It’s not merely an honest study of the exchange of goods and services on a micro and macro scale. Now, perhaps there are other disciplines that have similar problems. However, economics has come to dominate every single facet of modern life. That didn’t just happen. It has been a steady drumbeat that is far different than other fields of study. I simply do not believe that economics can be spoken of in the same way as history or anthropology.
Further, though I am complaining about right wingers, I can just as easily point out Marxist economists who are driven by ideology. I might believe that they are driven by a more honest and clear view of the world. However, they are still driven by a philosophy. Their studies are undertaken with the same biases.
I would like to believe that studies like this just happen to not include crucial indicators, that those indicators were merely overlooked. However, I don’t believe that. I’m a cynic when it comes to this stuff.
Pingback: A New Model: Schools as Ecosystems | GothamSchools
Pingback: A New Model: Schools As Ecosystems | Ecology of Education