Attention conservation notice: This post dives into some inside baseball stuff on social psychology, how the science of psychology is practiced, and how science is communicated online. It is kind of long, but I intend it as sort of a reference for this event, but also something to point to for my colleagues who either aren’t aware of, or doubt whether real scientific discussion can occur online.
There was a recent kerfuffle over a finding in social psychology.
No, not that one.
No, not that one either.
Not that one either.
I’m talking about the non-replication of a famous study in psychology in which people who are unconsciously primed with words associated with the elderly end up walking more slowly toward the elevator after the experiment is over. I am going to give a brief summary of the situation, but then break it down into why I found it so interesting, not necessarily in terms of the psychological concepts themselves, but in terms of what this means for the modern practice and communication of science. In the past I have been skeptical of online post-publication peer review this serving as a bona fide replacement of peer review. Comparing my own comments when I have served as a reviewer (and recieved for my submitted manuscripts to those that I leave as a web commenter, the formal peer review process (anonymous or otherwise) has always come out on top. But this latest episode, and some related conversations, have convinced me that online post-publication peer review has made an amazing amount of progress.
I’ll start with a sort of annotated bibliography of this “event.” (now updated with Bargh’s recent response)
1996: John Bargh, Mark Chen and Lana Burrows publish an article entitled “Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action” in one of the premier journals of social psychology, the Journal of Personality and Social Psychology. They find that one can unconsciously activate (“prime”) certain ideas (“trait concepts”) and that this unconscious priming can affect relevant social behavior. People knew priming existed, and that priming could influence things like how we interpret a social situation, but most people had nonetheless assumed that social behavior (action, not just thoughts or perception) was more under our conscious control, and less susceptible to unconscious priming. In other words, the paranoia of subliminal advertising was overblown. Flashing “BUY COKE” before the movie does not cause hordes of brainwashed puppets would shuffle to the concession stand. Regular advertising, on the other hand, remains amazingly effective.
Gustav Fechner, one of the founders of psychology, and certified old dude.
This article becomes very influential, inspiring a great amount of work on unconscious priming and the importance of unconscious attitudes on behavior. It fits into a spectrum of work on the modern unconscious (or what Tim Wilson calls “non-conscious” to separate from Freud’s psychoanalytic unconscious). Steele and Aronson’s stereotype threat research (beginning roughly in 1995), Anthony Greenwald’s research on implicit social cognition and many other social psychologists have studied how non-conscious stimuli and attitudes can affect our thoughts and behavior.
Despite this influence, there doesn’t seem to be a record of anyone replicating this particular Bargh study. That is, no one has re-run it with the same method and found the same results. Given the way that publication bias works (there is a huge bias to only publish studies that “work” or find statistically significant differences between groups) it is likely that at least some people have tried, but failed (see some of the links below for more details on this point).
2012 In the recent open access journal PLOS-One, a group published a non-replication of this study, apparently finding that even though they used the same methods (more on this in a second) they did not find the same results.
Then, this is all within a week:
1) Science writer extraordinaire Ed Yong writes up this failed replication attempt on his blog at Discover magazine: “Not Exactly Rocket Science.”
2) John Bargh responds on his blog at Psychology Today.
3) Ed Yong responds to the response on his blog.
4) The psychology blogosphere erupts with conversation and reaction. I am going to annotate a little here.
a) The comments on Ed Yong’s first post include someone who is a priming researcher (“Joe”), another research psychologist (“Chris”) as well as someone who I would assume is Bobbie Spellman (“Bobbie”), a well-respected social psychologist and one of the founders of a site devoted to rectifying the fact that unsuccessful studies often never see the light of day: PsychFileDrawer.org
b) The initial response by Bargh (62 comments) includes comments by Ed Yong, a neuroscientist who goes by Neuroskeptic, Publisher of PLoS One Peter Binfield, another founder of PsychFileDrawer.org Alex Holcombe, one Peter C who says he is an editor of a major journal in this field, in addition to a number of anonymous commenters.
c) Ed Yong’s response to the response has mostly named commenters, including: Well known cognitive (gorilla) psychologist Dan Simons, social psychologist Dave Nussbaum, social psychologist Michael Krauss, neuroscientist Chris Chambers, neuroscientist and editor at Nature Noah Gray, cognitive neuroscientist Matt Craddock, little old me, and social psychologist (and someone who published with Bargh)Gordon Moskowitz.
d) Dan Simons google+ post entitled “A primer for how not to respond when someone fails to replicate your work”. This includes a long conversation between Dan and another well-known cognitive psychologist, and one of the few reasons I would still visit Psychology TodayArt Markman.
e) Friend and social psychologist par excellence Sanjay Srinistava posts reflections on his blog.
f) Conversations on twitter about the difference between conceptual replication and direct replication involving Ed Yong and quantitative molecular biologist, Professor of genomics and evolutionary biology at Princeton, and guy you should totally follow on twitter Leonid Kruglyak. And me. Here’s my first attempt at a storify of this convo.
-
Ed Yong tweeted about one commenters distinction between direct replication and conceptual replication:
-
“Comments on the Bargh post have veered into a discussion on “conceptual replication”, a concept that troubles me bit.ly/yVLPxt
-
Leonid Kruglyak is also put off by the term “conceptual replication” which sounds alarmingly postmodern, as if it could have appeared in Sokal’s hoax paper.
-
“Conceptual replication: merely troubling or completely unscientific? @edyong209
-
He has a little fun with it…
-
““We know chemicals kill cancer cells; who cares if we can reproduce this drug killing this cell line” #conceptualreplication
-
““I found a gene causes a disease, who cares if you don’t get the same gene” #conceptualreplication
-
One of the things that sucks me into conversations online is defending psychology as a science. Chip, meet shoulder.
-
“@leonidkruglyak think u might be going bit too far here. Much of social psych is #conceptualreplication just bc prejudice isn’t chemical …
-
“@leonidkruglyak @edyong209 Bystander studies too. Started with false premise (wrong story of Genovese murder). But repl. und dif conds.
-
JP McGinnis cites the classic Doug Mook paper “In Defense of External Invalidity” on why we should run more tightly controlled, unnatural, lab studies in psychology. I think these kinds of artificial situations that eschew ecological validity would likely be more easily replicable. Except in this case.
-
“@leonidkruglyak @criener I’m way out of my depth on this, but thought this might be applicable to hard vs softer sci bit.ly/ylpGfl
-
Kruglyak goes for the jugular (yes, he is a heavy weapons guy, see second tweet)
-
“@criener either an experiment is directly replicable or it’s not science, no?
-
“FYI: I’m a heavy weapons guy RT @WvSchaik: Awesome! The nine types of peer reviewers: goo.gl/xSAzW HT @edyong209
-
I try to defend non-replication still leaving the possibility of science
-
“@leonidkruglyak If we don’t understand/can’t control all relevant factors of phenomena, some unknown could cause to fail to replicate
-
“.@criener I always understood replication to be central to scientific method. That’s why we write Methods sections. #conceptualreplication
-
Ed Yong refers to his comment at his blog, worrying that many conceptual replications would allow weak studies to appear to bolster each other, but without direct replication they would remain a house of cards.
-
-
I agree this can be a problem, especially from the outside, it can be hard to distinguish from well-worn replicated studies (like the bystander studies by Darley and Latane, or the obedience studies by Milgram) and sexy novel studies that turned out to be fraud like Diedrich Stapel.
-
“@edyong209 @leonidkruglyak Agreed. In social psych sometimes hard to separate Darley&latane, Milgram, wheat from Stapel chaff
-
-
“@edyong209 Don’t know. File Drawer journal trying to help. But to some degree, no shortcut to vetting – network of weak or strong results…
-
“@edyong209 being evaluated by social psych’s within in the field. I don’t actually know what makes one priming study better than another
-
In conclusion (for now) Kruglyak boils it down…
-
“.@edyong209 @criener I think it comes down to, can you really have a solid concept if no individual instance of it is solid/replicable?
-
I replied to this in the comments at Ed Yong’s blog post, but I think this is right, a solid concept isn’t so solid if no one of its legs is strong. My take on this for social psych, and this instance is that there may not have been a direct replication of this study, but priming in general is well-replicated, as is the concept of unconscious attitudes affecting behavior (for example Greenwald and colleagues work on the IAT). I guess I would say that different parts of the concept have different support. I am perfectly willing to be skeptical of the slow-walking based on elderly priming effect, but not that priming or unconscious drivers of behavior.The next day, a few more follow up posts are relevant.
-
Kruglyak cites Roddy Roediger’s column about the value of replication (and citing his own highly replicated results with the DRM word lists).
-
“Psychology’s Woes and a Partial Cure: The Value of Replication bit.ly/AEyA3K
-
Matt Lieberman posts an interesting blog post citing a recent paper by Scott Lilienfeld on some o the reasons for psychology’s reputation
-
“Why Are People So Skeptical About Psychology? @psychoBOBlogy bit.ly/ywM1K6
-
This is Dan Simons’s epic google post.
-
Nothing in Their Heads | Psychology Today
A primer for how not to respond when someone fails to replicate your work with a discussion of why replication failures happen In the lin… -
Sanjay Srivastava’s thoughtful follow-up post, including some social psych details and background
-
“New post: some reflections on the Bargh-Doyen-@edyong209 elderly-walking nonreplicastravaganza wp.me/pt9Wa-l2
*********
Update: Here is Bargh’s recent response from today (3/25/12)
Update2: Chris Chambers has a post up about conceptual replication. I agree with him, and I think looking back up at the storify, I was defending the value converging evidence and just accepting that social psychology seems to be calling converging evidence “conceptual replication.” As Chris points out, this devalues direct replication, and leads critical thinking away from the converging evidence, which is necessary, but can sometimes be weak.
*********
Post Publication Peer Review Online and the Future of Science?
A few things struck me about these conversations.
First, there is a generational divide here. Bargh’s response is defensive and snippy, but it made a certain kind of sense to me. He blew Ed Yong off in the first place, writing a very quick email in response to Ed’s request for his response to Doyen. Then, when the article came out, I think part of the reason he took offense was at being compared to William von Osten (Clever Hans owner, who apparently never gave up his faith in the amazing intellectual powers of his horse). As Dan Simons noted, Ed is one of the best young science writers today, with real integrity, and a desire to get things right, even if it involves going back and making corrections. Ed is also fun to read, which can involve creative comparisons and snarkiness. Bargh misread him in the first place (probably as a “blogger” instead of a well-respected science journalist). If Bargh had realized that talking to Yong in the first place would have been actually a much better forum to clarify his concerns than his own blog at Psychology Today, a lot of the heat of this debate wouldn’t have existed. Update: This generational divide still shows in his most recent response, which is more background on the article and studies, as well as more background on priming and social psychology. I happen to agree with most of what he says in this “Angry Birds” response, but I don’t think he is going to win any converts, partly because he is still treating his “blog” like a lectern (there are no external links and no one mentioned by name) and also because people’s attention span is likely over. That said, if he had given Ed Yong this lecture at the very beginning (when Ed asked for it), I think Ed would have been far more gracious.
Second, the distributed nature of the conversation makes it very hard to follow and participate. Trying to track comments on two different Ed Yong posts, Bargh’s psychology today post, as well as twitter and G+ made it overwhelming for me. I can only imagine psychologists not on twitter, or people loathe to sign up for a site to comment (like my advisor or many of my mentors) would be missing half of the conversation.
Third, it is fascinating to me how real scientific dialogue can now co-exist with science education and promotion of science. In a single blog comment feed, there can be someone who has the knowledge of an undergraduate asking a relatively basic question (and being answered), and someone who has done this research for ten years asking an advanced one (and having a dialogue with another expert for all to see). Daniel Bor’s recent blog post about the dilemma of weak neuroimaging results has some great explaining, but at the same time, some practicing expert scientists referring to specific papers and specific methods that are sometimes over the heads of people who haven’t had graduate study in neuroscience.
Finally, the upshot of all of this to me is a balanced optimism about post-publication peer review online. To other scientists I say: Click on some of these links (especially that Bor post) and be amazed. There is real scientific discussion taking place in public, it is really not that watered down. It is totally worth it to blog (and tweet) about your papers, not just to inform “the public” of your results, but to possibly engage with a scientific audience. But second, to the public, I say, real post publication peer review looks kind of like regular peer review: experts often disagree, and “scientific consensus” isn’t always immediately apparent, or easily communicable at that moment. If we are expecting post-publication peer review to be that much better than regular peer review, I am still somewhat skeptical that the right “peers” will show up and apply due diligence to interpreting and evaluating a study.
Cedar's Digest 
Cedar, thanks for pulling all these pieces of the conversation together — a number of which I’d missed. The conceptual vs direct replication conversation was one of the most interesting parts of this and I had missed the Twitter exchange.
I think your comment about a digital divide (split roughly, though not completely, by generation) is spot-on. Open-access journals, blogs, twitter, etc. are becoming places for substantive discussion and review of science, and people who dismiss them out of hand (as Bargh did) are misguided. At the same time, it depends so much on who is involved. That thread on Bor’s blog is a great example of how things can go right. But as a contrasting example, in the last week I saw some silly offhanded comments about the supposed invalidity of personality traits on a blog that usually has very good content. Nobody gave a substantively informed response until well after a lot of anonymous commenters had taken off running with the silliness. I was initially tempted to jump in and correct some of the obvious errors, then I thought of how much time I might spend responding to everything that’s wrong on the internet and walked away. Hopefully as more and more people engage with electronic media, the number of high-quality discussions will continue to increase.
Yeah, much agreed. Even websites with lots of very educated people end up with knee jerk stuff when it goes outside people’s area of expertise.
What I really like is a trend for well-known people to get involved in comment sections, either by going by their own name, or by giving a mini bio at the beginning. When this happens early, especially in a comment section that has heard of these people, I think the noise dies down, the way it would at a brown bag lunch talk when the two famous old professors go at it.
But yeah, those times are still few and far between.
I would hope at some point people would start realizing that the old shortcuts for credibility and popularity don’t always hold. A tweet from Ed Yong or Maria Popova (brainpicker) may steer more popular interest (and educated popular interest) than a segment on local news, or profile in local paper. As far as credibility, Ed Yong is a curious case to me, because he has arrived at his high level of respectability post by post and tweet by tweet. No big special, no tv presence, no best selling book, but science nerds revere him, as they should, because he writes great prose, with integrity
Pingback: Around the Web: Promise & perils of Pinterest, Abundance vs disruption, Beyond the textbook and more : Confessions of a Science Librarian
I am a master student of psychology in China, and keep an eye on this for a while.
This post give me a comprehensive view of this debate, but I still have two questions:
1, I really can’t grasp the idea of “conceptual replication” through these tweets.
2, Is there any comments or responses from Doyen et al.?
@hcp4175 – I think you might appreciate reading Bargh’s latest response.
I don’t know of any responses by Doyen, but I’d be happy to include them if any get posted.
Yes, I got a quote from Axel Cleeremans in my reply post.
Pingback: Navel gazing on steroids in social and personality science | pigee
I think the movie Drive used priming and unconscious cognition to get people to hate Jews, and act upon that hate. It is one thing to discuss theory, seeing it in action is another. Good Blog