Attention conservation notice: This post dives into some inside baseball stuff on social psychology, how the science of psychology is practiced, and how science is communicated online. It is kind of long, but I intend it as sort of a reference for this event, but also something to point to for my colleagues who either aren’t aware of, or doubt whether real scientific discussion can occur online.
I’m talking about the non-replication of a famous study in psychology in which people who are unconsciously primed with words associated with the elderly end up walking more slowly toward the elevator after the experiment is over. I am going to give a brief summary of the situation, but then break it down into why I found it so interesting, not necessarily in terms of the psychological concepts themselves, but in terms of what this means for the modern practice and communication of science. In the past I have been skeptical of online post-publication peer review this serving as a bona fide replacement of peer review. Comparing my own comments when I have served as a reviewer (and recieved for my submitted manuscripts to those that I leave as a web commenter, the formal peer review process (anonymous or otherwise) has always come out on top. But this latest episode, and some related conversations, have convinced me that online post-publication peer review has made an amazing amount of progress.
I’ll start with a sort of annotated bibliography of this “event.” (now updated with Bargh’s recent response)
1996: John Bargh, Mark Chen and Lana Burrows publish an article entitled “Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action” in one of the premier journals of social psychology, the Journal of Personality and Social Psychology. They find that one can unconsciously activate (“prime”) certain ideas (“trait concepts”) and that this unconscious priming can affect relevant social behavior. People knew priming existed, and that priming could influence things like how we interpret a social situation, but most people had nonetheless assumed that social behavior (action, not just thoughts or perception) was more under our conscious control, and less susceptible to unconscious priming. In other words, the paranoia of subliminal advertising was overblown. Flashing “BUY COKE” before the movie does not cause hordes of brainwashed puppets would shuffle to the concession stand. Regular advertising, on the other hand, remains amazingly effective.
This article becomes very influential, inspiring a great amount of work on unconscious priming and the importance of unconscious attitudes on behavior. It fits into a spectrum of work on the modern unconscious (or what Tim Wilson calls “non-conscious” to separate from Freud’s psychoanalytic unconscious). Steele and Aronson’s stereotype threat research (beginning roughly in 1995), Anthony Greenwald’s research on implicit social cognition and many other social psychologists have studied how non-conscious stimuli and attitudes can affect our thoughts and behavior.
Despite this influence, there doesn’t seem to be a record of anyone replicating this particular Bargh study. That is, no one has re-run it with the same method and found the same results. Given the way that publication bias works (there is a huge bias to only publish studies that “work” or find statistically significant differences between groups) it is likely that at least some people have tried, but failed (see some of the links below for more details on this point).
2012 In the recent open access journal PLOS-One, a group published a non-replication of this study, apparently finding that even though they used the same methods (more on this in a second) they did not find the same results.
Then, this is all within a week:
1) Science writer extraordinaire Ed Yong writes up this failed replication attempt on his blog at Discover magazine: “Not Exactly Rocket Science.”
2) John Bargh responds on his blog at Psychology Today.
3) Ed Yong responds to the response on his blog.
4) The psychology blogosphere erupts with conversation and reaction. I am going to annotate a little here.
a) The comments on Ed Yong’s first post include someone who is a priming researcher (“Joe”), another research psychologist (“Chris”) as well as someone who I would assume is Bobbie Spellman (“Bobbie”), a well-respected social psychologist and one of the founders of a site devoted to rectifying the fact that unsuccessful studies often never see the light of day: PsychFileDrawer.org
b) The initial response by Bargh (62 comments) includes comments by Ed Yong, a neuroscientist who goes by Neuroskeptic, Publisher of PLoS One Peter Binfield, another founder of PsychFileDrawer.org Alex Holcombe, one Peter C who says he is an editor of a major journal in this field, in addition to a number of anonymous commenters.
c) Ed Yong’s response to the response has mostly named commenters, including: Well known cognitive
(gorilla) psychologist Dan Simons, social psychologist Dave Nussbaum, social psychologist Michael Krauss, neuroscientist Chris Chambers, neuroscientist and editor at Nature Noah Gray, cognitive neuroscientist Matt Craddock, little old me, and social psychologist (and someone who published with Bargh)Gordon Moskowitz.
d) Dan Simons google+ post entitled “A primer for how not to respond when someone fails to replicate your work”. This includes a long conversation between Dan and another well-known cognitive psychologist, and one of the few reasons I would still visit Psychology TodayArt Markman.
e) Friend and social psychologist par excellence Sanjay Srinistava posts reflections on his blog.
f) Conversations on twitter about the difference between conceptual replication and direct replication involving Ed Yong and quantitative molecular biologist, Professor of genomics and evolutionary biology at Princeton, and guy you should totally follow on twitter Leonid Kruglyak. And me. Here’s my first attempt at a storify of this convo.
Update: Here is Bargh’s recent response from today (3/25/12)
Update2: Chris Chambers has a post up about conceptual replication. I agree with him, and I think looking back up at the storify, I was defending the value converging evidence and just accepting that social psychology seems to be calling converging evidence “conceptual replication.” As Chris points out, this devalues direct replication, and leads critical thinking away from the converging evidence, which is necessary, but can sometimes be weak.
Post Publication Peer Review Online and the Future of Science?
A few things struck me about these conversations.
First, there is a generational divide here. Bargh’s response is defensive and snippy, but it made a certain kind of sense to me. He blew Ed Yong off in the first place, writing a very quick email in response to Ed’s request for his response to Doyen. Then, when the article came out, I think part of the reason he took offense was at being compared to William von Osten (Clever Hans owner, who apparently never gave up his faith in the amazing intellectual powers of his horse). As Dan Simons noted, Ed is one of the best young science writers today, with real integrity, and a desire to get things right, even if it involves going back and making corrections. Ed is also fun to read, which can involve creative comparisons and snarkiness. Bargh misread him in the first place (probably as a “blogger” instead of a well-respected science journalist). If Bargh had realized that talking to Yong in the first place would have been actually a much better forum to clarify his concerns than his own blog at Psychology Today, a lot of the heat of this debate wouldn’t have existed. Update: This generational divide still shows in his most recent response, which is more background on the article and studies, as well as more background on priming and social psychology. I happen to agree with most of what he says in this “Angry Birds” response, but I don’t think he is going to win any converts, partly because he is still treating his “blog” like a lectern (there are no external links and no one mentioned by name) and also because people’s attention span is likely over. That said, if he had given Ed Yong this lecture at the very beginning (when Ed asked for it), I think Ed would have been far more gracious.
Second, the distributed nature of the conversation makes it very hard to follow and participate. Trying to track comments on two different Ed Yong posts, Bargh’s psychology today post, as well as twitter and G+ made it overwhelming for me. I can only imagine psychologists not on twitter, or people loathe to sign up for a site to comment (like my advisor or many of my mentors) would be missing half of the conversation.
Third, it is fascinating to me how real scientific dialogue can now co-exist with science education and promotion of science. In a single blog comment feed, there can be someone who has the knowledge of an undergraduate asking a relatively basic question (and being answered), and someone who has done this research for ten years asking an advanced one (and having a dialogue with another expert for all to see). Daniel Bor’s recent blog post about the dilemma of weak neuroimaging results has some great explaining, but at the same time, some practicing expert scientists referring to specific papers and specific methods that are sometimes over the heads of people who haven’t had graduate study in neuroscience.
Finally, the upshot of all of this to me is a balanced optimism about post-publication peer review online. To other scientists I say: Click on some of these links (especially that Bor post) and be amazed. There is real scientific discussion taking place in public, it is really not that watered down. It is totally worth it to blog (and tweet) about your papers, not just to inform “the public” of your results, but to possibly engage with a scientific audience. But second, to the public, I say, real post publication peer review looks kind of like regular peer review: experts often disagree, and “scientific consensus” isn’t always immediately apparent, or easily communicable at that moment. If we are expecting post-publication peer review to be that much better than regular peer review, I am still somewhat skeptical that the right “peers” will show up and apply due diligence to interpreting and evaluating a study.