Chapter 4: Case Studies and Testimonials

This chapter deals with an enduring theme in my own thoughts, and a tension between the practice of science and the limits of the human mind.
Those pieces of evidence which we naturally find convincing do not do real work in science.

To understand why, it is important to understand both what makes some types of evidence more important than others in science, as well as what makes some evidence more convincing to your mind.

As usual, Stanovich clearly and directly gets to the point:

“Case Studies and testimonials are virtually worthless as evidence for the evaluation of psychological theories and treatments”

Testimonials and case studies can support any theory, any treatment. Why?
For treatments, the worthlessness of testimonials can be attributed to the amazing powers of the body to heal itself. Sometimes I think scientists or skeptics invoke the placebo effect too often to dismiss shoddy medical research, that they forget that it is absolutely incredible how powerful the placebo effect truly is. The other factor that undermines the value of testimonials and case studies is spontaneous remission, in which people get better for inexplicable reasons.

In science of psychology (other than treatments), testimonials and case studies are not up to the task in helping us to decide between competing theories. Why is this? First, for testimonials a set of biases make our memories ill-suited to serve as scientific evidence. Our memories are intertwined with our emotions, and infected with our personal experience. Stanovich brings up the availability heuristic, which he calls “the vividness problem” (a better, more descriptive term). We judge the likelihood of an event, or how often it happens, in relation to how quickly or easily it comes to mind. In other words, we neglect to notice that we get a biased view of the world, and therefore what sticks in our mind is not an accurate picture of the world. The classic example is that car travel is far more dangerous than commercial plane travel (by accidents per miles traveled, per passenger, but people don’t feel nearly as scared of driving as they do of flying. Guns are another example. People sometimes buy a gun with the intention to protect their home and family. A gun in the home makes them feel safer. But by the numbers, more people commit suicide by gun than homicide by gun. Most gun deaths are either unintentional or suicides. But people buying guns don’t imagine shooting themselves with it, or their loved ones. They imagine a criminal breaking into their house to hurt them, and they see having a gun as a way to protect them in this event. But just because something is easier to imagine doesn’t make it more likely.

We don’t live in a world of facts, we live in a world of artifacts. Through our culture, through reinforcing fears, through many other channels, certain kinds of information shine brighter in our mind. Taking a testimonial at face value is a way of sweeping the vividness problem (and many other cognitive biases) under the rug and taking someone at their word. “This drug worked for me because I took it and felt better” ignores the amazing ability of your body to feel better when you truly believe it should. “I learn better with visual/auditory/kinesthetic material” ignores the unconscious processes, emotional biases, and previous knowledge that truly have an impact on our learning.

But let’s not throw the case study out with the bath water. Testimonials and case studies are worthless as evidence, but not as inspiration, and not all case studies are equivalent. The classic example in psychology are case studies of individual patients with neurological problems. Henry Moliason had extreme brain surgery in efforts to relieve his debilitating epilepsy. Intending to remove parts of his brain that apparently weren’t needed, his doctor William Beecher Scoville took out both side of Henry’s hippocampus. Thenceforth, Henry was unable to create new memories. This single case study, this single person’s brain and behavior, has told us more about memory than any other experiment. Why? First, Henry’s case was strong evidence for different categories of memory. Henry could remember plenty of facts from before his operation, but he could create no new facts. He could not learn anyone’s name, he could not read a newspaper article and remember anything from it, or even that he had read it! This suggests that storing old facts and creating new facts are separate processes, because one can be affected in the brain while the other is not. The second differentiation comes when it becomes apparent that despite being unable to learn new facts, Henry is nonetheless able to learn new skills. He can get better at tracing a figure in a mirror (actually very hard, give it a try), but at the same time, having no memory of ever practicing the task. This gives support to the idea of different systems for implicit (skill) memory and explicit (fact) memory. But does Henry serve as proof in isolation? No, he serves as a launching off point for many other studies.

Stanovich uses this discussion to lead us into a discussion of pseudoscience and the Amazing Randi, but to me, this avoids the harder problem.

How do we know when to resist our own “testimonials”? Nowhere has this been more clear to me than my own experience parenting. When my children don’t sleep, why is it that they aren’t sleeping? Is it because they woke up late? Is it because they are anxious about school? How about seasonal change? Maybe they are just going through a phase. In these instances, a scientific approach is rarely practical, so my wife and I are left with coming up with our own, unfalsifiable, case-study based hunches. After all, we don’t really want science, we just want peace and quiet after 9:30 at night. So we use whatever information comes vividly to mind, and we ask our children to explain why in god’s name are they still up? Stanovich reminds us, though, that while it may seem perfectly natural to apply this reason to the broader world, the tools of science are more effective in deciding between competing general theories, and these don’t include case studies or testimonials.

Advertisement