I've Been Preaching This For Years

The More You Hear, the Less You Know
A new study casts doubt on all the old studies

It has been a long time since the words "scientific study" automatically inspired respect. We've been fooled too often since they told us saccharin causes cancer and closed the dioxin-rich Love Canal, only to bring both back later. Apples and Oreos have been on a hit list; butter was bad, but then it was better than margarine. Even the government's own Food Pyramid, the one that once glorified pasta, has now been adjusted to reflect thinking closer to what many mothers in the 1950s already knew about a healthy, balanced diet.

There are many reasons why health scares and medical treatments come and go. Now we have a scientific study of scientific studies that sheds some light on the subject. An article in the July 13 Journal of the American Medical Association (JAMA) examines how often research that gets a lot of attention--in the scientific community and in the wider world--is then challenged by later studies that reach opposite or less dramatic conclusions.

The article's title, "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research," is simple compared with the complex methodology it describes. Suffice it to say that the article's author, Dr. John P.A. Ioannidis of the Tufts-New England Medical Center, ultimately determined that of 45 studies that revealed effective medical interventions, one-third were eventually contradicted either totally or in part by other studies.

Let's zero in on a few examples. Hormone therapy, initially said to reduce "coronary artery disease events" in women, is now believed to increase the risk of such events. Vitamin E therapy, touted as a heart protector for men and women, was later found to be ineffective. Several of Dr. Ioannidis's other comparisons--concerning, for example, a drug said to slow HIV disease progression or treatments to prevent strokes--show that positive results revealed in an initial study were later found to be much more modest, or to have only short-term effects.

In a number of cases, the explanation for the discrepancies may lie in precisely what you'd suspect: The shorter the study, and the smaller the group studied, the more likely it may be that subsequent, deeper, investigation will contradict or alter the original thesis. Dr. Ioannidis's article also suggests that research pointing to possible therapies tends to get more attention than research with "negative findings," e.g. that Vitamin A does not protect against recurrences of breast cancer.

Of course hopeful stories generate more excitement than ones that shoot down hopes. Even the reality of refuted research is not that shocking, since medicine has steadily advanced despite the fits and starts that mark every learning process. More alarming is the speed with which a single study can become fact--often with the assistance of a credulous or sensation-seeking media--or acquire a power it doesn't deserve in the hands of tort lawyers and activists whose central motive may be unrelated to improving public health.

The JAMA article doesn't address those phenomena. But Dr. Ioannidis's modest reminder about his own research--mine is not the final word--sets a great example for those who look to science for easy answers and quick fixes.