No Good Research
Yesterday the Wall Street Journal informed me that most scientific research is wrong. Specifically, Professor of Medicine John Ioannidis (plus two coauthors) recently published this study in The Journal of the American Medical Association. The paper examines 432 gender-based health claims in 77 articles found on PubMed. They concluded almost all of the claims were tainted by serious flaws in either experiment design or statistical analysis. Apparently Dr. Ioannidis has been researching this problem for some time (apparently this is PLoS’s most downloaded article).
Sadly I’m not surprised that plenty of medical research contains flawed statistical analysis. I often tell people that we should be teaching statistics, not calculus, to all of the AP hungry high school students out there. (Not to mention premeds, they take two semesters of calculus. When was the last time a med student took a derivative?)
What I don’t appreciate is the Wall Streets Journal’s choice of headline: “Most Science Studies Appear to Be Tainted By Sloppy Analysis”. Nor do I appreciate Ioannidis choice of paper title “Why Most Published Research Findings Are False” (this is the title of the PLoS paper, not the JAMA one). It turns out that a lot of scientific research isn’t medical research. In fact it isn’t even health related, or biology related. In my field (theoretical computer science), I am quite confident that most claims are not false. I’m even willing to go so far as to support the hard sciences in general.
I realize that by sheer number of papers, most research is biomedical (NIH’s budget is about 5 times larger than NSFs), but I’m sure anyone reading a phrase like “most published research is false” hears, “most research, in most fields, is false”. Certainly that’s what a biomedical researcher would think, what with their shabby understanding of statistics, and their understanding of stats is probably still above average. Once again the fundamental law of statistics overwhelmingly applies.