Can We Believe Scientific Results?

By Shlomo Maital


  The Oct. 19 issue of The Economist has “How Science Goes Wrong” on its cover. It contains a worrisome article that leads off with a quote from Nobel Economics Laureate Daniel Kahneman: “I see a train wreck coming”.  The article deals with the very foundation of credible scientific research: The ability to replicate (repeat) scientific experiments, to verify that the results are true.   It turns out, most scientific publications cannot be replicated.  The Economist reports:

     An American drug company Amgen tried to replicate 53 studies that they considered landmarks in the basic science of cancer.  They were able to replicate the original results in just six.

   What is the problem?  Why can results be reproduced?

   Here is a rather difficult explanation, by The Economist, based on work by Stanford statistician John Ioannidis, an epidemiologist.   Suppose 1 in 10 hypotheses are true.  Consider tests of 1,000 hypotheses, of which 100 are true.  These tests have a 5% false positive rate (5 times in 100, a test says a hypothesis is true when it is false).  So of 900 false hypotheses, 5% x 900 = 45 are proved true. 

    Most tests have a statistical ‘power’ of 0.8, meaning 8 of 10 true hypotheses are proven true.  So only 80 of the 100 true hypotheses are proven true.  This means there are 20 false negatives (true hypotheses proven false). 

     Summary:  80 true hypotheses are proven true; 45 false hypotheses are also proven true.  So 45/120  false hypotheses are said to be true, fully one third. 

   Ironically:  the negative results are far more reliable. But journals hate to publish negative results (i.e. no, broccoli is NOT great for your prostate). 

      At a festive dinner here in Paris for Technion I sat next to a researcher who runs a medical research lab with a one billion euro budget.  He told me of rising pressure to attain results, and collapsing budgets.  There is huge pressure on scientists to publish results, under the threat of grant cancellation.  One of this year’s Nobel Chemistry Laureates said he got no results at all for five years, and if he were repeating this work today, he would have lost his NSF grant long ago.

     Many journalists report scientific research, especially related to food, and many of us take it seriously.  We drink more or less coffee, eat more or less broccoli, based on it.  Perhaps we should stop and just eat and drink what we like.  Why forego coffee for twenty years just to learn the original research was erroneous?