The fields of psychology and cognitive neuroscience have had some rough sledding in recent years. (via Many Neuroscience Studies May Be Based on Bad Statistics | Wired Science | Wired.com)
- The bumps have come from high-profile fraudsters, concerns about findings that can’t be replicated, and criticism from within the scientific ranks about shoddy statistics. A new study adds to these woes, suggesting that a wide range of neuroscience studies lack the statistical power to back up their findings.
- This problem isn’t just academic. The authors argue that there are real-world consequences, from wasting the lives of lab animals and squandering public funding on unreliable studies, to potentially stopping clinical trials with human patients prematurely (or not stopping them soon enough).
- “This paper should help by revealing exactly how bad things have gotten,” said Hal Pashler, a psychologist at the University of California, San Diego. Pashler was not involved with the new study, but he and colleagues have previously raised concerns about statistical problems with fMRI brain scan studies in human subjects.
-
Many researchers consider a statistical power of 80 percent to be a desirable goal in designing a study. At that level, if an effect of a particular size were genuine, the study would detect it 80 percent of the time.
But roughly half of the neuroscience studies Munafò and colleagues included in their analysis had a statistical power below 20 percent. Those studies would fail to detect a genuine effect at least 80 percent of the time.
The raw material for the study was 49 meta-analyses, or studies that analyze data from other studies — 730 individual neuroscience studies in this case — published in 2011. The team concludes that most of the reported findings may not be reliable.
For human neuroimaging studies, the median statistical power was just 8 percent, meaning that half the studies were below this mark and half were above. In two different types of animal studies typically used to study memory, median statistical power was 18 percent and 31 percent, respectively, the teamreported last week in Nature Reviews Neuroscience, which has made the paper open access for one week, starting today.
