PhilSci Archive

Statistical Significance and its Critics: Practicing damaging science, or damaging scientific practice?

Mayo, Deborah and Hand, David (2022) Statistical Significance and its Critics: Practicing damaging science, or damaging scientific practice? [Preprint]


Download (602kB) | Preview


While the common procedure of statistical significance testing and its accompanying concept of p-values have long been surrounded by controversy, renewed concern has been triggered by the replication crisis in science. Many blame statistical significance tests themselves, and some regard them as sufficiently damaging to scientific practice as to warrant being abandoned. We take a contrary position, arguing that the central criticisms arise from misunderstanding and misusing the statistical tools, and that in fact the purported remedies themselves risk damaging science. We argue that banning the use of p-value thresholds in interpreting data does not diminish but rather exacerbates data-dredging and biasing selection effects. If an account cannot specify outcomes that will not be allowed to count as evidence for a claim—if all thresholds are abandoned—then there is no test of that claim. The contributions of this paper are:
• To explain the rival statistical philosophies underlying the ongoing controversy;
• To elucidate and reinterpret statistical significance tests, and explain how this reinterpretation ameliorates common misuses and misinterpretations;
• To argue why recent recommendations to replace, abandon, or retire statistical significance undermine a central function of statistics in science: to test whether observed patterns in the data are genuine or due to background variability.

Export/Citation: EndNote | BibTeX | Dublin Core | ASCII/Text Citation (Chicago) | HTML Citation | OpenURL
Social Networking:
Share |

Item Type: Preprint
Keywords: data-dredging, error probabilities, Fisher, Neyman and Pearson, p-values, statistical significance tests
Subjects: Specific Sciences > Probability/Statistics
Depositing User: Prof Deborah Mayo
Date Deposited: 21 Apr 2022 04:08
Last Modified: 24 Apr 2022 21:15
Item ID: 20482
Subjects: Specific Sciences > Probability/Statistics
Date: 18 April 2022

Monthly Views for the past 3 years

Monthly Downloads for the past 3 years

Plum Analytics

Actions (login required)

View Item View Item