“An irreproducibility crisis afflicts a wide range of scientific and social-scientific disciplines, from public health to social psychology. Far too frequently, scientists cannot replicate claims made in published research … This poses serious questions for policymakers. How many federal regulations reflect irreproducible, flawed, and unsound research? How many grant dollars have funded irreproducible research? In short, how many government regulations based on irreproducible claims harm the common good?”

So begins the introduction of a National Association of Scholars (NAS) white paper, “Shifting Sands: Unsound Science and Unsafe Regulation.” The authors are David Randall, a Center board member and director of research at NAS; Warren Kindzierski, adjunct professor in the School of Public Health at the University of Alberta-Edmonton; and Stanley Young, director of the Shifting Sands Project. 

This paper is part of an ongoing NAS project to examine how irreproducible scientific research negatively affects government policy and regulation. It focuses on irreproducible research associating airborne fine particulate matter smaller than 2.5 microns in diameter (PM2.5) with health effects, including mortality, heart attacks, and asthma. The EPA relied heavily on these analyses while considering regulations.

Using a straightforward statistical test called Multiple Testing and Multiple Modeling (MTMM) to assess the credibility of the literature on PM2.5 in regard to mortality, asthma, and heart attacks, the authors found strong evidence these studies had indeed been influenced by publication bias, p-hacking, and/or HARKing (Hypothesizing After the Results were Known). In other words, these studies should have been viewed by the EPA as insufficient to inform regulatory policy, but weren’t.

The authors wrote, “The EPA might not have regulated PM2.5 at all if they had applied more rigorous scientific reproducibility requirements to research that they used to justify their regulations.”

This NAS report highlights a problem that permeates the scientific community. There is intense pressure on scientists to publish the results of their research. And scientific publications, as well as traditional and social media, are more interested in bold new findings of danger or cures. It encourages researchers and journalists to exaggerate results and positive claims, even if they are possibly false, or worse, fraudulent. The latest kerfuffle over the presence of PFAS in cosmetic products is a case in point.

But what the media—not to mention regulators and ambitious trial attorneys—overlook is whether the research has been reproduced, or more importantly, whether it is retracted later on because scientists failed to replicate the results. This leaves consumers remembering the scientific sizzle but not the steak, and often results in unnecessary fear, unwarranted regulation, and unjustified litigation.

As the NAS report concludes, “The very best science and research procedures involve building evidence on the solid rock of transparent, reproducible, and reproduced scientific inquiry; not on shifting sands.” Our thanks to board member David Randall’s contribution to this report, and for his tireless leadership of the Center’s efforts to address this critically important issue.