Center for Truth in Science Board Chair Dr. Richard Williams, the first economist to serve in the Center for Food Safety and Applied Nutrition in the Food and Drug Administration (FDA), shares his thoughts on a recent presentation to FDA economists regarding the use of science in economic benefits analyses.

On December 2, 2020, I was invited to speak to approximately 50 FDA economists.  As the first and, I believe, longest serving economist from FDA, I was happy to oblige, especially since the topic was one that has concerned me over the last several years—the use of science, particularly epidemiology (but also risk assessments), in economic benefits analyses. 

More directly, I asked the question: how frequently, or infrequently, are health and safety benefit predictions based on sound science?

During my 27-year tenure, I wrote and supervised studies that went into cost-benefit analysis in Regulatory Impact Analyses of regulations.  I relied heavily on FDA scientists and risk assessors to provide me with initial estimates of risk we hoped to reduce and, in some cases, with estimates of how much risk would be reduced by various regulatory options.  Over time, the evidence was increasing that many, if not most, of my benefits estimates were off by a considerable amount. In many cases, I suspect costs would have outweighed benefits if the underlying science had been correct.

Economists have a very high bar when it comes to risk analysis, often higher than those lawfully imposed on the FDA by the Administrative Procedures Act.  They cannot simply take a hand-off from scientists and blindly move ahead. If the science used is biased, the benefits and costs will be unbalanced, and the decision makers will unknowingly be left making decisions based on bad information. 

In the presentation, I also noted problems associated with the incentives for scientists in the academy: they must be published for tenure, have positive results, make huge splashes with their findings, and rather than focus on getting the science “right”, the “showy” and “positive” science wins out.

Academic journals are not immune to these ills—they need big, splashy headlines to sell copies and often prefer results that fit within editors’ biases. Just look at what happened with the Lancet this year with the withdrawal of a coronavirus paper.

I also presented numerous quotations from prominent scientists and science journal editors about the state of science, particularly epidemiology used in nutrition and environmental sciences, and clinical studies.  We discussed fraud and, a more prevalent problem, errors in commission and omission in science. 

In the latter category, it has only been recently that there has been an emphasis on reporting null or negative findings.  Previously, those results were either ignored or intentionally omitted. 

For commission, we discussed how the drive to publish positive, high impact studies has led to “p-hacking” where data or theories are adjusted so that a positive finding can come out of a large data set. 

Both these errors present a problem for economists.  They usually do not have the time, budget or, in some cases, the expertise to go back and check every paper or scientific summary they are given for their analysis. 

Finally, I concluded my presentation on a positive note, discussing organizations like the Center for Open Science in Charlottesville, Virginia that has established new standards for scientific publications.  I was honored to discuss these issues with the FDA economists, and they were pleased to have someone who has been in their shoes address issues that are so close to what they do despite the new challenges it presents for them.

A copy of my presentation is available here for review.