The Center for Truth in Science is committed to supporting transparency in scientific research at every level. This includes funding top independent researchers to conduct ‘state of the science’ systematic reviews of the literature to better understand what the science is telling us.
Conducting systematic reviews is especially important for scientific questions where there is not yet consensus or consistency in published results.
Systematic reviews are not new, and have been around since the mid-18th century when one was published on the treatment of scurvy. Beginning with reviews of clinical trials of medical interventions, this methodology has moved into other health and scientific areas, including toxicology and environmental health research.
Currently, there is an abundance of published literature on thousands of scientific questions, and the systematic review method has become instrumental in assimilating the plethora of evidence from published studies into information usable to scientists, policymakers, clinicians, and other consumers of science.
There has been a growing area of guidelines and best practices published by top thinkers across fields that has improved the conduct and reporting of systematic reviews, including those published by the Cochrane Collaboration, World Health Organization, European Food Safety Authority, Agency for Healthcare Research and Quality, and National Research Council, among others.
In spite of this, while the number of published systematic reviews has increased, the rigor is not always there. A 2022 article in Critical Reviews in Toxicology outlines some of the problems, including a failure to follow good practices such as best search strategies, optimal methods for determining and reporting eligibility criteria, and reporting of study quality using reporting checklists.
An essential element in conducting a state of the science systematic review is registering or publishing the study protocol as soon as it is developed, as the work begins. This makes the field aware of the study and allows for comments in order to reduce bias and confounding.
In addition, there is value in the field knowing the study is coming soon, which has the potential to reduce the duplication of efforts and hold the authors accountable to finish the analysis and attempt to get published.
Additional accountability comes from the expectation that if there are any changes as the study evolves, they will be added to the publicly registered protocol. Making the study protocol public allows for the dissemination of current ideas about study design and data analysis on a research question or area. Publishing the protocol in a scientific journal when possible goes a step further and ensures another level of peer review important to improving the science.
The Center is doing its part in a number of ways. During our independent peer review process, experts in the field make sure the elements of a rigorous systematic review are addressed and derive their scores accordingly.
In writing our requests for proposals, moving forward, we are including a requirement that all applicants must agree to register protocols on a public site, such as PROSPERO or the Center for Open Science, if they want to be considered for funding. When possible, we may include a requirement for protocols to be published before moving forward with the review.
We also require our funded research to meet other areas of transparency regarding data and methods to be available, so that other scientists can replicate the reviews. Our formaldehyde research initiative serves as a good example of these criteria.
The research question for the study is as follows: Does inhaled formaldehyde lead to increased risk for myeloid leukemia and other LHP cancers?
In December 2022, the Center issued a Request for Proposals to answer the above question. We received several applications from established research firms in epidemiology, toxicology, and risk assessment. Applicants were required to provide:
- A clear description of the methods of identifying, selecting, and validating information
- How the searching and screening process would be accomplished
- How quality assessments would be done, with which systems, and why
- A plan for meta-analyses, if appropriate
- A plan for publication of results in a peer-reviewed scientific journal, including the transparency necessary for others to run the same analyses, in order to determine replicability of results
In March 2023, the Center selected ToxStrategies LLC to receive a grant to complete a state of the science systematic review, led by Daniele Wikoff and a team of experts in toxicology, epidemiology, formaldehyde, statistics, and systematic review methodology. The independent review committee scored the application high on each of the above criteria, and noted that the team would register its protocol with one of the leading public sites.
In July 2023, the research team announced that its systematic review protocol was publicly available on the Center for Open Science website. We encourage you to explore it yourself as we look forward to the results of this systematic review later this year!