In the September 2016 issue of “peer review notes,” the NIH Center for Scientific Review (CSR) shared what it has learned about implementing the new NIH rigor and transparency policies. Since the policies took effect in January 2016, there have been 525 peer review meetings with nearly 10,000 reviewers involved. 

CSR trained peer reviewers to consider the four new criteria meant to enhance the reproducibility of studies: scientific premise, scientific rigor, consideration of relevant biological variables such as sex, and authentication of key biological/chemical resources. Although it found most reviewers “rose to the challenge” of applying these within the peer review process, “work remains.” 

Some reviewers clearly kept the new policies in mind, but used “old language” in their critiques. Others confused “premise” with scientific significance or focused on whether hypotheses of studies were reasonable. NIH’s concept of premise, however, relies on reviewers to determine if studies or preliminary data leading to the proposed work are “scientifically sound.” 

Reviewers raised several questions about the inclusion of sex as a biological variable that must be factored into research design, analyses, and reporting for all vertebrate animal and human studies, unless a compelling scientific argument can be made for not doing so. These included what is an adequate incorporation of sex into a study’s design; can inclusion of both sexes actually reduce scientific rigor by increasing physiologic variability; and if a disease predominantly affects only one sex, is that sufficient justification for a single sex study. 

CSR noted, “There are no blanket answers to such questions at this point,” and called for reviewers to bring their best scientific thinking to the table to achieve the fundamental goal— “replicable, generalizable science.”