OPA Analysis of Nicholson and loannidis dataset

In a recent Comment in Nature, Nicholson and Ioannidis argue that the current peer review system at NIH favors incremental approaches rather than truly innovative research (Conform and be funded, Nature 492, 34-36; 2012 Exit Disclaimer). As David Lipman and I responded in the January 3 issue (NIH funding: agency rebuts critique, Nature 493, 26; 2013 Exit Disclaimer), their methodology is fundamentally flawed and therefore does not provide useful information to assess this issue. As shown below (Table 1 and Figure), over 60% of the 158 highly cited articles included in their analysis do not actually fit the profile of innovative primary research. Of the remaining 58 publications, NIH funded 83%, and private industry funded the other 17%. Finally, a fundamental feature of peer review is that applying for funding is an essential prerequisite to receiving an award; of the first and last authors of the 158 articles analyzed by Nicholson and Ioannidis, 84 (60.4%) have never submitted an NIH application (Table 2). NIH welcomes rigorously conducted analyses about how to encourage further innovation in biomedical research.

George M. Santangelo
Director, Office of Portfolio Analysis
Division of Program Coordination, Planning, and Strategic Initiatives 
National Institutes of Health
george.santangelo@nih.gov

Table 1

Category* Number of Publications Percentage (%)
Biomedical Articles 49 31.0
Reviews 35 22.2
Clinical Trials 29 18.4
Non-Biomedical Science 18 11.4
Reports (e.g. Cancer Statistics)  10 6.3
Clinical Guidelines 6 3.8
Epidemiology 5 3.2
Methods 4 2.5
Resources 2 1.3
Total 158 100.0%

*Categorization of the 158 papers with >1000 citations used by Nicholson and Ioannidis (Conform and be funded. 
  Nature 492:34–36 [2012]) as a representative sample of influential papers published between 2001 and 2012 in the
  life and health sciences.

Category representative of innovative primary research: 82.8% of papers in these three categories (Biomedical
  Articles, Methods, and Epidemiology) were supported by NIH; the remaining 17.2% were supported by industry.

Category that either falls clearly outside the NIH mandate to fund research that has the potential to improve human
  health, or does not fit the profile of innovative primary research.

 

Representation of the Table 1
Representation of the data in Table 1

Table 2

Author position Number of authors not receiving
current NIH funding*
Never applied for
NIH funding
Never applied for
NIH funding in FY11/12
First 76 45 (59.2%) 61 (80.3%)
Last 63 39 (61.9%) 59 (93.7%)
All 139 84 (60.4%) 120 (86.3%)

*Nicholson and Ioannidis reported that 104 of 262, or 39.7%, of eligible first and last authors who did not participate in NIH study sections received current NIH funding. We checked the remaining 158 authors who did not receive current NIH funding and found sixteen that have NIH funding and were evidently misidentified (three of whom are NIH intramural scientists). Three of the remaining 142 authors were duplicates, leaving 139 unique first and last authors without current NIH funding. Of these, 120 (86.3%) did not submit an application for NIH funding within the past 26 months (FY11/FY12), and 84 (60.4%) have never applied for NIH funding. This is not surprising given that many of these authors work in private industry, and/or made advances in areas of study that are at most marginally relevant to the NIH mission.

This page last reviewed on