J. Leslie Glick Ph.D. Independent Corporate Management Advisor
Much Needs to Be Done to Reduce Inappropriate Research Practices and the Associated Waste of Funds
In a recent GEN article (June 15, 2015, “Copy Me If you Can!”), Sean Muthian, Ph.D., director of strategic marketing and collaborations at Sigma-Aldrich, addressed an issue that has profound implications for the biomedical research community—the irreproducibility of research results. Dr. Muthian cited reports providing evidence that the majority of published research findings could not be replicated, thereby resulting in tens of billions of dollars of misspent funds.
In the online sequel to his GEN article, Dr. Muthian outlined a series of steps that could be taken to mitigate this problem, such as “rigorous statistical analysis”, “transparency in reporting”, and “data and material sharing”. He also discussed the variables that could undermine the accuracy of research results in both cell culture and animal studies and the need to validate reagents. His recommendations are all valid, but much more needs to be done to impact meaningfully on reducing inappropriate research practices and the associated waste of funds.
Forty years ago at conferences of the American Association for Cancer Research and of the New York Academy of Sciences, I first proposed that scientific research should be monitored by data audits conducted by independent auditing firms. Like financial audits, data audits would employ sampling techniques. They would not be analyzing all the research data of every scientist in an organization.
Later on, from 1989 to 1994, in the journal, Accountability in Research, I published a series of articles that dealt with the mechanics of how one conducts a data audit, the estimated prevalence of scientists indulging in questionable research practices (such as misrepresentation of data, inaccurate reporting, and fabrication of results), the receptivity of scientists to being audited, the experiences of some biotech companies practicing data audit, and estimates of the cost effectiveness of data audit in reducing questionable research practices.
In 1992, based on the then available data suggesting that 10–20% of all R&D involved questionable research practices, I estimated that data audits would result in this percentage being reduced to 5–10%. The estimate for the cost of such audits was only 1% of the funds spent on R&D. In other words, the costs of funding questionable research practices would be reduced by $5–10 for each dollar spent on data audits.
At the time, $150 billion were spent on R&D by both the private and public sectors in the U.S. Then in the complete absence of data audits, $15–30 billion would be wasted on funding questionable research practices. Data auditing of all U.S. organizations conducting R&D would, however, presumably reduce this waste to $7.5–15 billion. The cost of such auditing would amount to $1.5 billion, thereby freeing up $6–13.5 billion to fund productive R&D.
Returning to Dr. Muthian’s GEN article, it is clear that questionable research practices have ballooned since the early 1990s. Of the $59 billion spent by OECD countries on biomedical research in 2012, as much as 89% or over $52 billion might have been wasted on irreproducible studies. Other estimates fall as low as 51%, but that still amounts to $30 billion of wasted research funds.
Even back in 1993, at the Third Conference on Research Polices and Quality Assurance, held in Baltimore, Maryland, a survey of laboratory researchers conducted by Professor Adil E. Shamoo and I revealed that 86% of the respondents had personally suspected or determined that one or more other scientists had engaged in questionable research practices.
Prof. Shamoo and I had previously proposed that the research community develop what might be called Good Research Practices (GRP) guidelines, which would be somewhat analogous to Good Laboratory Practices (GLP) regulations promulgated by the U.S. Food and Drug Administration (FDA). Laboratories complying with GLP regulations were also subject to data audits. FDA began implementing data audits of clinical research labs in 1977. As a result of such data auditing, the incidence of serious deficiencies in clinical trials dropped significantly, from 12% to 6%, over the period from 1977 to 1989.
Similarly, over a period of 7 years following the implementation of data audits of National Cancer Institute-sponsored clinical trials by peer groups, the incidence of deficiencies was markedly reduced. Thus, knowing that ones data were subject to audit acted as a deterrent to engaging in questionable research practices, regardless whether the audit was conducted by the government or by peers.
In the survey mentioned above, 69% of the respondents thought that GRP guidelines should be developed, compared to 3% who thought not, and 28% who were undecided. Only 11% thought the guidelines should be in the form of federal regulations, compared to 72% who thought not, and 17% who were undecided. Indeed, while there was a consensus that the guidelines should be developed by some mix of the following entities—scientific societies, universities, ad hoc groups, and the federal government, only 31% preferred a role for the federal government, compared to 67%, 53%, and 42% preferring the involvement of scientific societies, universities, and ad hoc groups, respectively.
I conducted another, smaller survey in 1993, concerning the incidence of data audits of unregulated R&D in biotech companies. Of 15 such companies that I contacted, 9 of them (60%) reported that they had experienced data audits, all of which the companies voluntarily authorized. This was in complete contrast to academic labs, where routine data audits were nonexistent. Of the nine companies that experienced data audits, eight of them reported that data auditing revealed incidents of nonfraudulent, questionable research practices, and two of those eight also reported that data auditing revealed incidents of scientific fraud.
More recently, it appears that data auditing is beginning to penetrate the academic community. Last year, Science Foundation Ireland decided to implement a policy requiring all institutions it funds to be subject to routine data audits. Also last year, in Japan, Waseda University announced that it would begin vetting for signs of research misconduct every doctoral dissertation from all its science departments. Whether these two examples represent the wave of the future or are just exceptions that prove the rule remains to be seen. Clearly, though, as discussed earlier, data audits are an effective deterrent to scientific misconduct.
J. Leslie Glick, Ph.D. ([email protected]), is an independent corporate management advisor.