Almost two years ago, a Duke University researcher at the center of a data reliability dispute resigned from the university, with his research group dissolved, its papers retracted, its clinical trials halted, and its sordid story the subject of a “60 Minutes” segment. In response, the Institute of Medicine (IOM) developed guidelines, which were released last month, to help prevent such problems from happening again.
A 20-member ad hoc committee mapped out a detailed process designed to evaluate whether the data and computational steps behind omics tests are sound and whether the tests are ready for use in clinical trials. The process spells out responsibilities and best practices for investigators and others involved in clinical trial use of omics-based tests in a 272-page report titled “IOM’s Evolution of Translational Omics: Lessons Learned, and the Path Forward.”
The problems with data validity exposed by the Duke situation reflects academia’s emphasis on fast publication of observations over their translation into practical discoveries through statistically rigorous trials, contended a principal investigator for seven major clinical trials.
“If we set the threshold so incredibly low, and we don’t do quality control, and we publish things that don’t have statistical rigor, then we end up in the situation that we’re in right now, which is what I’d like to call a highway, but it’s full of roadblocks, or it’s more stopping points than what you’d like to have,” John T. McDevitt, Ph.D., the Brown-Wiess professor of chemistry and bioengineering at Rice University, told GEN.
“We have an information highway. We don’t have a biomarker highway. And we should have a biomarker highway, because then we would be able to tap into all these amazing things that are being discovered right now.”
The rush to publish hasn’t created a similar rush to commercialize new biomarkers, Dr. McDevitt added, given the extra time and work involved in validating them compared to available funding. He hopes greater commercialization of biomarkers will take place through NIH’s new translational research entity, the National Center for Advancing Translational Sciences.
“We’ve got to put the gold medal somewhere, in the validated biomarker, not in the paper or the unvalidated biomarker, because to industry, the unvalidated biomarker is useless. Academia needs to understand this. The sponsors need to understand this. And once there’s more of a focus on that, then there’s more money that’s going into the validation step,” said Dr. McDevitt.
The transparency and sharing recommended by IOM “are just really healthy drivers of validation and innovation, so I really support that recommendation,” Douglas E. Bassett Jr., Ph.D., CSO and chief technology officer for Ingenuity Systems, told GEN. “I think it’s definitely on the right track, and strikes a positive chord with me.” He pointed out that it would add some time and cost to conduct clinical trials, but the benefits yielded in terms of validation would far outweigh the cost.
IOM formed the committee after more than 30 statisticians and bioinformaticians sent a letter in July 2010 to NCI Director Harold E. Varmus, M.D., citing concerns over several genomics-based predictive tests being used in clinical trials at Duke to predict a personalized chemotherapy regimen.
Former Duke researcher Anil Potti, M.D., and colleagues published several papers claiming they could predict lung cancer progression based on gene-expression levels using microarrays. While the research and results were almost immediately questioned, they led to three clinical trials conducted at Duke in 2007 and 2008.
In detailed, if commonsense, recommendations, IOM said researchers should be assured their tests will have “a reasonable chance” of demonstrating clinical validity and utility before beginning further development. “Several criteria should be satisfied and fully disclosed (for example, through publication or patent application) to enable independent verification of the findings,” the report concluded.
Those criteria include data quality control; computational model development and cross-validation; confirmation of the computational model on an independent dataset; and release of data, code, and fully specified computational procedures to the scientific community. An independently managed database such as dbGaP should make available data and metadata used for development of omics-based tests, the report said.
The committee defined omics-based tests as including both an assay and fully specified computational procedures for analyzing assay data. The panel recommended that both be validated before use in a clinical trial. IOM also recommended that omics-based tests not be changed during clinical trials without a protocol amendment and discussion with FDA, with substantive changes possibly requiring that a study be restarted.
The report said institutions and funders can help prevent future debacles like the one at Duke by more closely watching for potential conflicts of interest, addressing questions or criticism about institutional science, and insisting that data be made available at time of publication or, for unpublished work, at end of funding. Journal editors should require lead and senior authors to attest to the integrity of the study, with the co-authors confirming shared responsibility for study integrity, while FDA should develop and finalize a risk-based guidance or regulation on bringing omics-based tests to the agency for review, over and above overseeing laboratory-developed tests.
As IOM noted, Duke allowed Dr. Potti’s work to exist outside of its established structures for reviewing and supervising clinical research, the Duke Cancer Center and the Duke Translational Medicine Institute, through the Institute for Genomic Sciences and Policy (IGSP). In finding facts and furnishing recommendations, both of which it did well, the IOM committee didn’t take the extra and necessary, albeit painful, step of answering why Duke allowed numerous entities to oversee clinical research, except for a weak explanation: “As Califf stated, the IGSP was supposed to be consultative with other research groups within the health sciences, but things got jumbled up.” Robert M. Califf, M.D., is vice chancellor for clinical and translational research at Duke University.
The multiplicity of research entities at Duke likely prevented any one from taking leadership and pursuing answers to numerous questions. Confusion existed over several points including whether researchers should disclose details to conflict-of-interest watchers (yes) and whether FDA rules requiring Investigational Device Exemption applications should apply to use of genomic predictors in clinical studies (yes).
IOM performed a valuable service in developing its guidelines. The rules, however, will prove only as valuable as the extent to which institutions follow them and especially enforce them the next time a research institution allows someone to bypass its established infrastructure and rush in to print faulty findings based on invalid data.