Leonard Freedman, Ph.D. President Global Biological Standards Institute
GBSI Is Taking Action to Reduce the Widespread and Systemic Irreproducibility of Research Findings
By now, most scientists are familiar with the global and enormously expensive problem of irreproducible biomedical research. Conservatively, 50% or more of preclinical research cannot be reproduced. We recently showed in a study published in PLOS Biology that this corresponds to the inefficient use of approximately $28B in funding dollars annually in the United States alone1. Flawed biomedical studies point to systemic inefficiencies in the way preclinical studies are designed, conducted, or reported. And this in turn delays the drug discovery process and, more importantly, creates false hope for patients anxiously waiting for diagnostics, therapies, and cures2.
Excluding scientific misconduct, which does not, thankfully, appear to be a major source of irreproducible research3, our analysis demonstrates that irreproducibility typically results from cumulative errors or flaws in one or more of four (non-discrete) categories of the research and publication process: biological reagents and reference materials (36% of total), study design (28%), data analysis and reporting (25%), and laboratory protocols (11%)1. Although each contributes to an exceedingly complex, systemic set of problems as well as challenges to correct them, our research indicates that considerable recovery of lost research and dollars can be gleaned from the systematic application of best practices and standards as they apply to biological reagents. These include cell lines4, antibodies5, nucleic acid reagents, etc.
The low-hanging fruit in this context are immortalized mammalian cell lines—workhorses in the lab for decades in basic and translational biomedical research. Cell lines are repeatedly cultured, frozen, passaged, and processed while QC procedures and systems vary dramatically between laboratories. Moreover, sharing cell lines in and among labs is endemic. As a result, cell line misidentification and cross-contamination (most commonly by HeLa) as well as microbial contamination (most commonly by mycoplasma and other bacteria) of cell systems continue to be major—and costly—contributors to research reproducibility problems6,7. How widespread is the problem? A widely cited review estimated that between 18% and 36% of cell lines might be misidentified or cross-contaminated8, while a very recent estimate by University of Colorado Geneticist Christopher Korch places the figure at 20%9.
Using just two well-known misidentified cell lines, HEp-2 and INT 407, Korch estimates that over 7,000 articles have been published that may have inappropriately used one or both cell lines at a total estimated cost of over $700M. Further, those articles produced over 200,000 citations with an estimated collective cost of $3.5B!
And yet a straightforward, low-cost (approximately $150) standard for authenticating human cell lines is widely available to research labs across the globe. This test, short tandem repeat (STR) profiling, can immediately convey key data as to the identity of given cell line. However, the use and even basic awareness of the STR standard (and the use of single-nucleotide polymorphism [SNP] analysis10), as well as general cell culture best practices, remain shockingly low. One survey reported that just one-third of laboratories tested their cell lines for identity11, despite multiple calls for expanded use of authentication by journals in their reporting guidelines.
The Global Biological Standards Institute (GBSI) recently concluded a detailed, online survey to analyze why cell authentication and the STR standard specifically is not used more broadly, the results of which will be published and presented later in 2015. We believe that understanding the barriers to cell authentication will be key to informing future policies and changing the research culture.
The National Institutes of Health (NIH) currently fund approximately 9,000 projects and sub-projects using cell lines, at a total annual spend of $3.7B. For an NIH-funded academic researcher receiving an average $450,000, 4-year grant, purchasing cell lines from a reputable vendor (or ensuring the identity and contaminant-free status of their own or borrowed stock) and then authenticating annually will only cost about $1,000 or 0.2% of the award. Given that approximately a quarter of these studies are likely using misidentified or contaminated cell lines, reducing this to even 10% through a broader application of the STR standard—a very realistic goal—would ensure a more effective use of nearly a billion dollars each year.
Unlike other complicated contributors to the reproducibility issue, here is a case where an inexpensive identification assay exists that will ultimately save a lot of money. Nevertheless, fixing this problem demands a systematic approach to raise awareness of the problem and its solutions.
We urge readers to participate in GBSI’s #authenticate campaign (www.gbsi.org/work/authenticate), a dynamic campaign to change the fundamental way that researchers approach cell culture and authentication. Major disease funders and almost 100 scientists have signed on with GBSI to advance the use of authentic, contaminant free cell lines. One important goal of the campaign is targeted training and education. At present, there is little or no standardized training on cell culture practices and authentication, so whether (or not) a young scientist receives the training, skills, and tools they need depends almost entirely on the knowledge and effectiveness of informal training that can be provided by their mentor, lab director, or colleagues. For starters, we advocate that basic cell biology training and mentoring for graduate students and postdoctoral fellows needs to explicitly include the importance of cell culture best practices and authentication.
As part of a much broader educational program aimed at improving the credibility, reproducibility, and translation of life science research, GBSI is developing an exportable “active learning” training module to reduce cell line misidentification, mislabeling, and contamination. GBSI’s cell authentication training module will emphasize highly interactive training units that include “how to” videos that will facilitate turning learning into practice by sending the trainees back into the laboratory to demonstrate and share their skills. To advance this effort, GBSI is building partnerships and developing strategies to disseminate and promote this module to all researchers who use cell lines—particularly graduate students, postdoctoral researchers, and junior faculty. Most importantly, the training will be freely accessible to the research community.
As a result of targeted training, effective policies, and expanded use of standards and best practices for authentication and cell culturing, (1) knowledge of why and how to perform cell authentication will improve, (2) biomedical research reproducibility rates involving cell lines will increase, (3) hundreds of millions of dollars in research expenditures will be used more efficiently, and (4) the translation from bench discoveries to bedside therapies and cures will be accelerated. Considering the vast amounts of research funds spent on cultured cell-based research each year, and its importance in the foundation of discovery of treatments and cures, expanded awareness and adoption of authentication protocols including STR analysis through targeted training is a relatively inexpensive way to considerably increase our annual return on biomedical research investment.
1 Freedman, LP. PLOS Biol. 2015. June; 3 (6): e1002165.
2 Perrin, S. Nature 2014. March; 507: 423.
3 Gunn, W. Nature 2014. January; 505: 483.
4 Freedman, LP. GEN Eng. 2015. December.
5 Bradbury, RM. Nature 2015. February; 518: 27.
6 Freedman, LP. Nature Meth. 2015. June; 12 (6): 493.
7 Lorsch, JR. Science 2014. December; 346 (6216): 1452.
8 Hughes, P. Biotechniq. 2007. November; 43 (5): 575.
9 Neimark, J. Science. 2015. February; 347 (6225): 938.
10 Yu, M. Nature. 2015. April; 520 (7547): 307.
11 Buehring, GC. In Vitro Cell Dev. Biol. Anim. 2004. July-August; 40 (7): 211.
Leonard P. Freedman, Ph.D. ([email protected]), President Global Biological Standards Institute (GBSI), has over 30 years of research, management, and program development experience in molecular and cell biology, biomedical research, and drug discovery in both the private sector and academia. He is a recognized leader in the field of nuclear hormone receptors.