November 15, 2016 (Vol. 36, No. 20)
In Terms of Reproducibility, All Labs Are Above Average—or So You Might Think
Estimates of reproducibility in the life sciences vary, but they tend to support the same conclusion—reproducibility is at an all-time low. That’s the bad news.
The good news is that much of the reproducibility crisis has a known cause: uncertainties surrounding cell lines. In other words, researchers in the life sciences know, or should know, that they need to authenticate their cell lines.
Cells advertised as “X” may be overgrown by or contaminated with cells from a different line or species. Alternatively, cells may be corrupted by genetic drift. They may simply be unreliable.
Still, it is clear what researchers need: cell-line authentication (CLA). Even better, the means to satisfy this need are at hand. Besides venerable CLA techniques such as isoenzyme analysis and karyotyping, there are newer approaches such as short tandem repeat (STR) analysis, DNA sequencing, and DNA barcoding.
With all these techniques, CLA should become routine and inexpensive. And yet it remains a problem. CLA, according to the experts interviewed for this article, is not so much a scientific or technical issue as it is a matter of perception. Even the researcher who is quick to acknowledge that the CLA problem is real, in general, may be slow to recognize that it exists in his or her own laboratory.
“This way of thinking intrigues me,” says Douglas Storts, Ph.D., head of research for nucleic acid technologies at Promega. “Even the cost of media and plasticware to grow the wrong cell line exceeds the expense of sending cells out for authentication.”
STR Analysis
Developed for forensic applications, STR analysis has become the go-to method for ensuring the identity, homogeneity, and purity of human cell lines. STR analysis systems include Promega’s GenePrint® Systems. The GenePrint 10 system analyzes loci recommended in an ANSI standard for human CLA (ASN-0002). In addition to the nine loci recommended in that standard, GenePrint 10 includes a probe for the sex marker amelogenin. GenePrint 24 consists of 22 STR loci, amelogenin, and another marker, DYS391, which targets a locus on the Y chromosome.
“The more loci the better the discrimination,” says Dr. Storts. “What sometimes occurs with cultured cell lines is the inability to detect amelogenin due to mutations occurring at the primer binding site, or deletions of the amelogenin region. DYS391 serves as a redundant marker for sex determination.”
STR analysis is suitable for authenticating human stem cells and their progeny in addition to other human lines. After cells differentiate, the genome remains intact even though gene expression changes. Dr. Storts advises that while genetic drift is unlikely, stem cell–derived products should still be authenticated to identify possible contamination.
This contrasts with immortalized cells, where chromosomal rearrangements may result in altered genotypes. “The authentication standards are written, so there’s some play to account for drift,” Dr. Storts explains. A “mere” 80% alignment with the test line (or in the case of stem cells the original donor) is not necessarily a negative, since it accounts for the instability associated with immortalization.
Range of Techniques
A review published in 2007 stated that as much as 36% of cell lines might be misidentified or cross-contaminated. (This figure, for the prevalence of cell-line misidentification from 1968 to 2007, has since been widely cited.) More recently, a study appearing in 2013 reported that only 43% of cell lines could be uniquely identified.
Figures such as these indicate that CLA remains crucial to understanding and documenting materials used in biopharmaceutical production. “It applies for contamination as well as basic verification of cell-line identity,” says Deborah Lee Dormady Letham, Ph.D., scientist II, Charles River Laboratories. “And it can be used to exclude potential recombinant DNA/genome rearrangements.”
Technologies for authenticating cells have moved beyond isoenzyme analysis, which is, according to some vendors, all but defunct. Technologies now include short tandem repeat (STR) profiling, DNA sequencing, and DNA barcoding. Also, some companies continue to use classic karyology methods, which examine chromosomal structure for species-level identity, in-depth phenotypic panels, and biomarker analysis targeting species-specific RNA transcripts.
“DNA sequencing is more cost- and time-effective, and useful for both top-level species characterization and very deep analysis,” Dr. Letham adds. “Although deep whole-genome or transcriptome sequencing is certainly the way of the future, traditional Sanger sequencing for multi-locus analysis is still viable and cost effective.” In general, however, analysis cost dictates which methods scientists use and the extent to which they characterize cells.
Charles River Laboratories uses both updated technology and the older methods for CLA. “Although isoenzyme testing has largely been discontinued,” notes Dr. Letham, “some clients still feel that regulations limit them to pursuing that methodology.”
Although CLA methods proliferate and evolve, they must continue to focus on answering basic identity questions. “CLA must be founded on solid science and all information,” Dr. Letham insists. “The old adage—garbage in, garbage out—still holds.
Expanding to Nonhuman Lines
In preclinical research, in terms of demand, mouse cell lines are second only to human cell lines. Characterizing murine lines is more difficult than the authentication of human cells, explains Yvonne Reid, Ph.D., manager and scientist at ATCC (American Type Culture Collection): “Mouse cells often derive from inbred strains, and they may experience significant genetic drift in culture, so it’s important not just to identify species, but to drill down to the individual cell-line level.”
ATCC is collaborating with NIST (National Institute for Standards and Technology), which has developed STR markers to identify different mouse cell-line strains, down to the individual cell-line level. The collaboration involves basic research, validation of STR markers, and eventually the development of a written consensus standard.
During the collaboration, the partners hope to create a consortium of scientists for validating NIST’s STR markers. The idea is to move toward reproducibility. The ultimate goal is a consensus standard.
The variable nature of STR regions analyzed for CLA allows for discrimination among individual lines. The likelihood of two murine lines having identical DNA profiles at nine loci is around one in five million. By analyzing a greater number of loci, investigators will be able to drill down to even more subtle differences within strains.
Dr. Reid views emerging standards to require CLA as a huge positive that will improve rigor in biology. More than 300 scientific associations, societies, and journals have addressed the need to categorize cell-line source and carry out authentication and mycoplasma testing. In addition, the NIH incorporated CLA provisions in its granting policies as of January 2016.
The ATCC is currently working on its fourth consensus standard for CLA. “A consensus standard is different from a published protocol because it undergoes validation,” explains Dr. Reid. “Consequently, a consensus standard is more robust and more likely to allow different laboratories to corroborate each others’ results.”
A Vendor’s Advice
MilliporeSigma provides a range of biological reagents, including media, sera, and internally manufactured cell lines. MilliporeSigma collaborates with academic colleagues to commercialize research enabling cell lines, and has developed an in-house CLA process that includes karyotyping, bioburden testing, mycoplasma testing, STR for human cell lines, and species identification to eliminate the possibility of cross-contamination.
“We assure that we have verified identity and screened for any potential contaminants before cells reach our customers,” comments Genova Richardson, head of MilliporeSigma’s cell biology franchise. “Customers face enough of a challenge with reproducibility at the assay level, so we want to minimize potential issues from cells.”
Richardson advises customers to use low passage numbers in cell lines to minimize the impact of genetic drift: “Always obtain cells from a qualified cell bank to assure that the line is authenticated and low passage.” Otherwise, customers need to recharacterize cells through STR, and they have to perform mycoplasma testing to ensure their labs are not introducing that contaminant.
Beyond Cells
The Global Biological Standards Institute (GBSI) is a nonprofit advocacy and policy organization focusing on standards and best practices in basic research. “Standards and SOPs exist in industry and clinical medicine, but there’s very little in basic research,” explains GBSI founder Leonard Freedman, Ph.D. “When we began, the issue of data reproducibility was emerging as a crisis in research.”
Standards and validation guidelines for CLA won’t solve every problem in the reproducibility space, but it could contribute significantly to the solution. “Standards per se can be boring,” admits Dr. Freedman. “But the topic is beginning to resonate with researchers.”
In addition to CLA and issues surrounding misidentification, The GBSI is involved in the validation of research-grade antibodies, whose authentication and reliability Dr. Freedman describes as the “wild west.” In late September 2016, the GBSI held a conference on antibody standards, attended by more than 100 experts, on this very topic. The challenge for these reagents is the lack of a unifying set of guidelines, and the disparity of what is required for disparate applications such as flow cytometry and immunoassays.
Where an established, validated, ISO/ANSI-endorsed method exists for CLA for human cell lines in the form of STR analysis, nothing similar is available for antibodies, Dr. Freedman complains. “The issues around CLA are more granular.”
For CLA, researchers can rely on publicly available databases. Contract research organizations provide authentication services, and the costs are modest as compared with the overall costs of conducting biomedical research. A few universities even maintain core facilities that handle the authentication needs of its own researchers.
“The good news is there’s an accepted way to get CLA done,” Dr. Freedman notes. An accessible standard already exists, and since it was adopted three or four years ago, additional techniques such as single-nucleotide polymorphism analysis have emerged.
The journals Nature and Science are on board; the NIH’s “rigor and reproducibility” initiative is already making a difference; and funding agencies and leading journals now routinely ask for authentication assurances in grant applications and manuscripts. “They don’t ask for any sort of certification,” declares Dr. Freedman, “but they should!”
This authentication policy is being observed by a broader set of journal publishers, a trend that coincides with a salutary change cited by Genova Richardson: “Early publications claimed that up to 36% of cell lines used in biomedical research were misidentified or cross-contaminated; our internal estimate is that this is now closer to 10%.”
Vendors have simultaneously improved communication with customers regarding assays to use with specific cells. “This represents something of a shift,” observes Richardson. “Now that CLA is accepted, researchers are more interested in assuring that assays and cell lines are appropriate for the biological questions that are being asked.”
Richardson also notes a greater awareness of assay robustness, particularly with respect to the reproducibility of results among laboratories. Authentication is the first key, but robust protocols require methods, reagents, and techniques that are easily transferable.
Richardson acknowledges that “it’s a lot easier to walk down the hallway to get cells,” but warns that this approach may be costly in the long run. “It’s cheap now to borrow, but it can get very expensive if you subsequently need to authenticate for data publication.”
Richardson has an interesting take on the low compliance rate: “I recognize that many groups aren’t doing authentication, but a lot of repeat customers are coming back to us. You have to take the 30% figure and qualify it against customers who are already buying cells from reputable vendors.” Still, she says, compliance in academic laboratories lags behind the pharmaceutical and biopharmaceutical industries.
Now for the bad news. A recent GBSI survey of 500 respondents shows that despite the existence of a readily available standard, and that researchers recognize the importance of CLA, the compliance rate is less than 30%.
Dr. Freedman places some of the blame on skewed incentives entrenched in academia, as described by consultant Glenn Begley, Ph.D. in articles and talks. Specifically, junior researchers operating in a publish-or-perish environment too readily accept results that are interesting but possibly bogus.
“It’s a mindset that is baked into life as a researcher, especially in academia,” Dr. Freedman tells GEN. “Human nature is human nature.” Yet, despite statistics showing that as much as 60% of biomedical research is not reproducible, he believes that outright fraud is rare.