Scientists, Journal Editors, Funders, Antibody Producers, and Others—Shared Their Perspectives on How to Reach Consensus
You may have heard the horror stories. Millions of dollars wasted. Why? An antibody problem emerged during the development of a diagnostic test. Confusion over which treatment would be best for cancer patients. Why? An improperly vetted diagnostic antibody.
Such distressing episodes come from irreproducibility, a growing concern in scientific research. According to recent surveys about the causes of reproducibility problems, researchers are in broad agreement. They all have concerns about the antibodies they use.
To address these concerns, the Global Biological Standards Institute (GBSI) held a workshop September in California to bring together all the stakeholders in the world of research antibodies: scientists, journal editors, funders, private foundations concerned with reproducibility in science, funders of scientific research, and producers of antibodies. The tone of the workshop was always professional. No loud arguments erupted between stakeholders. Everyone worked toward the goal of helping researchers make better choices about which antibodies to use for different applications.
Some debate emerged in the preconference discussions about which antibody type (recombinant, monoclonal, or polyclonal) is “the best.” The consensus emerged that all types of antibodies are important; however, antibodies need to validated for their particular application.
Application-specific guidelines for validation came forward as the overarching principal. For example, a standard for a Western blot may not be the best standard for an immunohistochemical stain, and vice versa.
Transparency was another theme of the workshop, with producers wanting to ensure that the tests used to validate their products were freely available on company websites. In the digital age of message boards and user reviews, the younger generations (usually the ones actually using the antibodies is the lab) are completely comfortable in submitting their data and reviewing the products they use. Additionally, posting of negative results will allow users to see which antibodies do not work for a particular application.
Other topics that surfaced were the need for better training of scientists using antibodies in their studies. One possible method for this is requiring training in the validation of antibodies for scientists, perhaps analogous to the required ethics courses for grad students.
Commercial companies suggested offering training in antibody validation. Journal editors considered requiring proper antibody validation for all papers that they publish. Finally, another idea put forward was to compile a library of tissues or cell lines that would serve as a standard reference for anyone to use for antibody validation.
Transparency and Its Limits
“Transparency is definitely a value embraced by our company,” expounded Matt Baker, director, R&D and business development, antibodies and immunoassays, Thermo Fisher Scientific. “We completely back the idea of sharing with everyone the information we gain while validating antibodies, complete with all the technical details. For example, even subtle differences, like a Western blot done with a nitrocellulose membrane, can provide significant variability from one done with a polyvinylidene difluoride membrane.
“Additionally, the samples against which we test antibodies are critical. Having a reference library of source material, such as cell lines with specific genes knocked out via CRISPR, would be wonderful. Other sets of collective standards could be fixed tissues from knockout lines of mice with appropriate controls.
“Thermo Fisher Scientific intends to grow its antibody selection while expanding antibody specificity testing. Hence, attending this workshop is key to ensuring that our standards are aligned with the broader community.
“While consensus building can be a challenge, we see it as the way forward,” continued Baker. “This workshop facilitates the answer to what are the really important qualities of antibody validation. Fortunately, everyone has the same goal: Make certain that antibodies are measuring what we think they are measuring. Thermo Fisher Scientific fully supports the recent Nature Methods paper on the conceptual ‘five pillars’ of antibody validation: genetic strategies, orthogonal strategies, independent antibody strategies, expression of tagged proteins, and immunocapture followed by mass spectroscopy.
“An analogy for antibody standards are nutritional labels, which are required on all processed foods sold in the United States. It takes a while to get to know what they mean and how to read them, but once users become familiar with them, the labels are easy to understand. This standardization creates familiarity over time.
“As we speak more and more to researchers, we find a common request is the need for more training in antibody applications,” stated Baker. Some researchers don’t obtain formal training using antibodies, and they are very interested in participating in proficiency programs, a theme that was repeated multiple times during the conference.
“With over 100 scientists dedicated to validating antibodies, Thermo Fisher Scientific has a depth of knowledge that we can share with the research community, helping scientists test and optimize antibodies for their specific experiments,” Baker noted.
The training of scientist in antibody validation was also touched upon by Achim Knappik, Ph.D., group R&D manager, life science group, Bio-Rad Laboratories. “There are several areas that need to be improved in antibody validation,” Dr. Knappik stated. “First, standards need to be agreed upon. Second, data on antibody characterization should be publicly available. Third, we must educate buyers on the strengths and pitfalls of antibody validation.
“Also, a standardized repository of validation tools, like cells and tissue lysates, would be great, because then we could compare results more directly between different antibodies. I echo my colleagues’ calls for a set of cell lines that could be used for testing or a collection of knockout mouse tissues where we could look for a reduction in levels of a protein.
“While transparency is great, certain information would be currently disadvantageous for us to reveal,” Dr. Knappik insisted. “Specifically, the precise amino acid sequence of an antibody would allow a competitor to easily make the same or a very similar antibody.”
Not Playing the Blame Game, But …
“Organizing opinion leaders from academia, industry, scientific journals, and funders is no small feat,” said Karen Padgett, vp, antibody business unit, Bio-Techne. “The presence of stakeholders in one room calls attention to how important the concepts of antibody and reagent validation are to the reproducibility of scientific research. We know that well-validated antibodies are a fundamental part of scientific exploration and accurate data interpretation.
“I believe Bio-Techne’s current validation methods meet industry standards, especially because we also serve diagnostic customers and have rigorous quality standards. However, we are glad to have a voice at the table and hope a consensus on standards will be reached, as it will ultimately improve the quality of scientific discovery.
“As part of this discussion on reproducibility, we have to consider what we cannot fix,” continued Padgett. “Antibody providers can provide highly validated and specific antibodies to scientists, but we cannot control how these antibodies are used. What I mean by that is that we cannot force scientists to use standardized protocols, since experimental systems differ. We also cannot ensure everyone using antibodies is properly trained on antibody-based techniques, or using proper controls, nor can we do much about limited reporting of reagent protocols in published articles.
“To drive our field forward and properly address this question of reproducibility, an orchestrated effort is required by the entire scientific community, which includes antibody developers, bench scientists, scientific journals, and funders,” Padgett concluded. “It is our responsibility as antibody developers to ensure we are providing excellent quality antibodies to generate reproducible results—we take that responsibility very seriously.”
Community and Consensus
“We all want reproducible science. Science that cannot be reproduced often traces back to not using the proper antibody or using an antibody improperly,” stated Roberto Polakiewicz, Ph.D., CSO, Cell Signaling Technology. “We want to enable good science by all agreeing on a set of antibody validation standards reached by consensus.
“I do have a concern about getting the community to adopt the standards,” continued Dr. Polakiewicz. “We can always write a white paper, but then nothing happens. One aspect that needs to be addressed is the education of researchers in the process of antibody validation.
“This training can be promoted by entities such as the NIH. The NIH could encourage all higher education centers with training grants for graduate [to] students require a program on antibody validation, similar to the programs in place requiring ethics training for all grad students.
“As we progress through the process [of] determining how to validate antibodies, we could potentially offer a certification program for antibody vendors analogous to the ISO9000 certification for industrial standards or LEED certification for green buildings. Of course, it will be a while before this happens.
“Another potential concern is how to rate the performance of individual antibodies. It is unclear who would be rating them, and it is possible that reviewers may not be adept at antibody validation. Unfortunately, this problem then cascades into the publication of scientific papers where an antibody is used to demonstrate something that it does not.”
Dr. Polakiewicz asserted that the standards followed by Cell Signaling Technology are rigorous: “Ninety-five percent of all the antibodies found in our catalog are developed by us. One hundred percent are tested by us for use in Western blot, immunofluorescent microscopy, as well as less popular approaches like ChIP (chromatin immunoprecipitation) analysis and flow cytometry.
“We employ systems to knockdown or knockout expression in cells with shRNA and CRISPR. Once developed, before antibodies are even put into market, a production team must be able to replicate the initial results before the antibody is sold.
“Reputable businesses should follow exacting testing and validating processes. Cell Signaling Technology was the first company to put validation data on their spec sheets; providing the data to support our claims for our antibodies has been done for years. We welcome this initiative. There will not be a problem in compliance for us; we want to encourage others to do the same.”
User Reviews
Some companies are already responding to the need for better validated antibodies on the market, and they are doing so by posting user reviews on their websites.
“We are curating our catalog,” stated Alejandra Solache, Ph.D., head of reagents product development and manufacturing at Abcam. “We are removing the duplicates and those antibodies that don’t perform as expected, and we are leaving in the best antibodies.”
For each antibody, if available, Abcam’s website has verified user-reviews (Abreviews), typically with images of popular applications: immunohistochemistry, immunocytochemistry, Western blot, flow cytometry, and ChIP.
“Presently, we have approximately 35,000 reviews of our antibodies on Abcam’s website,” explains Dr. Solache. “We speak to about 120,000 scientists per year about our antibodies. So we have a good sense of what kinds of questions researchers are asking.
“User reviews are a great way for scientists to see how well a product performs in the hands of their peers. This is really useful when supplemented with our own extensive antibody validation data, making it easier for scientists to choose the best product for their research needs.”
Room for Flexibility
“Most antibody applications that are being discussed at this workshop involve using antibodies as reagents in research, not necessarily for diagnostics or therapeutics. Antibodies used in research still need to be validated for their intended use,” remarked W. Blaine Stine, Ph.D., associate director, AbbVie Global Biologics. “Biopharmaceutical companies such as AbbVie are typically both consumers and producers of research antibodies to support internal research and discovery efforts. We share concerns and desires to ensure commercial antibodies are well-validated.
“Looking forward, we hope the guidelines proposed at this workshop are broadly applicable and meaningful. We want the guidelines to be useful and broadly applicable, but not be overly proscriptive and narrow.
“In general, all attendees (researchers, antibody producers, journal editors) should be commended for working toward the best possible goal: getting good antibody validation standards.”
In additional to not wanting overly restrictive guidelines for validation of antibodies, Dr. Stine also thinks that scientists who rely on antibodies would also benefit from training on best practices. This could help avoid common pitfalls and misinterpretation of results.
“Not properly validating antibodies is thought to be one of the key reproducibility issues in basic science,” concluded Dr. Stine. “If investigators are better trained and there are meaningful guidelines in place for antibody validation, this will help improve the quality of research and accelerate discovery.”
The Drive for Antibody Standards—Time to Herd the Cats
Leonard P. Freedman, Ph.D.Imagine that you have been given the important task of developing an immunodetection assay for an emerging, potentially lethal pathogen, perhaps a variant strain of the Zika virus. You purchase an antibody from a producer company whose catalogue describes a sensitive and selective antibody to a key antigen of the virus, and you publish a paper describing the assay—fast, easy, and inexpensive—in a leading journal, one with a single-word title. The publication is in turn picked up by leading newspapers and online sites throughout the world.
A month later, reports begin to emerge from labs using what appear to be the same antibody (but perhaps sold by a different vendor) that do not detect the protein under the same assay conditions you’ve described in the paper. Whose results are valid? How can they be interpreted?
This scenario and the questions that it raises are all too common. Research antibodies are indispensable biological reagents in basic and translational laboratories as well as in clinical settings around the world. They are used in a wide variety of biomedical applications and assays. Such assays posted global sales of $1.6 billion in 2011 and are estimated to reach $2.6 billion by 2019.
But antibodies are also a major contributor to the current research reproducibility crisis. A variety of issues—from production variations and resale concerns, to improper or inconsistent validation techniques—can jeopardize even the best-designed biomedical experiments. Moreover, the inability to reproduce biomedical research can often be traced to false-positive or false-negative results produced with antibodies that were not properly validated or used properly for the intended application.
A Workshop Convenes
Leading experts from the international biomedical research community met recently to hash out the fundamentals of what will become the first detailed, standardized guidelines for validating research antibodies. The workshop, entitled “Antibody Validation: Standards, Policies, and Practices” and organized by the Global Biological Standards Institute (GBSI), took place September at the Asilomar Conference Center in California.
More than 100 scientists and researchers spent three days determined to herd the the cats, as it were, by working through the unique complexities presented by research antibodies. The meeting, attended by academics, producer companies, online database representatives, funders, and editors from leading journals, was the most representative gathering of scientific experts ever convened specifically on this topic.
The consensus reached at the Workshop, and the follow-up that is underway, will lead to antibody validation standards and promises to increase reproducibility in preclinical research and accelerate breakthroughs in disease treatments and cures.
The Asilomar workshop was organized around four major themes:
• Creating antibody standards—establishing validation techniques and guidelines for biochemical and cell-based assays using antibody-based applications and methodologies.
• Responsibilities of the antibody companies—engaging suppliers to produce higher-quality, internally validated, and certified antibodies.
• Drivers for adoption—ensuring that validation standards are accepted and practiced by the research community, partly through proficiency training by users, and partly through certification by producers and resellers.
• The future of antibodies—integrating new technologies, including greater investment in recombinant antibodies. We also discussed the key roles and responsibilities of journals, funders, and academia in the solution.
Areas of Agreement
Building upon the conceptual framework that the International Working Group on Antibody Validation recently published in Nature Methods, we reached consensus on several key points. Three in particular were met with strong, broad agreement.
First, antibody validation must be conducted using the experimental application and conditions in which the antibody will be used. These conditions include sample type (for example, cells, lysates, primary tissue, or purified protein), antibody and antigen concentration, native or denaturing conditions, and other conditions specific to the experimental application.
Second, information-sharing and transparency of critical reagent and validation data were also viewed as essential.
Third, users and producers pledged to work together to develop and implement a scoring system that enables purchasers to identify and compare antibodies that may best suit their experimental needs, based on ratings for individual characteristics and overall quality.
Future Challenges
Going forward, GBSI will convene application-based working groups that will elucidate specific validation standards and steps for implementation that will be published in 2017 for use by the global biomedical research community.
My colleagues and I who are leading this effort have no illusions. The biological underpinnings of antibodies make them an exceedingly challenging target for validation standards. From production variations, to transparency of reagent information and resellers, to how a user should appropriately validate an antibody, to the fact that each antibody application has its own unique issues as it relates to validation: all present tremendous challenges and obstacles to success.
All the research community’s stakeholders—producers, resellers, researchers, funders, journals—are being called upon to cooperate in the creation of tools and practices that can facilitate new discoveries, treatments, and cures. And yes, these tools should enable researchers to generate results that are actually replicable and reproducible.
Each of us—whether supplier, producer, researcher—has our own perspective, which means we may have difficulty recognizing that we are all part of the same problem. Regardless, we all came to Asilomar with great enthusiasm. We are all determined to be part of the solution.
Leonard P. Freedman, Ph.D. ([email protected]), is president of the Global Biological Standards Institute.