https://orcid.org/0000-0003-2960-9322
Abstract
From animal research through adverse events in clinical trials to health scares around food contamination, toxicology has frequently been a focus of scientific and societal concern. As these concerns shift with each new drug, new technology or public health scare, how can toxicology stay current, relevant and ethical? Two of the biggest ethical challenges in pharmaceutical toxicology are the use of animals in testing and the high safety-related attrition rates in new drug development. Both of these require progress in the discipline that will only be driven by research funding. Yet, very little is invested in these two fields compared with investment in new efficacy models, new disease targets and new technologies. How can this be addressed? Here, we explore current paradigms in toxicology that may have the potential for perceived or actual unethical ramifications. We discuss the underpinnings of such practices and make recommendations for change around peer review, resourcing, transparency and data sharing. These ideas build on the analysis presented in the 2004 Paton Prize lecture (Purchase, 2004) where issues around conflict of interest (COI), collaboration and competition in the context of ethical behaviours were highlighted. These areas are clearly relevant to many aspects of scientific research but here we focus on toxicology and specifically toxicology in the pharmaceutical industry.
Toxicology has frequently been a focus of scientific and societal concern. It's time to examine practices such as peer review, resourcing, transparency and data sharing to ensure the field stays current, relevant and ethical.
Introduction
There has never been a better time to talk about ethics. Now more than ever, what we choose to research and why is coming under scrutiny. University courses, consortia, think tanks and campaigns have emerged in their hundreds over the past decade to challenge who is making these decisions. PubMed notes around a thousand articles published each year since the turn of the century on bioethical concerns. Today, we cannot conduct science without an awareness of the political, economic and ethical undercurrents that drive its direction. From animal research through adverse events in clinical trials to health scares around food contamination, toxicology has frequently been a focus of scientific and societal concern. As these concerns shift with each new drug, new technology or public health scare, how can toxicology stay current, relevant and ethical?
Toxicology broadly falls into two categories; research and regulatory (Fig. 1). Regulatory toxicology is defined as work done to support testing of a potential new product as well as its subsequent registration and ongoing stewardship. Tests that must be conducted are clearly defined by international guidelines for pharmaceuticals (International Committee on Harmonization [ICH]),1 industrial chemicals (European Chemical Agency [ECHA], 2014)2 and agrochemicals (WHO, 2009).3 These testing programmes are generally designed by the sponsor (the company wishing to progress the product) based on their expert interpretation of what studies are needed to progress a project through key milestones whilst ensuring human (patients, volunteers, workers, the public at large) and environmental (land and aquatic animals and plants, water and air quality) health is protected. In addition, further work may be requested by the regulatory authorities in the different domains such as Europe, USA, South America, Japan and China where product registration is sought.
Research, applied and regulatory toxicology. The schematic depicts a continuum from research toxicology through to regulatory toxicology with the middle ground of applied research toxicology. Research toxicology is aimed at expanding our knowledge base without a clear route to application of this new knowledge. In contrast, regulatory toxicology is defined as work done to support testing of a potential new product as well as its subsequent registration and ongoing stewardship. As well as regulatory and research toxicology, there is also a mid-ground of applied toxicology where research projects and experiments are specifically designed such that their output will have a direct impact on risk assessment. 3Rs: Reduction, refinement and replacement of animals; AI: artificial intelligence; CAR: constitutive androstane receptor; iPSC: induced pluripotent stem cells; miRNAs: microRNAs; MPS: microphysical systems; MOA: mode of action; SNPs: single nucleotide polymorphisms; TK/TD: toxicogenetics/toxicodynamics.
On the other hand, research toxicology is aimed at expanding our knowledge base without a clear route to application of this new knowledge. The focus of research toxicology evolves over the decades as trends and innovation in the science develops. For example, in the 1990s, apoptosis was a strong trend in toxicological research whereas today there is much interest in the basic science of epigenetics. Research toxicology also focuses extensively on the development of new models and methodologies of potential application such as the current focus on Clustered Regularly Interspaced Short Palindromic Repeat (CRISPR) and on induced pluripotent human stem cells.
As well as regulatory and research toxicology, there is also a mid-ground of applied toxicology where research projects and experiments are specifically designed such that their output will have a direct impact on risk assessment. This could be through offering new methodologies of immediate use such as toxicokinetic models to help interpret in vivo data or exploration and establishment of replacements for animal tests such as the local lymph node assay (LLNA) (van Huygevoort, 2017)4 based on work conducted at Central Toxicology Laboratory (CTL) by Ian Kimber and his team.5 Another major focus of applied toxicology is to establish a proposed mode of action (MOA) for toxicological findings observed in regulatory studies and also to establish if these toxicological findings are relevant to humans. One good example of this is the work carried out at CTL in the 1980s and 1990s on peroxisome proliferators and species differences by Cliff Elcombe and others.6–9
In this special issue, we describe the current status of the conduct and resourcing of toxicology as well as the pressures on the discipline. These ideas build on the analysis presented by Iain Purchase in his 2004 Paton Prize lecture where he elegantly summarized the concerns of the time, especially the emergence of gamesmanship where, for example, ‘strong assertions of conflict of interest are used to justify particular points of view’.10 We discuss the evolution of these ideas especially around conflict of interest (COI), transparency, reproducibility and funding of animal research in the context of ethical behaviours and suggest new ways of working for consideration and comment. These are aspects of ethics that are perhaps most visible, but this is just the tip of the iceberg (Fig. 2) with many other confounding and underlying issues such as resourcing, peer review policy. Challenges highlighted herein are clearly relevant to many aspects of scientific research but here we focus on toxicology and specifically toxicology in the pharmaceutical industry.
The toxicology ethics iceberg. There are aspects of ethics such as conflict of interest (COI), animal testing, transparency and reproducibility that are highly visible. However, this is just the tip of the iceberg with many other confounding and underlying issues such as resourcing, data sharing, peer review, economics and politics.
Resources in toxicology
One of the most influential factors in the direction of scientific research and regulation is funding. In toxicology, the majority of funding and resources come from primary industries (pharmaceutical, agrochemical, chemical, petrochemical and food companies), from Contract Research Organizations (CROs) and from governmental research agencies and charities (Fig. 3). The balance of these three primary sources is constantly evolving both in terms of available budgets (largely driven by economic cycles) and in terms of priorities. Research priorities are vulnerable to changing political climate and are heavily influenced by the understandable desire to back the next wave of game-changing science.
Resources, pressures and outcomes in toxicology. Resources into toxicology are scarce yet some are escaping through holes in the system. Resources come mainly from contract research organisations (CROs), industry and the research councils and charities. There are many pressures creating a squeeze; pressure from patients/government, pressure to publish, pressure to meet profit expectations and pressure to stay relevant. Limited resources combined with pressures create a number of consequences that may have the potential for perceived or actual unethical ramifications. Specifically, there may be competition rather than collaboration, unnecessary/repeated experiments, misguided investments into outdated research, lack of reproducibility and a lack of trained scientists entering toxicology. We propose open discussion and challenge to the current ideology along with interdisciplinary collaboration to share resources.
Research councils and charities
As with most branches of biomedical research, resourcing for research toxicology is largely by grants from governmental research councils and charities. This is a highly competitive process based on peer review of the project proposal, backed up by evidence of previous success such as the authors’ publication record. Resources are scarce for most of the biomedical sciences but toxicology is especially difficult with few if any of the main granting bodies open to applications with toxicology as the primary theme. Of the limited resources invested in research toxicology, there is a trend towards academic work that uses in vitro models of toxicity. Typically, these projects aim to investigate mechanisms of a specific target organ toxicity such as hepatotoxicity, renal toxicity, neurotoxicity or gastrointestinal toxicity. The outputs may be basic knowledge and research papers but often there is a secondary aim in that this work could offer a potential route to detection and prediction. Typically, only a small proportion of this ‘stand-alone’ academic type of work has found its way into these detection/prediction testing cascades; most examples of this were driven via academic/industry partnership such as the bovine corneal endothelial assay11 and the local lymph node assay.5
Broadly speaking, there are two categories of toxicology research funded by charities. The first is in the context of new drug discovery for disease/patient-focused charities such as those that aim to tackle heart disease, Alzheimer's and cancer. In this, toxicology is not the primary focus but rather might manifest as an issue to be resolved to move a project forward. Experience has shown that although driven by good intentions this ‘tick box’ motivated work may be misguided in the absence of hands-on pharmaceutical toxicology experience on the research panel or in the proposing research team. There are confounding challenges here in that some charities will not support work conducted in the dog (as a second toxicology species) even though a second non-rodent species is usually required under ICH1 for first time in human trials.
A major source of funding in the UK for toxicology comes from the National Centre for the Replacement, Refinement & Reduction of Animals in Research (NC3Rs), a not-for-profit organisation that aims to ‘foster collaborations between universities and industry to develop and commercialise 3Rs technologies, and provide information on the latest advances to put the 3Rs into practice’. The NC3Rs receives core funding from the Medical Research Council (MRC) and the Biotechnology and Biological Sciences Research Council (BBSRC) as well as funding for specific programmes from the charitable and commercial sectors. The NC3Rs is generally regarded as being highly successful in selecting the right projects for funding since the organisation engages heavily across sectors (academia, industry, regulators) to ensure their programmes are relevant, realistic, measurable and useful. However, projects delivered by NC3Rs must be aligned to the NC3Rs mission and budget and as such this organisation cannot be expected to resource the broader issues that challenge research, applied and regulatory toxicology.
Industry
Much of primary pharmaceutical industry funding for toxicology is assigned to the conduct of good laboratory practice (GLP)-compliant regulatory studies that are required to protect patient and volunteer safety during the transitioning of promising drug projects through key milestones. Larger pharmaceutical companies also invest heavily in early discovery strategies to identify and mitigate toxicological risks in projects as well as in internal toxicology research into new science and new models. A 2015 Association of the British Pharmaceutical Industry (ABPI) study also demonstrated that Industry resource was a primary driver behind work done to develop and implement in vitro alternatives to animal tests.12 Many of the larger companies also have external collaboration budgets that fund academic collaborations via joint students and postdoctoral fellows. These funds also support larger partnerships with academia and innovative small companies in areas of mutual interest. With the changing economic climate, budgets for these types of activity have disappeared or have been squeezed. There is a big emphasis on the need to demonstrate clear return; although this seems reasonable, basic research does not always fit well with such impact metrics.
CROs
The majority of CRO resources in toxicology are assigned to the conduct of GLP-compliant regulatory studies and earlier non-GLP screening cascades on behalf of clients. There are also resources available for work that is likely to translate to a commercial test such as new screens for immunotoxicology or for liver toxicity but it would be unusual to find a more basic research programme in a CRO. This is entirely appropriate since CROs exist to deliver work defined by others. CROs are also proving to be a new and valuable provider of work experience or sandwich student placement projects – an especially valuable contribution as the reduction in the overall size of large pharma research and development (R&D) reduces the number of these opportunities available. CROs also contributed extensively to the work invested in in vitro alternatives to animal tests in the pharmaceutical industry as demonstrated in the 2015 ABPI study.12
Although it may seem more straightforward, there are also challenges in how we deploy resources in regulatory toxicology across pharmaceutical companies and CROs. Experienced toxicologists tend to design more focused ‘first time in man’ packages compared with their less experienced counterparts who may lack the knowledge and confidence to deviate from a perception of regulatory expectations based on published guidelines. Many smaller companies and academics rely upon CROs to design and deliver their toxicology programme and as such the work is unlikely to represent a minimal ‘fit for purpose’ package.
Pressures
Pressure to demonstrate impact
Measuring impact of research, applied and regulatory toxicology presents some common and some unique challenges. The impact of regulatory toxicology appears reasonably clear in that work is conducted to meet agreed milestones in product development, registration and commercialization. In this, impact equates not necessarily to the progress of a project but more to a clear decision to progress or to stop based on quality data and its expert interpretation. This is well illustrated in the 2014 paper from AstraZeneca where the key data required for a decision at each step are clearly identified.13 Industry also tends to set itself speed and quality impact metrics for measuring success. Although the validity of these metrics can be debated, they are clear, measurable and are generally comparable across companies, portfolios, modalities and timeframes. Many organisations exist offering to ‘benchmark’ pharmaceutical companies14–16 as a way of comparing speed, quality and cost. In preclinical toxicology, there are several high-profile metrics used for benchmarking such as ‘time from first GLP dose to first time in man trial’ and/or ‘percentage attrition during GLP toxicology testing’.
Impact in academic research has a different interpretation to that in regulatory toxicology and many academics would argue that genuine scientific impact is hard to assess. Thus, impact in toxicological research can be qualitative but more often refers to the formal ‘impact factor’ as calculated from citations over previous publication years. Building on this, there are many system-based scores such as the ‘H-index’ which summarizes publications and their impact over time for individuals. Some of the best explanations of different ways to measure impact are found on university websites, probably because it's in the Institute's best interest to guide its academics to fully record and track impact. Regarding publication, authors compete for limited space in the higher impact journals since success in this competition can have a profound and long-lasting effect on institutional and individual reputations and careers. High impact papers lead to invitations to present work at international conferences, invitations to write reviews and commentary and therefore provide a route to perpetuate impact, a differentiating factor in the career of a scientist.
Recently, ‘impact’ has also been added to assessments within the Research Excellence Framework (REF)18 which exists to assess the quality of research in UK higher education institutions. Alongside the need to submit the 4 best papers (judged by impact factor) over the 5-year window, there is also a requirement to submit case studies. Such case studies broadly assess the translatability and utility of research findings to improve the health and wealth of the nation.
Regarding assessing impact of applied toxicology, it is relatively straightforward to determine if work carried out has met its objectives, assuming those objectives were clearly defined. For example, did the work conducted succeed in determining a MOA for a particular toxicity and/or establish its relevance to humans? Did the work conducted provide evidence that a new assay or way of measuring/predicting toxicity can be used in prioritization and risk assessment? It's interesting to note that industry, CROs and all academia contributed extensively to work with direct application such as the development and use of in vitro tests that replace animals in safety screening.12 A previous paper from this ABPI project19 also highlights the value of collaboration between these sectors in the delivery of applied toxicology.
Pressure to meet profit, patient and government expectations
Toxicology research carried out in industry has been subject to societal challenge on the grounds that these organizations are ‘for profit’ and as such may be driven by motives other than pursuit of ‘scientific truth’. Specifically, the challenge posed is that organizations and individuals working for those organizations may stand to gain financially and reputationally by a preferred outcome from a piece of research.
In response to this and other concerns raised by patients and society, international governments and organisations have put in place a Code of Conduct or Code of Practice. In the UK, this is set out in the ABPI Code of Practice for the Pharmaceutical Industry.16 The Code sets standards for all working in the industry, including a requirement for the provision of information to patients and the public. A code of practice would typically cover all aspects of the conduct of toxicology such as supply of samples, promise of benefit, provision of hospitality, provision of information to the public and relationships with patient organisations.
Although the detailed provisions in the Code aim to ensure that pharmaceutical companies operate in a responsible, ethical and professional manner, the issue often raised in public debate is that drug companies make a profit at all. It's curious that profits are an acceptable motivator for other sectors where there are both risks and benefits such as food, drinks and the motor industry. Greater transparency in how pharmaceutical companies operate coupled with opportunities for public engagement may help overcome this.
Pressure to meet patient and government expectations: COI and transparency
An issue that generates a lot of attention in scientific research, especially toxicology is conflict of interest (COI). A definition of the term ‘conflict of interest’ was suggested by Dennis Thompson in 1993 and can still be considered valid in 2017: ‘A conflict of interest is a set of circumstances that creates a risk that professional judgement or actions regarding a primary interest will be unduly influenced by a secondary interest’.17
Current adherence to codes of conduct by the pharmaceutical industry and by the CROs that support them means that individuals and organisations are held accountable for their actions under law. Together with the introduction of Good Laboratory Practice (GLP) legislation, occurrences such as faking of data as highlighted by Purchase (2004) are fortunately largely in the past. Despite this, today the circumstances referenced in Thomson's definition around COI have been restricted almost exclusively to the pharmaceutical industry.17
It was pointed out more than a decade ago that COI is being used in gamesmanship where, for example, ‘strong assertions of conflict of interest are used to justify particular points of view’.10 This challenge still persists today with ongoing attention on COI and some unfortunate consequences such as the exclusion of industry scientists from governmental panels and review committees.17 But surely all these same challenges apply to research conducted elsewhere where a successful outcome can have a profound impact on institutional and individual reputations and on the funding streams that inevitably follow? Going beyond this, obtaining grants and public visibility has a direct impact on the renewal of temporary employment contracts and on conversion to academic tenure. As highlighted in 2004,10 the main casualty from errors and from gamesmanship is the perceived status of the science itself. In this paper, the case is made that the key issue to be considered in assessing potential COI is the intention of the person carrying out the work. In the context of toxicology, intent includes discovering toxic effects or mechanisms of action as well as issues such as commercial intent, environmental concerns and responsible use of animals. It's time to take another look at these currently ignored or overlooked potential areas of COI and ensure that all are held to the same standards based on intention and conduct.
A second key area is around transparency. This can have many interpretations but one area of great relevance to regulatory and research toxicology is the availability of published work. Many researchers experience frustration when trying to access or download a PDF of a potentially interesting paper. This is because for scientists that are not members of an institution with an institutional licence, access for papers is subject to a charge (typically around $35). Realistically, options open to individuals are then to overlook certain papers or cite based just on the abstract rather than an analysis of the actual data. To overcome this, there has been a rise in Open Access over the past decade where papers are free to all at source. As outlined on the Open Access Week website,20 open access gives free, immediate, online access to the results of scholarly research, and the right to use and re-use those results as needed. As such, open access has the power to transform the way research and scientific inquiry are conducted. Organisations such as the Health and Environmental Sciences Institute (HESI) and many others that are committed to generating science for public benefit highlight open access as part of their commitment to transparency.21 Around 90% of HESI's publications are open access.21 Similarly, the Research Councils UK (RCUK) organisation mandates that all funded work is open access and provides resources to do so via the RCUK Open Access Block Grant.
Pressure to stay relevant: fast moving science and dragging regulation
As with other disciplines, toxicology evolves over time with trends ebbing and flowing. With limited resources available, research toxicology must move quickly to ensure it stays relevant and that work carried out moves the field forward. One interesting consequence of the rapid shifting of scientific paradigms is that regulatory policy may drag behind technological capabilities. For example, at present there is much interest and investment in microphysiological systems based on differentiated human stem cells since these may better predict human responses compared with animal tests. But how will these be incorporated into toxicology and especially into regulatory decision making around safety? We are a long way from any clear view of this. Similarly, the discovery and early development portfolios of pharmaceutical companies feature many so called new modalities ranging from oligonucleotides, through novel peptides to stem cells. Despite the evidence that some of these new therapeutic modalities are close to being clinic ready, we do not yet have a toxicology testing paradigm. In the absence of this, conservatism may trigger a full small molecule two species package which might be inappropriate and could lead to unfortunate delays and waste of resources. We need a mechanism for regulatory policy to keep pace with the science it regulates.
Challenges and opportunities
Resourcing models and pressures on the system have created a number of consequences to be addressed (Fig. 3). In this section, we explore current paradigms in toxicology that may have the potential for perceived or actual unethical ramifications.
Peer review
Peer review is essential to scientific success in two main ways; success in publication and success in winning grants. These have interdependencies where success in grant applications depends heavily on high impact publications and publications are driven by data generated largely during grant-funded programmes. Arguably, established laboratories with existing reputations perform better in this process especially if they have high impact publications. There are several issues here; it's hard for new groups and ideas to break into the establishment which in turn may hold back innovation. It is also extremely difficult to publish applied toxicology in high impact journals, even those journals with a toxicology focus. Thus, those academics looking for grant funding for toxicology research face substantive challenge even when the work proposed is outstanding, well-designed and novel, with immediate application. As a consequence of this, funding for applied toxicology seems to come mainly from industry sponsored projects and from mission-lead groups such as the NC3Rs.
Confounding this is the issue of reproducibility; there are many reports that papers are not reproducible, occasionally even by the originating laboratory. The issue of reproducibility in toxicology research was raised in detail in the Paton Prize lecture10 and several solutions proposed. However, more recent analyses have shown that the problem persists since the majority of preclinical cancer papers could not be reproduced, even by the investigators themselves.22,23 Specifically pertaining to toxicology, Glenn Begley presented compelling evidence during his Society of Toxicology 2016 webinar on lack of reproducibility24 and made the point that based on current performance of the peer review system, the majority of the discoveries that form the basis of 21st century toxicology will not stand the test of time. This is of course a point that applies to all science and not just to toxicology.
Peer review in toxicology as with other disciplines is generally regarded as anonymized and based on merit.
However, this peer review is usually single blind in that the identity of and institute of the authors are visible to the reviewer. In the recent New Scientist issue on Bioethics, Andrew Stirling25 argues that experts have a responsibility to drop the pretence that they can be perfectly impartial; our collective decision making only has benefits if those perspectives are shared freely. Thus, we all have intrinsic bias that should be acknowledged and managed.
Tennant et al. concurred that peer review is prone to bias and abuse and can fail to detect even fraudulent research.26 In their paper, they present a detailed analysis to consider the pros and cons of systems such as single blinded peer review (reviewers are anonymous), double blinded peer review (authors and reviewers are anonymous) and triple blinded peer review (where authors and their institutions are reciprocally anonymous to reviewers and editors).26 Suggestions to modify peer review vary between fairly incremental small-scale changes to those that encompass an almost total and radical transformation of the present system such as “Open Peer Review” (OPR). However, there is no formally established definition of OPR that is accepted by the scholarly research and publishing community making this hard to implement at present.
We propose that peer review is double blinded routinely at least while newer OPR systems that align with the latest advances in ‘open science’ are developed. Double-blind peer review can increase author diversity; if the authors and reviewers are mutually anonymous then it seems reasonable to assume that the work will be judged purely on merit, making the review more objective. However, this double-blind review is still prone to bias since identities can often be deduced. On the downside, it also appears to be more time consuming.
Building on these ideas of new systems for peer review, wider dissemination and implication of the 6-red flags rules as described by Begley27 could be incorporated to give a more solid basis for review of research findings. These flags include such fundamental issues as inappropriate reagents, lack of reproducibility and absence of controls (Table 1). Steps have been proposed to overcome this for toxicology as a discipline by Gary Miller.28 It is timely to consider an integrated and unified approach to addressing this issue across all basic and applied pharmacology and toxicology.
Six red flags for suspect work. Six key common findings are highlighted with suggestions for remediation (concepts taken from Begley (2013)27 with some modification)
| 6 Red flags | Assertion | Implementation |
|---|---|---|
| Were experiments performed blinded? | Animal studies, in vitro work and reading of gels can and should all be done, blinded to the experimental versus control groups. | Check the methods and figure legends |
| Were experiments repeated? | Unfortunately, repetitions are seldom performed. Western blotting and similar analyses are often performed only once, and when the desired result is obtained, that result is shown. | If reports fail to state that experiments were repeated, by sceptical |
| Were all results presented? | Most western blots show only a sliver of the gel with the majority of bands cropped. Although many of these cropped bands may be extraneous, their removal falsely implies that the antibody could detect only the desired protein, which is rarely the case. | Compare the results of other experiments in the paper that used the same antibody: the pattern of bands should be the same across experiments. |
| Beware the ‘typical result’; ask to see all of the results. | ||
| Were positive and negative controls included? | Often in the non-reproducible, high-profile papers, the crucial control experiments were excluded or mentioned as ‘data not shown’. Some photos of gels are over-exposed and well outside the linear range of the film. | A publication that hides the controls should be viewed with caution |
| Over-exposed controls obscure an alleged difference between samples may simply be the consequence of loading more total sample. | ||
| Were reagents validated? | Data that validate reagents are not shown or reference an earlier paper (also not shown). Antibodies have been used when the manufacturer declares it unfit for that purpose. | Ask to see reagent validation data such as antibody binding specificity and small molecule off target profiles. |
| Experiments with small-molecule inhibitors focus on the pathway of interest, overlooking multiple other targets. | ||
| Were statistical tests appropriate? | Improper statistical analysis is commonly seen in animal studies, in which results are collected over a long time. On such a time curve, two points may be highlighted and declared to be significantly different from points on the control curve, even though the totality of the two curves is essentially the same. | Check that the statistical test has been applied to the whole curve, rather than just to selected points along it (the position of the asterisk marking the statistical P value is an important clue). |
| 6 Red flags | Assertion | Implementation |
|---|---|---|
| Were experiments performed blinded? | Animal studies, in vitro work and reading of gels can and should all be done, blinded to the experimental versus control groups. | Check the methods and figure legends |
| Were experiments repeated? | Unfortunately, repetitions are seldom performed. Western blotting and similar analyses are often performed only once, and when the desired result is obtained, that result is shown. | If reports fail to state that experiments were repeated, by sceptical |
| Were all results presented? | Most western blots show only a sliver of the gel with the majority of bands cropped. Although many of these cropped bands may be extraneous, their removal falsely implies that the antibody could detect only the desired protein, which is rarely the case. | Compare the results of other experiments in the paper that used the same antibody: the pattern of bands should be the same across experiments. |
| Beware the ‘typical result’; ask to see all of the results. | ||
| Were positive and negative controls included? | Often in the non-reproducible, high-profile papers, the crucial control experiments were excluded or mentioned as ‘data not shown’. Some photos of gels are over-exposed and well outside the linear range of the film. | A publication that hides the controls should be viewed with caution |
| Over-exposed controls obscure an alleged difference between samples may simply be the consequence of loading more total sample. | ||
| Were reagents validated? | Data that validate reagents are not shown or reference an earlier paper (also not shown). Antibodies have been used when the manufacturer declares it unfit for that purpose. | Ask to see reagent validation data such as antibody binding specificity and small molecule off target profiles. |
| Experiments with small-molecule inhibitors focus on the pathway of interest, overlooking multiple other targets. | ||
| Were statistical tests appropriate? | Improper statistical analysis is commonly seen in animal studies, in which results are collected over a long time. On such a time curve, two points may be highlighted and declared to be significantly different from points on the control curve, even though the totality of the two curves is essentially the same. | Check that the statistical test has been applied to the whole curve, rather than just to selected points along it (the position of the asterisk marking the statistical P value is an important clue). |
Six red flags for suspect work. Six key common findings are highlighted with suggestions for remediation (concepts taken from Begley (2013)27 with some modification)
| 6 Red flags | Assertion | Implementation |
|---|---|---|
| Were experiments performed blinded? | Animal studies, in vitro work and reading of gels can and should all be done, blinded to the experimental versus control groups. | Check the methods and figure legends |
| Were experiments repeated? | Unfortunately, repetitions are seldom performed. Western blotting and similar analyses are often performed only once, and when the desired result is obtained, that result is shown. | If reports fail to state that experiments were repeated, by sceptical |
| Were all results presented? | Most western blots show only a sliver of the gel with the majority of bands cropped. Although many of these cropped bands may be extraneous, their removal falsely implies that the antibody could detect only the desired protein, which is rarely the case. | Compare the results of other experiments in the paper that used the same antibody: the pattern of bands should be the same across experiments. |
| Beware the ‘typical result’; ask to see all of the results. | ||
| Were positive and negative controls included? | Often in the non-reproducible, high-profile papers, the crucial control experiments were excluded or mentioned as ‘data not shown’. Some photos of gels are over-exposed and well outside the linear range of the film. | A publication that hides the controls should be viewed with caution |
| Over-exposed controls obscure an alleged difference between samples may simply be the consequence of loading more total sample. | ||
| Were reagents validated? | Data that validate reagents are not shown or reference an earlier paper (also not shown). Antibodies have been used when the manufacturer declares it unfit for that purpose. | Ask to see reagent validation data such as antibody binding specificity and small molecule off target profiles. |
| Experiments with small-molecule inhibitors focus on the pathway of interest, overlooking multiple other targets. | ||
| Were statistical tests appropriate? | Improper statistical analysis is commonly seen in animal studies, in which results are collected over a long time. On such a time curve, two points may be highlighted and declared to be significantly different from points on the control curve, even though the totality of the two curves is essentially the same. | Check that the statistical test has been applied to the whole curve, rather than just to selected points along it (the position of the asterisk marking the statistical P value is an important clue). |
| 6 Red flags | Assertion | Implementation |
|---|---|---|
| Were experiments performed blinded? | Animal studies, in vitro work and reading of gels can and should all be done, blinded to the experimental versus control groups. | Check the methods and figure legends |
| Were experiments repeated? | Unfortunately, repetitions are seldom performed. Western blotting and similar analyses are often performed only once, and when the desired result is obtained, that result is shown. | If reports fail to state that experiments were repeated, by sceptical |
| Were all results presented? | Most western blots show only a sliver of the gel with the majority of bands cropped. Although many of these cropped bands may be extraneous, their removal falsely implies that the antibody could detect only the desired protein, which is rarely the case. | Compare the results of other experiments in the paper that used the same antibody: the pattern of bands should be the same across experiments. |
| Beware the ‘typical result’; ask to see all of the results. | ||
| Were positive and negative controls included? | Often in the non-reproducible, high-profile papers, the crucial control experiments were excluded or mentioned as ‘data not shown’. Some photos of gels are over-exposed and well outside the linear range of the film. | A publication that hides the controls should be viewed with caution |
| Over-exposed controls obscure an alleged difference between samples may simply be the consequence of loading more total sample. | ||
| Were reagents validated? | Data that validate reagents are not shown or reference an earlier paper (also not shown). Antibodies have been used when the manufacturer declares it unfit for that purpose. | Ask to see reagent validation data such as antibody binding specificity and small molecule off target profiles. |
| Experiments with small-molecule inhibitors focus on the pathway of interest, overlooking multiple other targets. | ||
| Were statistical tests appropriate? | Improper statistical analysis is commonly seen in animal studies, in which results are collected over a long time. On such a time curve, two points may be highlighted and declared to be significantly different from points on the control curve, even though the totality of the two curves is essentially the same. | Check that the statistical test has been applied to the whole curve, rather than just to selected points along it (the position of the asterisk marking the statistical P value is an important clue). |
As mentioned earlier, open access seems like a good step towards transparency and to making all science equally accessible. However, journals are usually profit-making businesses and need to cover their costs and margins; so in open access the cost is passed from the reader to the author and/or to their institute that pays the open access fee of around $2000–$3000. There are also challenges in that different journals have different financial and legal policies such that there remain hurdles to even the most well-intentioned open access policies.
It's early days in the open access era, but how will this impact on accessibility, dispersion, citations and impact of papers? It may be interesting to divide individual publication and overall journal impact factors into those derived from open-access papers versus those that weren't. Will open access papers have higher impact over time? Will they prove to be pay-to-play?
Trained toxicologists
Another challenge faced by Toxicology is the supply of trained toxicologists. Traditionally, toxicologists have come through two routes: applied academic courses and industrial training. Both of these have declined in recent years, resulting in far fewer trained toxicologists. Many of today's mid- to late-career toxicologists were ‘grown’ in well-funded academic departments or were recruited as fresh PhDs or postdoctoral fellows from related fields such as biochemistry and molecular biology, and were then trained on the job, many at CTL. Here, they were able to develop their scientific thinking and careers whilst developing the pragmatism required for an industrial or applied toxicologist role. Regarding academia, there were many postgraduate courses across the UK and Europe but these centres of expertise have suffered a severe decline.29
Regarding industry training, pressures on headcount and metrics mean it is rarely feasible today to recruit inexperienced toxicologists and have them train over 5 years or more ‘on the job’ via shadowing and mentoring. Graduates are also emerging with less practical experience due to cost pressures on university-funded laboratory time; degree programmes also struggle to find industrial placements. Attempts to correct this were undertaken by the MRC in 2008 when they introduced the Integrated Toxicology Training Partnership (ITTP), an initiative aimed to improve and boost capacity in the toxicological sciences by sponsoring PhD studentships.30 In total, 48 four-year PhD studentships have been awarded since 2008 but with a steady decline in numbers from >10 in 2008 to 4 in December 2017. The continuation of this scheme is vital for the discipline of toxicology in the UK; more should be done to support and extent schemes like this.
Funding outdated research: are some investments into in vitro models misplaced?
There is significant investment in mechanistic toxicology in vitro where a chemical is added to a cell line and the response described. A simple google search of ‘in vitro models of hepatotoxicity’ returns >4 million results. But how much of this work is reproducible? Metrics are hard to find, but many of these pieces of research fail to meet the most basic of criteria such as validity of the model, specificity of the endpoint for the chemical being studied and adherence to concentrations ranges relevant to real world exposures. Journals are rigorous in reminding authors and reviewers of their obligations31 but as highlighted by Begley,27 data that validate reagents are not shown and experiments with small-molecule inhibitors only focus on the pathway of interest, overlooking multiple other targets.
Impact in academic research has a different interpretation to that in regulatory toxicology and many academics would argue that genuine scientific impact is hard to assess. One argument is that a lot of research has potential for application and as such benefits society but not all science can be applied and as suggested recently, it shouldn't all be funded.32
Aside from validity and reproducibility, there is the issue of translational impact. It seems obvious to suggest that research priorities should be set based on evidence of benefit but this is notoriously hard to measure. In a recent New Scientist review of ethics, the idea that scientists ‘ought to pursue whatever stimulates their curiosity because no one knows what the next practical application is’ is challenged as ‘really nothing more than a convenient just-so story’.33 So, how much of this data will actually move the field forward in a meaningful way? How many of these in vitro assays will ever actually be useful over and above the simple cytotoxicity assays that are used at present in screening cascades? Resource waste could be reduced by ensuring funded research is relevant and up to date. Resourcing of cell line and animal tissue-based in vitro research should be challenged since industry has largely moved onto research in humanized 3D models, PK/PD modelling, read-across and big data approaches to predict human risk.
Regarding basic research, academics tend to argue that impact cannot be measured but as highlighted by Brooks32 ‘It is odd that a bunch of empirically minded people will not actually be able to produce empirical evidence supporting the idea of unqualified benefits of basic scientific research’.
Funding outdated research: are all in vivo studies merited?
Another question raised is whether all in vivo studies are merited. What criteria do granting bodies, researchers and journals use to judge whether or not animal studies are appropriate to answer the question? The literature is crowded with studies of animal models of human disease and human toxic responses but how many of these are of any use? This is especially pertinent in the light of a move away from these outdated and discredited animal models towards humanized tissues and models. How do we address this? One suggestion is to have a clear hierarchical guide of complexity staring with in silico, progressing to in vitro before allocation of funds for and publication of in vivo work. This hierarchy should also encompass all available human data gained from any source. Such a requirement would force new thinking on how the scientific community addresses its research questions.
Collaboration and competition: better use of resources?
Resources into toxicology are scarce yet some are escaping through holes in the system (Fig. 3). For example, the system of competition for grants and funding can drive researchers to work in the same area but without collaboration. Failure of institutes to collaborate and share findings means experiments are repeated or expensive equipment is duplicated across universities.
Some of the most promising science is being performed at the boundary between disciplines or institutes. A recent collaboration between Physiology and Mathematics departments at King's College London33 is a compelling example of interdisciplinary specialists applying new, exciting techniques to old problems. Dr Nandi's research focuses on the cardiovascular dysregulation that occurs in septic shock and she routinely collects mammalian cardiovascular waveforms in her laboratory. Prof. Aston has applied non-linear mathematics to extract more information from these waveforms by plotting and visualizing the raw data in a novel way to create an ‘attractor reconstruction’ (AR). Together they hope to see if this new mathematical approach can extract more information from the signal in order to detect the onset of disease earlier. There is something remarkable about this particular project in that it has challenged an age-old paradigm with phenomenal success; from our very first science lesson to conclusions drawn in peer reviewed papers, we depend on averaging to simplify and interpret data. Yet the data analysis techniques used in this pioneering research used every data point collected, wasting nothing. In fact, their results came from analysing the specific changes in their rich data set. Again, what other discoveries could be made from looking within data? Which other institutes are currently expending money and people on problems that could be solved using unconventional collaborations?
As well as boundary free research, the debate on collaboration and competition is focusing increasingly on data use and data sharing as data evolves to become the world's most profitable and most powerful commodity. Organisations such as HESI that generate information and resources of benefit to public health have clear policies on data sharing and transparency to maximize dissemination via peer reviewed literature, deposition of data in publicly available databases, workshops and other outreach efforts.21 However, scientific data are being generated at a higher rate than ever before yet our abilities to mine and interpret those data have not kept up to date. Thus, the majority of data are not used to full capacity. This poses ethical issues with in vivo experimentation – especially when one considers the resources going into animal alternatives.
However, this may change with the advent of new tools such as artificial intelligence (AI) (when a machine mimics cognitive functions such as learning and problem solving) and big data (data sets that are so large and complex that traditional data handling approaches are inadequate). Big data and AI offer exciting possibilities for re-use and repurposing of existing data and improved curation of new data as they emerge. These principles are well illustrated by the attractor reconstruction (AR) example where the collaborators used big data approaches to extract more information from existing physiological information. What else can be achieved by taking a different approach to data interpretation?
Conclusions
Two of the biggest ethical challenges in toxicology are the use of animals and especially for the pharmaceutical industry the high attrition rates in drug discovery and development. Both of these require progress in the discipline of toxicology that will only be driven by research funding. Yet, we invest very little in these two fields. In contrast, much is invested in new efficacy models, new disease targets and in new technologies. This disconnect must be addressed via several routes:
• A fundamental and data-driven re-assessment of the real challenges in treating disease.
• A recognition of toxicology as a central discipline in medical progress.
• A revision of resourcing to ensure relevance and reproducibility of in vitro toxicology studies.
• Introduction of a hierarchical system to evaluate the necessity for animal studies.
• Revision of the peer review system.
As argued by Andrew Stirling,25 progress in making the ethically right decision depends on three things: responsibility, precaution and participation. He argues that all members of society should have the chance to participate in the debate. ‘We shouldn't be scared about involving ordinary people in decisions about science and technology. The technologies we pursue, the innovations we support, the sciences we prioritise, are as genuine matters for democratic discussion as anything else’. This certainly poses a valid challenge to science and especially to the discipline of toxicology.
Conflicts of interest
Ruth Roberts is co-founder and co-director of Apconix, an integrated toxicology and ion channel company that provides expert advice on nonclinical aspects of drug discovery and drug development to academia, industry and not-for-profit organisations.
References
ICH, http://www.ich.org.
