Workshop on ‚Quality assurance, and reproducibility of results‘, Oslo, January 2017
There is an ongoing debate in leading science journals regarding the irreproducibility of data. The alarming claim is that many or even most biomedical research results are false. NEURON Cofund has a strategic aim and mission to promote high-quality and high-impact translational research in the area of brain-related diseases. NEURON’s core activity encompasses planning, launching, and con-ducting Joint Translational Calls (JTCs). All of these activities rely heavily on evidence, both data and information of the funded research projects, which improve the link between preclinical and clinical research, and may facilitate collaboration between academia and industry.
Qualified data obtained from pre-clinical research will support and substantiate if translation into clinical research questions is justified. Reproducibility of funded research projects is therefore one of the most important assets, since it clearly affects the perception and acceptance of research funding in society. As well, quality assurance of the funded projects provides reassurance to NEURON’s funding organizations and ministries, stakeholders, patients and – not least – the taxpayer.
NEURON is fully committed to the highest possible standard of quality assurance, including working to nationally and internationally recognized best practices. NEURON accepts its duty to ensure the accuracy and completeness of all evidence involved in the full range of the JTC undertakings. To explore which measures NEURON can embrace to maintain and improve international quality standards, a workshop on ‘Quality assurance and reproducibility of results’ was organized on January 19th, 2017 in Oslo.
The workshop covered three important areas:
- a) The scientific perspective including the description of the problem and tools for prevention.
- b) The funding organizations perspective was explored by research councils such as the Netherlands Organization for Scientific Research (NWO) and the Medical Research Council (MRC, UK) who have taken a lead in operationalization of measures to improve the reproducibility of data.
- c) The already implemented measures for NEURON’s JTCs.
Prof. Dr. Ulrich Dirnagl
The scientific perspective was introduced by Prof. Dr. Ulrich Dirnagl (Germany, Charité Berlin). He cited the NATURE survey ‘Is there a reproducibility crisis?’ (NATURE, 452, Vol. 533, May 2016) that had asked scientists what led to problems in reproducibility. More than 60% of respondents said that each of two factors – pressure to publish and selective reporting – always or often contributed. More than half pointed to insufficient replication in the lab, poor oversight or low statictical power. Respondents were asked to rate 11 different approaces to improving reprodiucibility in science, and got ringing endorsements. Nearly 90% – more than 1000 people – ticked ‘More robust experimental design’, ‘better statistics’ and ‘better mentorship’. Dirnagl dubbed the problem of non-reproducible research findings and failure to translate bench findings into effective therapies as ‘vicious cycle of academic biomedical research’. Where scientists need to publish new, positive and spectacular results for professional advancement, journals need to publish new, positive and spectacular results to promote their Impact Factor (IF), and institutions and funders support researchers who publish new, positive and spectacular results in high IF journals. To brake this vicious cycle he suggested a number of measures: distinguishing between exploratory and confirmatory pre-clinical research will improve translation (PLoS Biol., 2014, 12:e1001863), publication of NULL results, and preregistration of pre-clinical studies, to promote an Open Science Policy in which to find, access, interoperate, and reuse data. As tools the ‘Training in critical appraisal of publications in biomedicine’ (IICARUS), and ‘The Experimental-Design-Assistant’ (EDA; NC3R, UK) could serve.
Dr Nathalie Percie du Sert
Dr Nathalie Percie du Sert (UK; NC3Rs London) explained the ARRIVE Guidelines and the NC3Rs Experimental Design programme. The ARRIVE guidelines concern Animal Research, and specifically the ‘Reporting of In Vivo Experiments’. The guidelines were developed to improve the reporting of biomedical research using animals. It comprises a checklist of 20 items dedicated to internal validity, reproducibility and relevance/context, containing key information necessary to describe a study comprehensively and transparently. The guidelines received consensus between scientists, statisti-cians, journal editors, and research funders and are used to ensure reproducibility of animal research and avoid unnecessary animal use. The guidelines are endorsed by main UK funders and universities and over 1000 journals. ARRIVE resources (pocket guide, checklist, presentations) can be found under www.nc3rs.org.uk/ARRIVE. The Experimental Design Assistant (EDA, https://eda.nc3rs.org.uk) was developed to improve the design of animal experiments. It is a web-based tool aimed at in vivo research and was developed as collaboration between in vivo researchers, statisticians, academia and industry, and software designers specialized in artificial intelligence. The EDA offers the ability to build a stepwise visual representation of an experiment – the EDA diagram – and feedback and advice on the individual experimental plan together with dedicated support for randomization, blinding and sample size calculation. This tool serves two purposes: Practical information to improve knowledge of experimental design and improved transparency of the experimental plan, allowing more efficient communication among e.g. a consortium of researchers.
Dr. Arthur Liesz
Dr. Arthur Liesz (Germany) shared insights into a pre-clinical randomized controlled multicenter trial (pRCT; Llovera et al., Sci. Translat. Med. , 2015, 7, 299). While experimental research is hypothesis-driven exploratory research with often the drawbacks of being single-centre, underpowered, and with poor reporting, clinical trials are confirmatory efficacy and/or safety trials with numerous legal, ethical, organizational preconditions. Results from pre-clinical research often fail to validate in clinical studies. The interim steps in the translational route from bench to bedside are thus pre-clinical randomized controlled multicenter trials (pRCTs). The Munich group performed indeed the first pRCT to test a potential stroke therapy in order to improve the translation of treatment effi-cacy. The consortium structure comprised one coordination centre (Munich) and five study centres (Barcelona, Caen, Milan, Berlin, Munich) with a study design analogous to a clinical trial. The central tissue processing of all samples was performed by the same method and the analysis of all data sets by two independent, blinded raters. Finally, data deposition was secured after data lock so that all raw data sets are publicly available at FigShare (https://figshare.com/). In terms of feasibility the entire study took two years from initiation to publication (design of study protocol, experimental phase, blinded data analysis, and data lock and unblinding). With slightly less than 200 k€ the study ranged well within the range of monocentric endeavours. With more reliable results, providing evidence!
Dr. Daniël Lakens
Dr. Daniël Lakens (The Netherlands) reported on the first, and largest, replication grant in the world. In September 2016 The Netherlands Organisation for Scientific Research (NWO) launched a call for proposals on replication studies. Replication lies at the heart of the scientific method and makes it possible to build upon previously demonstrated and confirmed scientific findings. However, many studies have proved not to be reproducible. As Lakens pointed out: “There is the social dilemma of – It’s very important that ‘We’ do/publish replication studies, but ‘I’ get more out of novel research – ”. If research is not reproducible then this is often attributed to chance, or unintended errors, but p-hacking, publication bias and especially selective reporting will undoubtedly play a major role in this as well. By encouraging the realisation of replication research, NWO wants to make a contribution to increasing the transparency of research and the quality and completeness of the reporting of results. With regard to reward such efforts replication studies must be published and cited. Thus it is highly appreciated that journals such as ‘The Journal of Finance’ has launched a section for publishing replications. But replication studies should also be socially rewarded. NWO hopes that by encouraging replication research it can contribute to making replication research more commonplace and to improving insights into the reproducibility of research.
Dr. Jacqui Oakley
Dr. Jacqui Oakley (UK, MRC) introduced the Medical Research Council (MRC) peer review of experimental design. MRC is the major UK funder of animal research, and thus concerned with the justification of animal numbers in the experimental design of applications specifically from the ‘Re-placement, Refinement, Reduction’ (3Rs)/ethical perspective. To address the alarming news from the irreproducibility of research findings especially for animal experiments MRC, Biotechnology and Biological Sciences Research Council (BBSRC), the Academy of Medical Sciences (AMS) and Wellcome Trust held a workshop on the ‘Reproducibility and Reliability of biomedical research’ in spring 2015 (http://www.acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research). This was followed by a joint-funder action of updated guidance for applicants, an application form for “Experimental design, avoidance of bias and statistical considerations” and adapted guidance to reviewers. The dissemination of these activities to the research community was implemented by blogs, social media, strategic visits, and workshops for researchers. However, as Dr Oakley pointed out: ‘Culture change takes time’ perhaps because established practices are hard to shift but funders have an important role in championing change. The MRC noted already an improvement in application quality and is convinced that animal research was a good starting point for encouraging improvements in experimental design more broadly.
Dr. Sascha Helduser
Dr. Sascha Helduser (Germany, DLR-PT) explained NEURON’s measures for quality assurance. The Joint Transnational Calls are the core element of NEURON’s activities and during the last decade more than 100 projects were funded, comprising more than 400 research groups. The quality assurance measures that already have been implemented by NEURON include application of legal and ethical international/EU as well as national and institutional standards, application form specific for clinical trials, an international peer-review process, and continuous project monitoring. Key ele-ment of quality assurance is a precise shaping of the call text with distinct requirements for appli-cants. Standardized proposal templates including a detailed work plan, ethical issues, and a template for clinical studies with justification of the study design, key inclusion/exclusion criteria for recruitment and a statistical analysis are implemented. On the level of project selection, the review panel consists of internationally renowned experts selected for each call topic. The panel comprises clinicians as well as basic scientists. Careful matching of proposal topic and the reviewers‘ expertise ensures fair and qualified evaluations. On the funding level, only projects that fulfil the legal and ethical international/EU and national and institutional standards are funded. Funding for this kind of projects is dependent on a positive vote from the responsible ethical and legal committee(s).
Summary and future perspectives
Funders and researchers acknowledged the general problem of low reproducibility in biomedical research where most pre-clinical studies have a low statistical power leading to a high number of false positives and inflated effect sizes. The well-known bias to publish mainly positive results is another issue that hampers research. In order to improve this issue a system-wide approach involving funders, research institutions, academies & professional/scientific societies, publishers and individual researchers – at all levels – is required.
NEURON is committed to quality assurance and reproducibility to maintain the credibility of past and future investments of public resources, the trust in science and research, and the public acceptance of animal research. To address the issue of irreproducibility effectively and responsibly NEURON needs to implement a consistent approach to evidence quality. Positive examples that can serve as role models for potential measures by NEURON are the ARRIVE guidelines and the Experimental Design Assistant (EDA) that were developed by NC3Rs. The gap between pre-clinical studies and clinical trials may be bridged by pRCTs, thus facilitating the translation of research results into clinical application. Also, replications studies can provide accurate, robust and defensible evidence. And there are more challenges ahead: How to demonstrate and value/acknowledge reproducible and valid results? How to promote publication of null/negative results? Promoting data sharing – open access policies – and good data management is another challenge. And, not least, the improvement of support and training in experimental design and statistics is essential to ensure the cultural change.