BY STEVE RANGER | EDITOR-IN-CHIEF
Reproducibility underpins the scientific method – the observational method for acquiring knowledge that has been fundamental to the conduct of science since at least the 17th century.
The scientific method involves careful observation coupled with rigorous scepticism, because cognitive assumptions can distort the interpretation of the observation.
Scientific inquiry includes creating a hypothesis that is testable through what is generally termed as inductive rather than deductive reasoning. That is, testing it through experiments and statistical analysis; and adjusting or discarding the hypothesis on the basis of the results obtained. All of this will be familiar to anyone who has ever worked on a research project.
Importantly, for the findings of a research study to be reproducible means that results obtained by experiment or an observational study – or indeed a statistical analysis of a data set – should be repeatedly obtained with a high level of reliability when the study is replicated.
This approach is most obviously applied in chemistry, for example, where a new route for a chemical synthesis should be reproducible both in the originating laboratory and during the development phase as a possible new entity, either as a pharmaceutical active ingredient or a commercially desirable chemical substance.
But in recent years, concern has been increasing that many published scientific results fail the test of reproducibility, generating a so-called reproducibility or replication crisis.
Back in 2016, a survey showed that over 70% of researchers had tried and failed to reproduce another scientist’s experiments, while more than half of those surveyed failed to reproduce their own experiments (Nature, 2016, 533, 452).
There were, however, some contradictory attitudes revealed by the survey. While some 52% of those surveyed agreed that there was a significant crisis of reproducibility, most respondents, around 71%, also said they still trusted the published results. And this confidence was most pronounced among chemists and physicists.
But, despite this recognition of the need for further work, when PLOS Biology published a survey of biomedical researchers, late last year, the results were very similar, with 72% of respondents agreeing that there is a reproducibility crisis, and 27% saying the crisis is significant (PLOS Biology, doi: 10.1371/journal.pbio.3002870). Yes, it’s the reproducibility problem, all over again.
Respondents in the survey also identified the leading cause of a lack of reproducibility as ‘pressure to publish’ with 62% indicating it ‘always’ or ‘very often’ contributes to this situation.
Some 54% of the respondents had run replications of their own previously published studies, while 57% had repeated other researchers’ work. In terms of institutions encouraging appropriate rigor in research, only 16% of respondents reported their institution had established procedures to improve reproducibility of biomedical research, while 67% said they believed their institution valued new research over the replication of previous work.
And in terms of providing training to enhance reproducibility, almost half of respondents said such training was not available at their institution.
Survey participants also reported few opportunities to gain funding for replication and 83% said they believed it would be harder to obtain funding for such work, compared with obtaining funding for new research.
So where do we go from here? While it’s understandable that organisations will focus more on new knowledge than repeating the work of others, it’s vital that all knowledge is on strong foundations.
More broadly, new technologies are going to help – like automation and robotics (see page 5) which are gradually appearing in more labs, and which hold out the promise of both taking away some of the more boring elements of research and also making experiments more consistent. Robot assistants should be able to conduct an experiment the same way every time without needing to take a coffee break in the middle.
AI is also likely to help by allowing researchers to capture and analyse data to discover the hidden patterns they may have missed; combined these two advances will help usher in data- and automation-driven research.
It’s also time for a change in attitudes towards research. This will require a focus on relevant training and other measures to improve reproducibility and to ensure the quality and reproducibility of research is appropriately and demonstrably valued.
After all, we don’t want to be complaining about the same problem in another decade’s time.
As well as ensuring we don’t repeat the mistakes of the past we’re also looking to the future this issue. On page 10 we take a look at a list of innovations across health and materials that could just change the world, a report on pioneering work to remove micropollutants from water is on page 18 and a look at how science is preparing for future health threats can be found on page 26. And on page 46 we meet the co-editors-in-chief of the new SCI Sustainability journal who talk about their plans.
Visit Blog
