Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 May:133:140-151.
doi: 10.1016/j.jclinepi.2020.11.003. Epub 2020 Nov 7.

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews

James Thomas  1 , Steve McDonald  2 , Anna Noel-Storr  3 , Ian Shemilt  4 , Julian Elliott  5 , Chris Mavergames  6 , Iain J Marshall  7
Affiliations

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews

James Thomas et al. J Clin Epidemiol. 2021 May.

Abstract

Objectives: This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews.

Methods: A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification.

Results: The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published.

Conclusions: The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.

Keywords: Automation; Cochrane Library; Crowdsourcing; Information retrieval; Machine learning; Methods/methodology; Randomized controlled trials; Searching; Study classifiers; Systematic reviews.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
The Cochrane Evidence Pipeline workflow, depicting the flow of records from the centralized search service, through machine and crowd classification services to the CENTRAL database.
Fig. 2
Fig. 2
Development and evaluation of the classifier, showing where the various data sets were used in the classifier development process.
Fig. 3
Fig. 3
Calibration plot showing bootstrap estimates of predicted vs. observed probabilities of an article being an RCT in Clinical Hedges dataset (each blue point represents an estimate of a model generated from one bootstrap sample) and the performance of the final model (orange). (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)
Fig. 4
Fig. 4
Distribution of classification scores for RCTs and non-RCTs in Clinical Hedges data set. RCT, randomized controlled trials.
Fig. 5
Fig. 5
RCTs “lost” by the classifier per 1,000 published, by year of publication, showing that the risk of “losing” a publication decreases over time.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Cochrane Cochrane Library. 2019. https://www.cochranelibrary.com/ Available at.
    1. Lefebvre C., Glanville J., Briscoe S., Littlewood A., Marshall C., Metzendorf M. Chapter 4: searching for and selecting studies. In: Higgins J., Thomas J., Chandler J., Cumpston M., Li T., Page M., editors. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. John Wiley & Sons; Chichester, UK: 2019. pp. 67–99.
    1. Bastian H., Glasziou P., Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. - PMC - PubMed
    1. Shojania K.G., Sampson M., Ansari M.T., Ji J., Doucette S. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147:224–233. - PubMed
    1. Macleod M.R., Michie S., Roberts I., Dirnagl U., Chalmers I., Ioannidis J.P.A. Biomedical research: increasing value, reducing waste. Lancet. 2014;383:101–104. - PubMed

Publication types

MeSH terms

LinkOut - more resources

-