Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

doi:10.1136/bmjopen-2015-009368

. 2015 Sep 30;5(9):e009368.

doi: 10.1136/bmjopen-2015-009368.

Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

Elie A Akl¹, Alonso Carrasco-Labra², Romina Brignardello-Petersen³, Ignacio Neumann⁴, Bradley C Johnston⁵, Xin Sun⁶, Matthias Briel⁷, Jason W Busse⁸, Shanil Ebrahim⁹, Carlos E Granados¹⁰, Alfonso Iorio¹¹, Affan Irfan¹², Laura Martínez García¹³, Reem A Mustafa¹⁴, Anggie Ramírez-Morera¹⁵, Anna Selva¹³, Ivan Solà¹³, Andrea Juliana Sanabria¹³, Kari A O Tikkinen¹⁶, Per O Vandvik¹⁷, Robin W M Vernooij¹³, Oscar E Zazueta¹³, Qi Zhou¹⁸, Gordon H Guyatt¹¹, Pablo Alonso-Coello¹³

Affiliations

¹ Department of Internal Medicine, American University of Beirut, Beirut, Lebanon Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada.
² Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Santiago, Chile.
³ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Santiago, Chile Institute of Health Policy, Management & Evaluation, University of Toronto, Toronto, Ontario, Canada.
⁴ Department of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile.
⁵ Institute of Health Policy, Management & Evaluation, University of Toronto, Toronto, Ontario, Canada Department of Anesthesia & Pain Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada.
⁶ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Clinical Research and Evaluation Unit, Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, Chengdu, China.
⁷ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland.
⁸ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada The Michael G. DeGroote Institute for Pain Research and Care, McMaster University, Hamilton, Ontario, Canada.
⁹ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada Department of Medicine, Stanford Prevention Research Center, Stanford University, Palo Alto, California, USA.
¹⁰ Área de investigaciones, Facultad de Medicina, Universidad de La Sabana, Chía, Colombia.
¹¹ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Medicine, McMaster University, Hamilton, Ontario, Canada.
¹² Department of Cardiology, University of Louisville, Louisville, Kentucky, USA.
¹³ Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau-CIBER of Epidemiology and Public Health (CIBERESP- IIB Sant Pau), Barcelona, Spain.
¹⁴ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Departments of Medicine, Nephrology and Biomedical and Health Informatics, University of Missouri-Kansas City, Kansas City, Missouri, USA.
¹⁵ Health Care Development Division, IHCAI Foundation & Central America Cochrane, San José, Costa Rica.
¹⁶ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Departments of Urology and Public Health, Helsinki University Central Hospital and University of Helsinki, Helsinki, Finland.
¹⁷ Norwegian Knowledge Centre for the Health Services, Oslo, Norway Department of Medicine, Innlandet Hospital Trust, Gjøvik, Norway.
¹⁸ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada.

PMID: 26423858
PMCID: PMC4593136
DOI: 10.1136/bmjopen-2015-009368

Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

Elie A Akl et al. BMJ Open. 2015.

. 2015 Sep 30;5(9):e009368.

doi: 10.1136/bmjopen-2015-009368.

Authors

Affiliations

¹ Department of Internal Medicine, American University of Beirut, Beirut, Lebanon Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada.
² Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Santiago, Chile.
³ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Santiago, Chile Institute of Health Policy, Management & Evaluation, University of Toronto, Toronto, Ontario, Canada.
⁴ Department of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile.
⁵ Institute of Health Policy, Management & Evaluation, University of Toronto, Toronto, Ontario, Canada Department of Anesthesia & Pain Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada.
⁶ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Clinical Research and Evaluation Unit, Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, Chengdu, China.
⁷ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland.
⁸ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada The Michael G. DeGroote Institute for Pain Research and Care, McMaster University, Hamilton, Ontario, Canada.
⁹ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada Department of Medicine, Stanford Prevention Research Center, Stanford University, Palo Alto, California, USA.
¹⁰ Área de investigaciones, Facultad de Medicina, Universidad de La Sabana, Chía, Colombia.
¹¹ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Medicine, McMaster University, Hamilton, Ontario, Canada.
¹² Department of Cardiology, University of Louisville, Louisville, Kentucky, USA.
¹³ Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau-CIBER of Epidemiology and Public Health (CIBERESP- IIB Sant Pau), Barcelona, Spain.
¹⁴ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Departments of Medicine, Nephrology and Biomedical and Health Informatics, University of Missouri-Kansas City, Kansas City, Missouri, USA.
¹⁵ Health Care Development Division, IHCAI Foundation & Central America Cochrane, San José, Costa Rica.
¹⁶ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Departments of Urology and Public Health, Helsinki University Central Hospital and University of Helsinki, Helsinki, Finland.
¹⁷ Norwegian Knowledge Centre for the Health Services, Oslo, Norway Department of Medicine, Innlandet Hospital Trust, Gjøvik, Norway.
¹⁸ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada.

PMID: 26423858
PMCID: PMC4593136
DOI: 10.1136/bmjopen-2015-009368

Abstract

Objectives: To describe how systematic reviewers are reporting missing data for dichotomous outcomes, handling them in the analysis and assessing the risk of associated bias.

Methods: We searched MEDLINE and the Cochrane Database of Systematic Reviews for systematic reviews of randomised trials published in 2010, and reporting a meta-analysis of a dichotomous outcome. We randomly selected 98 Cochrane and 104 non-Cochrane systematic reviews. Teams of 2 reviewers selected eligible studies and abstracted data independently and in duplicate using standardised, piloted forms with accompanying instructions. We conducted regression analyses to explore factors associated with using complete case analysis and with judging the risk of bias associated with missing participant data.

Results: Of Cochrane and non-Cochrane reviews, 47% and 7% (p<0.0001), respectively, reported on the number of participants with missing data, and 41% and 9% reported a plan for handling missing categorical data. The 2 most reported approaches for handling missing data were complete case analysis (8.5%, out of the 202 reviews) and assuming no participants with missing data had the event (4%). The use of complete case analysis was associated only with Cochrane reviews (relative to non-Cochrane: OR=7.25; 95% CI 1.58 to 33.3, p=0.01). 65% of reviews assessed risk of bias associated with missing data; this was associated with Cochrane reviews (relative to non-Cochrane: OR=6.63; 95% CI 2.50 to 17.57, p=0.0001), and the use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology (OR=5.02; 95% CI 1.02 to 24.75, p=0.047).

Conclusions: Though Cochrane reviews are somewhat less problematic, most Cochrane and non-Cochrane systematic reviews fail to adequately report and handle missing data, potentially resulting in misleading judgements regarding risk of bias.

Keywords: EPIDEMIOLOGY; STATISTICS & RESEARCH METHODS.

Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

PubMed Disclaimer

Figures

**Figure 1**
Flow chart of the screening literature process (RCT, randomised controlled trial).

See this image and copyright information in PMC

Cited by

An alternative treatment for degenerative triangular fibrocartilage complex injuries with distal radioulnar joint instability: first experience with 48 patients.
Shaer SA, van der Palen J, Teunissen J, Fink A, van der Heijden B, Zöphel O; Hand Wrist Study Group. Shaer SA, et al. J Hand Surg Eur Vol. 2024 Feb;49(2):240-249. doi: 10.1177/17531934231197942. Epub 2023 Sep 11. J Hand Surg Eur Vol. 2024. PMID: 37694851 Free PMC article.
Methodological survey of missing outcome data in an alteplase for ischemic stroke meta-analysis.
Garg R. Garg R. Acta Neurol Scand. 2022 Sep;146(3):252-257. doi: 10.1111/ane.13656. Epub 2022 Jun 2. Acta Neurol Scand. 2022. PMID: 35652287 Free PMC article.
Pattern-mixture model in network meta-analysis of binary missing outcome data: one-stage or two-stage approach?
Spineli LM, Papadimitropoulou K, Kalyvas C. Spineli LM, et al. BMC Med Res Methodol. 2021 Jan 7;21(1):12. doi: 10.1186/s12874-020-01205-6. BMC Med Res Methodol. 2021. PMID: 33413138 Free PMC article.
Meta-Analyses Proved Inconsistent in How Missing Data Were Handled Across Their Included Primary Trials: A Methodological Survey.
Kahale LA, Khamis AM, Diab B, Chang Y, Lopes LC, Agarwal A, Li L, Mustafa RA, Koujanian S, Waziry R, Busse JW, Dakik A, Hooft L, Guyatt GH, Scholten RJPM, Akl EA. Kahale LA, et al. Clin Epidemiol. 2020 May 27;12:527-535. doi: 10.2147/CLEP.S242080. eCollection 2020. Clin Epidemiol. 2020. PMID: 32547244 Free PMC article.
Analytical methods used in estimating the prevalence of HIV/AIDS from demographic and cross-sectional surveys with missing data: a systematic review.
Mosha NR, Aluko OS, Todd J, Machekano R, Young T. Mosha NR, et al. BMC Med Res Methodol. 2020 Mar 14;20(1):65. doi: 10.1186/s12874-020-00944-w. BMC Med Res Methodol. 2020. PMID: 32171240 Free PMC article.

See all "Cited by" articles

References

1. Fleming TR. Addressing missing data in clinical trials. Ann Intern Med 2011;154:113–17. 10.7326/0003-4819-154-2-201101180-00010 - DOI - PMC - PubMed
1. Altman DG. Missing outcomes: addressing the dilemma. Open Med 2009;3:e21–3. - PMC - PubMed
1. Akl EA, Briel M, You JJ et al. . Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review. BMJ 2012;344:e2809 10.1136/bmj.e2809 - DOI - PubMed
1. The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Secondary Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] 2011. http://www.cochrane-handbook.org
1. Moher D, Liberati A, Tetzlaff J et al. , PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151:264–9, W64 10.7326/0003-4819-151-4-200908180-00135 - DOI - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

[1] Fleming TR. Addressing missing data in clinical trials. Ann Intern Med 2011;154:113–17. 10.7326/0003-4819-154-2-201101180-00010 - DOI - PMC - PubMed

[2] Fleming TR. Addressing missing data in clinical trials. Ann Intern Med 2011;154:113–17. 10.7326/0003-4819-154-2-201101180-00010 - DOI - PMC - PubMed

[3] Altman DG. Missing outcomes: addressing the dilemma. Open Med 2009;3:e21–3. - PMC - PubMed

[4] Altman DG. Missing outcomes: addressing the dilemma. Open Med 2009;3:e21–3. - PMC - PubMed

[5] Akl EA, Briel M, You JJ et al. . Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review. BMJ 2012;344:e2809 10.1136/bmj.e2809 - DOI - PubMed

[6] Akl EA, Briel M, You JJ et al. . Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review. BMJ 2012;344:e2809 10.1136/bmj.e2809 - DOI - PubMed

[7] The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Secondary Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] 2011. http://www.cochrane-handbook.org

[8] The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Secondary Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] 2011. http://www.cochrane-handbook.org

[9] Moher D, Liberati A, Tetzlaff J et al. , PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151:264–9, W64 10.7326/0003-4819-151-4-200908180-00135 - DOI - PubMed

[10] Moher D, Liberati A, Tetzlaff J et al. , PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151:264–9, W64 10.7326/0003-4819-151-4-200908180-00135 - DOI - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

Affiliations

Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

Authors

Affiliations

Abstract

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Other Literature Sources

Abstract

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Related information

LinkOut - more resources

Full Text Sources

Other Literature Sources