Apparently conclusive meta-analyses on interventions in critical care may be inconclusive-a meta-epidemiological study
- PMID: 31200004
- DOI: 10.1016/j.jclinepi.2019.05.011
Apparently conclusive meta-analyses on interventions in critical care may be inconclusive-a meta-epidemiological study
Abstract
Objectives: Risks of random type I and II errors are associated with false positive and false negative findings. In conventional meta-analyses, the risks of random errors are insufficiently evaluated. Many meta-analyses, which appear conclusive, might, in fact, be inconclusive because of risks of random errors. We hypothesize that, for interventions in critical care, false positive and false negative findings frequently become apparent when accounting for the risks of random error. We aim to investigate to which extent apparently conclusive conventional meta-analyses remain conclusive when adjusting statistical significance levels and confidence intervals considering sparse data and repeated testing through Trial Sequential Analysis (TSA).
Methods: We searched The Cochrane Library, MEDLINE, and EMBASE for reviews of interventions in critical care. We used TSA with the relative risk reduction from the estimated meta-analyzed intervention effects adjusted for heterogeneity based on the observed diversity. We report proportions of meta-analyses and potential inconclusive findings of positive, neutral, and negative conclusions based on conventional naïve meta-analyses, which use an alpha of 5% and 95% confidence intervals. In TSA-controlled meta-analyses showing a beneficial or harmful intervention effect, we assessed the risk of bias by six Cochrane domains.
Results: A total of 464 reviews containing 1,080 meta-analyses of (co-)primary outcomes were analyzed. From the 266 conventional meta-analyses suggesting a beneficial effect, 133 (50%) were true positive and 133 (50%) were potentially false positive according to TSA. From the 755 conventional meta-analyses suggesting a neutral effect, there were 214 (28%) true neutral and 541 (72%) were potentially false neutral according to TSA. From the 59 conventional meta-analyses suggesting a harmful effect, 17 (29%) were true negative and 42 (71%) were potentially false negative according to TSA. When the true beneficial and true harmful meta-analyses according to TSA were evaluated for risk of bias, new TSAs conducted on only trials with overall low risk of bias showed only firm evidence of a beneficial effect on one outcome and a harmful effect on one outcome.
Conclusions: Of all meta-analyses in critical care, a large proportion may reach false conclusions because of unknown risks of random type I or type II errors. Future critical care meta-analyses should aim for establishing an effect of interventions accounting for risks of bias and random errors.
Keywords: Meta-analysis; Meta-epidemiological study; Trial sequential analysis.
Copyright © 2019 Elsevier Inc. All rights reserved.
Similar articles
-
Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses.Int J Epidemiol. 2009 Feb;38(1):287-98. doi: 10.1093/ije/dyn188. Epub 2008 Sep 29. Int J Epidemiol. 2009. PMID: 18824466
-
Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses.J Clin Epidemiol. 2008 Aug;61(8):763-9. doi: 10.1016/j.jclinepi.2007.10.007. Epub 2008 Apr 14. J Clin Epidemiol. 2008. PMID: 18411040
-
Vitamin D supplementation for prevention of cancer in adults.Cochrane Database Syst Rev. 2014 Jun 23;(6):CD007469. doi: 10.1002/14651858.CD007469.pub2. Cochrane Database Syst Rev. 2014. PMID: 24953955 Review.
-
Evidence based evaluation of immuno-coagulatory interventions in critical care.Dan Med Bull. 2011 Sep;58(9):B4316. Dan Med Bull. 2011. PMID: 21893014 Review.
-
Optimal information size in trial sequential analysis of time-to-event outcomes reveals potentially inconclusive results because of the risk of random error.J Clin Epidemiol. 2013 Jun;66(6):654-9. doi: 10.1016/j.jclinepi.2012.11.007. Epub 2013 Feb 8. J Clin Epidemiol. 2013. PMID: 23403248
Cited by
-
Percutaneous catheter drainage versus percutaneous needle aspiration for liver abscess: a systematic review, meta-analysis and trial sequential analysis.BMJ Open. 2023 Jul 30;13(7):e072736. doi: 10.1136/bmjopen-2023-072736. BMJ Open. 2023. PMID: 37518084 Free PMC article.
-
Cerebral Oximetry in Preterm Infants-To Use or Not to Use, That Is the Question.Front Pediatr. 2022 Feb 2;9:747660. doi: 10.3389/fped.2021.747660. eCollection 2021. Front Pediatr. 2022. PMID: 35186815 Free PMC article.
-
Current evidence of survival benefit between chest-compression only versus standard cardiopulmonary resuscitation in out-of-hospital cardiac arrest : Updated systematic review and meta-analysis of randomized controlled trials with trial sequential analysis.Herz. 2021 Sep;46(Suppl 2):198-208. doi: 10.1007/s00059-020-04982-4. Epub 2020 Sep 25. Herz. 2021. PMID: 32975628 English.
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources
Medical
Research Materials