Article Text
PDF
Abstract
Objectives To describe how systematic reviewers are reporting missing data for dichotomous outcomes, handling them in the analysis and assessing the risk of associated bias.
Methods We searched MEDLINE and the Cochrane Database of Systematic Reviews for systematic reviews of randomised trials published in 2010, and reporting a meta-analysis of a dichotomous outcome. We randomly selected 98 Cochrane and 104 non-Cochrane systematic reviews. Teams of 2 reviewers selected eligible studies and abstracted data independently and in duplicate using standardised, piloted forms with accompanying instructions. We conducted regression analyses to explore factors associated with using complete case analysis and with judging the risk of bias associated with missing participant data.
Results Of Cochrane and non-Cochrane reviews, 47% and 7% (p<0.0001), respectively, reported on the number of participants with missing data, and 41% and 9% reported a plan for handling missing categorical data. The 2 most reported approaches for handling missing data were complete case analysis (8.5%, out of the 202 reviews) and assuming no participants with missing data had the event (4%). The use of complete case analysis was associated only with Cochrane reviews (relative to non-Cochrane: OR=7.25; 95% CI 1.58 to 33.3, p=0.01). 65% of reviews assessed risk of bias associated with missing data; this was associated with Cochrane reviews (relative to non-Cochrane: OR=6.63; 95% CI 2.50 to 17.57, p=0.0001), and the use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology (OR=5.02; 95% CI 1.02 to 24.75, p=0.047).
Conclusions Though Cochrane reviews are somewhat less problematic, most Cochrane and non-Cochrane systematic reviews fail to adequately report and handle missing data, potentially resulting in misleading judgements regarding risk of bias.
- EPIDEMIOLOGY
- STATISTICS & RESEARCH METHODS
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first study to assess the reporting, handling and assessment of risk of bias associated with missing participant data among systematic reviews.
The study used systematic, rigorous and transparent approaches to define eligibility criteria, search for eligible studies, select studies and abstract data.
The broad eligibility criteria and inclusion of both Cochrane and non-Cochrane systematic reviews make the results more generalisable.
One limitation of the study is the restriction of our search to MEDLINE database.
The study did not analyse the individual trials that contributed to eligible systematic reviews.
Background
Although clinical trial investigators may strive to reduce the amount of missing data, they will in most instances fail to achieve complete follow-up.1 ,2 In a recent survey of the top five general medical journals, 87% of published trials reported participants with missing data for the primary outcome.3 The median percentage of participants with missing data was 6% (IQR 2–14%).3 Moreover, the way the trials handled missing participant data varied and was unclear in about a fifth of reports.3
The Cochrane Collaboration Handbook and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement recommend that the systematic review authors provide a description of incomplete reporting of data in the included trials.4 ,5 Furthermore, they recommend reporting how missing data are incorporated into the review findings. Although the Cochrane Handbook encourages the reanalysis of a study's effect estimate by including all randomised participants, it lacks a detailed guidance on how to handle missing data.4
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to judging the confidence in pooled effect estimates takes into account the risk of bias associated with missing data.6 The Cochrane Collaboration Handbook and PRISMA statement recommend that the authors of systematic reviews clearly describe how they assessed the risk of bias associated with missing data.4 ,5 The Cochrane Handbook describes situations in which an analysis can be judged to be at low or high risk of bias. For example, exclusion of participants due to ‘inefficacy’ or ‘failure to improve’ would be judged as introducing a high risk of bias.
There is a need to better understand how systematic review authors handle missing participant data in systematic reviews. Therefore, the primary objective of this methodological survey was to describe how systematic review authors report, handle and judge the risk of bias associated with missing data for dichotomous outcomes. Given the evidence that Cochrane reviews tend to be of higher methodological quality compared with other reviews,7 ,8 our secondary objective was to investigate differences in findings between Cochrane and non-Cochrane reviews.
Methods
Design overview
This study is part of a larger project (the ARROW project) examining methodological issues in systematic reviews; we have reported the full details of the methodology elsewhere.9 We used standard systematic review methods to survey how Cochrane and non-Cochrane systematic reviews report, handle and judge the risk of bias associated with missing participant data. We defined Cochrane systematic reviews as systematic reviews published in the Cochrane Database of Systematic Reviews. We considered all other systematic reviews as non-Cochrane systematic reviews. Since no human participants were involved, ethical approval was not required. The Cochrane Methods Innovation Fund funded this project.
Eligibility criteria
Eligible studies met the following inclusion criteria: described as a meta-analysis or a systematic review; describes a search strategy of at least one electronic database; published in the Cochrane Database of Systematic Reviews or in a journal indexed in MEDLINE;4 includes randomised controlled trials comparing an intervention with another intervention or with no intervention (or placebo) in humans; and reports measures of effect for at least one dichotomous outcome either from a single study or from a pooled analysis.
Search strategy
We conducted the searches in the MEDLINE database through the OVID interface (see online supplementary appendix S1). For non-Cochrane systematic reviews, we employed a systematic review filter designed by the Health Information Research Unit of McMaster University.10 For Cochrane systematic reviews, we restricted the search results to the Cochrane Database of Systematic Reviews as the journal type. We limited the searches to the year 2010. We used no language restrictions.
Selection process
We conducted the selection process for Cochrane and non-Cochrane reviews separately. Of the search results, we screened consecutive citations in a random order until we included the target sample size, with approximately equal representation of Cochrane and non-Cochrane reviews.
Teams of two reviewers conducted title and abstract screening independently and in duplicate. We obtained the full texts of citations judged as potentially eligible by at least one reviewer. The same teams of reviewers conducted full text screening. They resolved discrepancies by consensus, and when unsuccessful, with the help of a third reviewer. Reviewers participated in calibration exercises and used standardised and pilot-tested forms with detailed written instructions. They selected for each study a pairwise comparison and the most patient-important outcome.9 We defined a patient-important outcome as an outcome for which one would answer with ‘yes’ the following question: ‘If the patient knew that this outcome was the only thing to change with treatment, would the patient consider receiving this treatment if associated with side effects or cost?’.11 The patient-important outcome did not necessarily have to be the primary outcome.9 We used an online systematic review software application (DistillerSR, Evidence Partners, Ottawa, Canada; http://systematic-review.net/) to facilitate screening and data abstraction.
Data abstraction
Teams of two reviewers abstracted data using DistillerSR. They resolved discrepancies by consensus, and when unsuccessful, with the help of a third reviewer. They participated in calibration exercises, and used standardised and pilot-tested forms with detailed instructions.
We abstracted information about the following general characteristics of the systematic review:
Number of included trials;
Number of included participants in intervention and control arms;
Quality of the systematic review using the ‘assessment of multiple systematic reviews’ (AMSTAR) tool.12 The tool consists of 11 questions with 4 answer options (yes, no, can't answer, not applicable). The score for each study consisted of the number of ‘yes’ answers, with higher values indicating better scores (0–4: low quality; 5–8: moderate quality; 9–11 high quality);
Type of intervention (pharmacological vs surgery/invasive procedure);
Type of meta-analysis (standard meta-analysis vs meta-regression vs individual participant data meta-analysis);
Evaluation of the risk of bias (using Cochrane Risk of Bias (RoB) tool vs by dimensions vs point system scale);
Use of the GRADE approach to rate confidence in effect estimates;13
Source of funding (for profit source vs source other than for profit vs not funded);
Whether any of the authors reports industry ties.
We abstracted the following information regarding the collection and reporting of information about missing participant data within each systematic review:
Plan to collect the number of participants with missing data;
Plan to collect the reasons for missing participant data;
Reporting of the number of participants with missing data;
Reporting of categories for participants with missing data (eg, withdrawal of consent, cross-over, dropouts, non-adherence);
Reporting on missing participant data as a separate outcome;
Inclusion of information on missing data in tabular format.
We abstracted data on how each systematic review handled missing participant data:
Plan for handling missing categorical data;
Plan for sensitivity analysis using different methods for handling missing participant data;
Provision of a justification for the method(s) for handling missing participant data.
We also assessed whether the authors judged the risk of bias associated with missing participants using a specific tool as such as the Cochrane RoB tool. Finally, we assessed whether the implications of missing participant data were incorporated in the Results or Discussion sections.
Analysis
We conducted a descriptive analysis of all variables. We used frequencies and percentages for categorical variables, and median and IQR for continuous variables as these data were not normally distributed. We analysed the data combined as well as stratified by type of review (Cochrane vs non-Cochrane). For this stratification, we compared continuous variables using the independent t test or Wilcoxon test depending on the distribution of the data. We compared categorical variables using the χ2 test or Fisher's exact test if the expected event number is less than 5.
We conducted two multivariable logistic regression analyses and prespecified the following dependent variables: (1) whether the systematic reviewers used complete case analysis; and (2) whether the systematic reviewers judged the risk of bias associated with missing participant data. We included the following independent variables in our adjusted models: type of review (Cochrane vs non-Cochrane); number of included trials; type of intervention; use of the GRADE approach to rate confidence in effect estimates; and whether any of the authors reported ties to industry. Following descriptive analyses, we omitted the latter two independent variables from the first regression analysis due to the low number of events. We hypothesised that Cochrane reviews and the use of the GRADE approach, but not industry ties, would be associated with the use of complete case analysis and judgement of the risk of bias associated with missing data. We used SPSS statistical software, V.18.0 (SPSS INC, Chicago, Illinois, USA).
Sample size calculation
We originally calculated the sample size for the purpose of evaluating the association of study characteristics with the reporting of absolute effects (the ARROW project).9 As we used the same sample (N=202) to conduct the current study, we estimated whether its size would be appropriate for a regression analysis to study the association of study characteristics with using complete case analysis. The regression analysis includes five independent variables, and would require 10 events per variable. As we were not aware of studies providing an estimate for the dependent variable, we considered a conservative estimate of 30%. The resulting sample size is 167, making the 202 systematic reviews sample we have large enough for our purpose.
Results
Out of 2328 citations identified by the search strategy, we included a total of 202 systematic reviews: 98 Cochrane and 104 non-Cochrane reviews (figure 1). Table 1 reports the general characteristics of included studies, with a p value for the test of difference between these two types. Cochrane reviews included fewer trials and participants, less frequently conducted meta-regression, had a higher AMSTAR score, but more frequently addressed pharmacological interventions, assessed risk of bias, and used the GRADE approach for rating certainty in estimates.
General characteristics of the included SRs
Flow chart of the screening literature process (RCT, randomised controlled trial).
Reporting missing participant data
Table 2 reports the collection and reporting of information about missing participant data in the included systematic reviews. The percentages of reviews that reported plans for collecting the number and reasons for missing data were 34% and 17%, respectively. The percentages that actually reported the number and categories with potentially missing participant data were 26% and 19%, respectively. Five per cent of systematic reviews reported missing participant data as an outcome measure. About half of the reviews included information about missing participant data in a tabular format. Cochrane reviews compared favourably to non-Cochrane reviews for all these reporting variables.
The collection and reporting of information about missing participant data in Cochrane and non-Cochrane SRs
Handling missing participant data
Table 3 shows the characteristics of systematic reviews in terms of handling missing participant data. The percentage of systematic reviews reporting a plan for handling missing categorical data was 25%. The two most reported approaches were using complete case analysis and assuming no participants with missing data had the event (8.5% and 4%, respectively, of all 202 reviews). A small percentage of reviews (11%) reported a planned sensitivity analysis using different method(s) for handling missing participant data, with only 2% providing a justification for those methods.
Handling of, and assessing the risk of bias associated with missing participant data in Cochrane and non-Cochrane SRs
In our multivariable logistic regression, whether the systematic review used complete case analysis was associated only with the type of review (Cochrane vs non-Cochrane: OR=7.25; 95% CI 1.58 to 33.3, p=0.01). The variables for which no statistically significant associations were identified were: number of included trials and type of intervention.
Assessing risk of bias associated with missing participant data
Table 3 describes the assessment of the risk of bias associated with missing participant. Of all included systematic reviews, 65% reported assessing risk of bias associated with missing data: 42% used the Cochrane RoB tool; 23% used a tool other than the Cochrane RoB tool. Differences between Cochrane and non-Cochrane reviews were statistically significant (p<0.0001). Eleven per cent of included reviews discussed the implications of missing participant data in the Results section, and 10% addressed them in the Discussion section.
In our multivariable logistic regression, whether the systematic review judged the risk of bias associated with missing participant data was associated with Cochrane versus non-Cochrane type (OR=6.63; 95% CI 2.50 to 17.57, p=0.0001), and whether the systematic review used the GRADE approach to rate the confidence in effect estimates (OR=5.02; 95% CI 1.02 to 24.75, p=0.047). The variables for which no statistically significant associations were identified were: number of included trials, type of intervention and whether any of the authors reported ties to industry.
Discussion
Although Cochrane systematic reviews are less problematic, most Cochrane and non-Cochrane reviews do not adequately report and handle missing data. In general, 50% or less of Cochrane reviews met the criteria we explored, versus 20% or less of non-Cochrane reviews. Only 13% of reviews reported planned sensitivity analyses for handling missing participant data. Reporting on risk of bias associated with missing participant data was the least problematic item (65%). Better performance was associated with Cochrane versus non-Cochrane type, and whether the systematic review used the GRADE approach to rate confidence in effect estimates.
Our study has a number of strengths. This is the first study that assesses the reporting, handling and assessment of risk of bias associated with missing participant data among systematic reviews. We used explicit eligibility criteria, and sensitive search strategies to identify eligible studies. We also employed systematic, rigorous and transparent approaches to study selection and data abstraction, including calibration exercises, duplicate processes and use of standardised pilot-tested forms with detailed instructions. We included both Cochrane and non-Cochrane systematic reviews and used broad eligibility criteria to make our results more generalisable.
The main limitation of our study is that we did not review the individual trials that contributed to eligible systematic reviews, and the very low percentage of systematic reviews reporting on potential reasons for missing data might be due to the poor reporting of such information at the trial level. While the restriction of our search to MEDLINE database could have affected the representativeness of systematic reviews, we believe the included reviews represent those typically accessed by clinicians.
Although all reviews performed poorly in terms of reporting, Cochrane reviews performed better than non-Cochrane reviews. Likely explanations include the availability of a methodological guidance (ie, the Cochrane Handbook), and the use of standardised tables in Cochrane (ie, the Cochrane RoB tool). Another potential explanation is the lack of space constraints for the publication of Cochrane systematic reviews. It is important to note that non-Cochrane reviews are likely to be a highly heterogeneous in terms of methodological and reporting characteristics. Unfortunately, our study lacked the power to explore that hypothesis.
A recent study examined how 100 Cochrane systematic reviews and 100 non-Cochrane systematic reviews published in 2012 assessed risk of bias of primary studies and incorporated them into their statistical analysis and overall findings.7 The investigators found that incomplete outcome data, defined as missing outcome data due to attrition, were reported in 95% of Cochrane reviews compared with 61% of non-Cochrane reviews. We found substantially lower figures that are probably related to lower percentage of reviews using the GRADE approach in our sample (16% vs 45%).
That study found that 8% and 1% of Cochrane and non-Cochrane reviews, respectively, assessed incomplete outcome data for more than one outcome. In our study, we assessed whether the systematic reviews reported the number of participants with missing data for each outcome. The percentages were similarly very low at 2% and 0% for Cochrane and non-Cochrane reviews, respectively. This is problematic because missingness of data may actually vary across outcomes (eg, due to different follow-up times for different outcomes).
We recently suggested a simple guidance for addressing dichotomous data for participants excluded from analyses of randomised trials and for assessing the associated risk of bias.14 Briefly, the guidance suggests for the primary analysis, either a complete case analysis or making plausible assumptions about the outcomes of participants with missing data. When the primary analysis suggests important benefit, the guidance recommends sensitivity meta-analyses using relatively extreme assumptions that may vary in plausibility to assess the associated risk of bias. We have developed a similar guidance for continuous data, including situations where continuous data are measured with different instruments.15 ,16 Other authors have similarly published recommendations for addressing missing data in meta-analysis.17–20 Our findings highlight the need for better compliance with and enforcement of the Cochrane Collaboration Handbook and the PRISMA statement's recommendations related to reporting of missing participant data.4 ,5 This would be facilitated by better reporting of missing data at the trial level,21 given the systematic review authors can typically rely on what the authors of primary studies report, as well as easing of space constraints in reporting, of non-Cochrane reviews in particular. Systematic reviewers also need to assess the risk of bias associated with missing participant data, and how it affects the confidence in the effects estimates.
The findings have also implications for methodological research. A number of approaches for handling missing data in meta-analysis are available.14–20 We now need to compare the impact of these methods on the statistical significance of pooled effect estimates and on the associated quality of evidence.
Acknowledgments
The authors thank Ms Lara A Kahale for her help with developing the tables.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors EAA contributed to the conception and design of the paper, paper selection, data abstraction, data analysis, interpretation of results and manuscript drafting. EAA is the guarantor. PA-C contributed to the conception and design of the paper; design of the search strategy, paper selection and data abstraction; and interpretation of results. GHG contributed to the conception and design of the paper, and interpretation of results. AC-L contributed to the design of the search strategy. AC-L, IN, BCJ, SE, CEG, AR-M and OEZ contributed to paper selection and data abstraction. RB-P, XS, MB, JWB, ALI, AFI, LMG, RAM, AS, IS, AJS, KAOT, PV and RWMV contributed to paper selection, data abstraction and interpretation of results. QZ contributed to data analysis and interpretation of results. All authors revised the manuscript and approved.
Funding This project was funded by the Cochrane Methods Innovation Fund. PA-C was funded by a Miguel Servet research contract from the Instituto de Salud Carlos III (CP09/00137). LMG and AJS were funded by a Río Hortega research contract from the Instituto de Salud Carlos III (CM10/00014 and CM12/00168). MB was supported by santésuisse and the Gottfried and Julia Bangerter-Rhyner Foundation. JWB was supported by a New Investigator Award from the Canadian Institutes of Health Research and Canadian Chiropractic Research Foundation. The work of KAOT was supported by unrestricted grants from the Finnish Cultural Foundation, the Finnish Medical Foundation, Jane and Aatos Erkko Foundation and Sigrid Jusélius Foundation.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.