Protocol for the development of a reporting guideline for umbrella reviews on epidemiological associations using cross-sectional, case-control and cohort studies: the Preferred Reporting Items for Umbrella Reviews of Cross-sectional, Case-control and Cohort studies (PRIUR-CCC)

Abstract

Introduction

There is evidence that the number of systematic reviews and meta-analyses in the literature has increased geometrically over the past two decades.^{1 2} Due to the increasing number of systematic reviews and meta-analyses on a given topic over the years, the field of knowledge synthesis has developed systematic reviews of systematic reviews, also called ‘reviews of reviews’, ‘overviews of (systematic) reviews’, ‘meta-reviews’ or ‘umbrella reviews’.^3–8 Umbrella reviews can include interventional studies or observational studies.⁹ Overviews of reviews and umbrella reviews ideally aim to provide a comprehensive and systematic synthesis following the steps of a systematic review (ie, literature search, methodological quality appraisal, quantitative analysis where feasible and appropriate, etc) with systematic reviews as the unit of analysis.^{2 9 10} The field has seen a sharp increase in the number of published overviews of reviews and umbrella reviews over the past decade. For example, just limiting to umbrella reviews, approximately 56 were published in 2010, while 560 (10 times increase in yearly publications) were published in 2021 (PubMed (umbrella review)).

There is important variability between and within the approach of overviews of reviews and umbrella reviews, making results hardly comparable.^11–13 This heterogeneity is not surprising, considering the large heterogeneity in the conception and implementation of both systematic reviews and meta-analyses.^{14 15} The methodology used to conduct systematic reviews and meta-analyses, as well as the quality of reporting, can affect conclusions and potentially lead to misleading interpretation of findings, misinforming policy makers, professional organisations and regulatory bodies, practitioners, patients, the public and other stakeholders. The quality and credibility of evidence synthesis efforts is also largely based on the quality of the credibility of their unit of inclusion (ie, individual studies). There is consensus that randomised controlled trials start from a higher credibility in the evidence-based medicine pyramid, and lower credibility is assigned to observational studies, which are more prone to bias. Interventional studies are not free from limitations, but in general experimental designs such as randomised controlled trials can protect from a number of biases, such as confounding by indication, or reverse causality. By contrast, observational evidence is prone to these and other biases, including excess of significance bias.¹⁶ Among observational studies, different study designs are adapted depending on the research question to be answered. For instance, studies measuring prognostic accuracy or prediction models are typically cohort studies that need internal development of the model, internal and external validation, calibration and accuracy measures.¹⁷ Cross-sectional studies can instead be used to investigate biomarkers or diagnostic accuracy of a given construct/test, or the prevalence of a disease. Other research questions, and typically epidemiological associations between two factors are generally explored with cohort, case-control and cross-sectional studies.^{18 19} More specifically, among studies investigating epidemiological associations, cross-sectional studies are typically used to measure the association between two factors, neglecting the direction of such association, while case-control and cohort studies are frequently used to measure associations between a construct of interest, and putative risk factors,^{1 14 20–23} or its outcomes,^{24 25} with the exposure occurring before the outcome. Research questions for which umbrella reviews of observational studies can be used are reported in table 1.

Reporting guidelines, defined as ‘ checklist, flow diagram or explicit text to guide authors in reporting a specific type of research, developed using explicit methodology’,²⁶ can be useful to improve the transparency, quality and reporting of individual studies, reviews or umbrella reviews. Interventional and observational evidence pose different methodological quality and reporting challenges, which are reflected from different reporting guidance for the two study designs. For observational evidence, several checklists of essential information to be reported in a paper are available for individual studies, systematic reviews and meta-analyses. For observational studies, the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network has disseminated Standards for Reporting Diagnostic accuracy studies,^{27 27} Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis²⁸ and Strengthening the Reporting of Observational Studies in Epidemiology.²⁹ For systematic reviews, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 (PRISMA 2020)³⁰ is a broad guide inclusive of a range of primary study designs, and more specific statements are available, that is, the PRISMA for Diagnostic Test Accuracy,³¹ and PRISMA for reviews including harms outcomes,³² or the guidance on conducting systematic reviews and meta-analyses of observational studies on aetiology³³ (table 2). The different checklists addressing different study designs and research questions well reflect their different methodological challenges, from original research to evidence synthesis.

Regarding overview of reviews and umbrella reviews, virtually all reporting checklists proposed so far have focused on interventional evidence. The first proposed checklist³⁴ was based on A MeaSurement Tool to Assess systematic Reviews quality assessment tool (AMSTAR)³⁵ and Cochrane guidance.³⁶ The same year, one further checklist for overviews of reviews³⁷ merged evidence from PRISMA for abstracts,³⁸ the Preferred reporting items for overviews of systematic reviews (PRIO) checklist³⁹ and the Overview Quality Assessment Questionnaire.⁴⁰ Then, a checklist for overviews of reviews⁴¹ was developed based on Cochrane recommendations,³⁶ and the older versions of PRISMA⁴² (and not PRISMA 2020³⁰), and AMSTAR^{35 43} (and not AMSTAR 2⁴⁴). Later, checklist was developed for systematic reviews of reviews including harms, called PRIO,³⁹ based on an older version of PRISMA,⁴² PRISMA harms³² and PRISMA for systematic review protocols.⁴⁵ The same group also published a checklist for abstract of overviews of reviews of healthcare interventions.⁴⁶ Recently, the Preferred Reporting Items for Systematic Reviews of Systematic Reviews/Meta-analyses (PRIOR) statement has been published to guide reporting of overviews of reviews of health interventions,³ adhering to EQUATOR guidance.²⁶ In PRIOR protocol,³ authors acknowledged that several relevant sources exist that provide guidance on overviews of reviews or umbrella reviews, but they did not adhere to guidance endorsed by the EQUATOR Network.

Regarding umbrella reviews (ie, observational evidence investigating epidemiological associations), to the best of our knowledge no EQUATOR-adherent guidance has been developed, registered with or disseminated by EQUATOR group, nor any specific checklist has been previously proposed. Given that PRIOR focuses on interventional evidence, and that different reporting guidelines are needed for observational evidence (cross-sectional, case-control, cohort studies) on epidemiological associations versus interventional evidence, the aim of this project is to develop evidence-based and agreement-based guideline PRIOR-extension for reporting umbrella reviews (ie, cross-sectional, case-control, cohort studies testing epidemiological associations), adhering to established guidance²⁶ and building on PRIOR statement.³ Specifically, this project will yield a Preferred Reporting Items for Umbrella Reviews of Cross-sectional, Case-control and Cohort studies (PRIUR-CCC), which will be published in a peer-reviewed journal and available via a dedicated website.

Methods and analysis

Study design

This study will follow EQUATOR guidance for developing reporting checklists,²⁶ and will be composed on four steps, namely project launch, literature review, Delphi survey and guideline statement preparation (figure 1).

Open in a separate window

Figure 1

Development of PRIUR-CCC statement flow diagram. EQUATOR, Enhancing the QUAlity and Transparency Of health Research.

Project launch

Project launch will consist of reaching an agreement on roles and responsibilities of core team members (eg, identifying stakeholders that will participate in the Delphi study). A core group of researchers have prepared the protocol of the project. These authors will be the core team of the project, and have extensive record of umbrella reviews on cross-sectional, case-control and cohort studies investigating epidemiological associations. The project’s day-to-day steps and its finalisation will be responsibility of shared first authors, and last author. All other authors will contribute to the protocol, literature review, initial set of checklist items, Delphi survey and final PRIUR-CCC statement. The identification and involvement of stakeholders will follow the Practical Guidance for Involving Stakeholders in Health Research from the Multi Stakeholder Engagement Consortium.⁴⁷ Reporting of involvement of patients and public will adhere to Guidance for Reporting Involvement of Patients and the Public 2 (GRIPP2).⁴⁸

Literature narrative review

We will review the systematic reviews conducted in the context of PRIOR development regarding available guidance to conduct umbrella reviews. Then, since two systematic reviews of umbrella reviews have already been conducted by the members of this project,^{2 12} those two systematic reviews will be used to assess methodology and reporting of included umbrella reviews. Given that PRIUR-CCC will not focus on diagnostic test accuracy and prediction models, we will not cover those study designs as they are out of scope for PRIUR-CCC. After having reviewed PRIOR documentation and having reviewed the two previous systematic reviews of umbrella reviews, we will extract key methodological factors from identified umbrella reviews, and publish a review of methodology and reporting of previous umbrella reviews. The findings of this review will be used to identify PRIOR domains and items where changes are needed, and to produce an initial PRIOR checklist to start the Delphi study with.

Delphi survey

We will conduct a Delphi study. Delphi studies use social science survey techniques to structure communication between participants in order to drive consensus and make a collective decision.⁴⁹ Typically, Delphi studies use several rounds of surveys in which participants, vote on issues. Between rounds, results of voting are aggregated and anonymised. They are then presented back to participants along with their own individual scores, and feedback on why others voted as they did.^{50 51} Among others, two strengths of this method include allowing for effective communication, limiting direct confrontation between individuals and giving participants the opportunity to consider the group’s thoughts and to compare and adjust their own score in the next round. Participants: we will aim to include 100 participants in Delphi study round 1, as done in PRIOR development.³ Participants to the Delphi study will meet any of the following criteria: (a) they have publication track of umbrella reviews in various fields, or (b) have publication track in Delphi surveys or (c) have publication track of reporting checklists of cross-sectional, case-control and cohort studies, or editors of peer-reviewed journals that published umbrella reviews on observational evidence or that have interest in umbrella reviews, or (d) funders of research or meta-research, or (e) practitioners, (f) policy makers, (g) evidence synthesis associations (eg, Cochrane, Campbell collaboration, Joanna Briggs Institute, others).³ Participants will be recruited via a two-step process. First, we have established a core group of participants based on criteria (a) or (b) or (c) described above (project launch), and past solid collaboration track record, that are authoring the present protocol. Second, additional stakeholders will be invited, according to (a) to (h) criteria above. Delphi study methods: this Delphi study will consist of three rounds. For the first two Delphi rounds, we will ask participants to complete an online survey which will be administered using Google Forms, structuring forms based on purpose-built platform for Delphi survey development and management.⁵² The third round will consist of a facilitated online consensus group meeting (using Zoom⁵³).

Round 1

Age, gender, geographical area (https://www.who.int/countries) and stakeholders group will be collected anonymously. Similar to what has been done for scoping reviews,⁵⁴ we will build on existing PRIOR statement, which parallels PRISMA 2020³⁰ statement, for consistency and continuity with existing established reporting guidance for evidence syntheses. All participants will be asked what items of PRIOR³ will have to (i) remain unchanged, (ii) what will have to be changed and how, (iii) what will have to be removed (three answer options, one possible choice). In addition, participants will have the possibility to propose new items. As a starting point, the core team composed of authors of this protocol will provide a set of suggested items for participants to vote on. This starting set of items will be built based on experience in umbrella reviews. Participants will respond indicating their agreement/disagreement to have each item included in the PRISMA-CCC reporting guideline. Participants will be provided with a free-text box to fill in with additional comments to explain why they voted how they did or to propose wording amendments to the item, or new items. To decide to keep or remove a PRIOR item, or to add a new item, will require a minimum of 80% consensus among participants (based on findings from a systematic review of Delphi studies⁵⁵). Items where an 80% consensus has not been reached, as well as changes to PRIOR items and new items will be voted on again in round 2. We anticipate the survey will take about 20 min to complete and will provide participants with a 3-week window to take part, with reminders sent after 1 and 2 weeks, respectively. We will pilot test the survey among the authors of the present protocol.

Round 2

All participants who completed round 1 of the Delphi will be re-invited to take part in round 2. Items which achieved consensus in round 1 will be shared with the participants. We will then ask participants to re-vote on any items that did not reach consensus or that were newly suggested. When re-voting on items that appeared in round 1, participants will also be provided with all comments provided by participants to justify their responses. Again, an 80% of consensus will be used to determine what items to include/exclude from the PRIUR-CCC guideline. If a new proposed item will be overlapping with existing items, Delphi moderator will add a comment to the new proposed item pointing to existing overlapping item.

Round 3

The core group and a purposeful selection of participants will be invited to round 3. We will aim to ensure representation from each of our geographically diverse institutions, each of the stakeholder groups and demographic variables including gender. Based on previous Delphi surveys and feasibility, we will invite no more than 30 participants in total to ensure feasibility to round 3.^{56 57} Round 3 will be moderated by core group members authoring the protocol, who will rotate every two items. We will present all participants with the results (ie, frequency of responses for each item, comments, changes and new items) of round 2 of the Delphi prior to the meeting and summarise these again briefly at the start of the consensus meeting. Participants will have the opportunity to discuss outstanding items one-by-one. They will then be asked to vote anonymously using real-time voting technology available via Zoom on each of these items. While diverse time zones will present challenges in other ways a virtual meeting may foster equity, diversity and inclusion of participants who might otherwise not have had funding or capacity to travel to an in-person event.

Analysis

For completers and drop-outs, demographics and responses will be presented using descriptive statistics using SPSS28. We will identify which items have and have not reached consensus for inclusion or exclusion based on our definition of 80% agreement and report this information for each Delphi round. The list of items identified for inclusion in PRIUR-CCC will be collated after round 3 and we will report the outcomes of participants ranking of these items in a table.

Timeline

The project has not started yet, and will be started in January 2024 and completed by December 2024.

Guidance statement

Co-first and last authors of this protocol will prepare the first draft of final PRIUR-CCC statement, that will then be approved after reiteration with other authors. This PRIUR-CCC statement will include the report of the whole project, the PRIUR-CCC checklist, the PRIUR-CCC flow diagram and an explanatory document that will inform on how to use PRIUR-CCC, with examples. PRIUR-CCC statement will be published in peer-reviewed journals, and on dedicated platforms. The Delphi process reporting will be informed by the Conducting and REporting of DElphi Studies checklist⁵⁸ (table 3).

Table 3

CREDES checklist for survey studies

Items of reporting	Reported on page
Purpose and rationale: the purpose of the study should be clearly defined and demonstrate the appropriateness of the use of the Delphi technique as a method to achieve the research aim. A rationale for the choice of the Delphi technique as the most suitable method needs to be provided.
Expert panel: criteria for the selection of experts and transparent information on recruitment of the expert panel, sociodemographic details including information on expertise regarding the topic in question, (non)response and response rates over the ongoing iterations should be reported.
Description of the methods: the methods employed need to be comprehensible; this includes information on preparatory steps (How was available evidence on the topic in question synthesised?), piloting of material and survey instruments, design of the survey instrument(s), the number and design of survey rounds, methods of data analysis, processing and synthesis of experts’ responses to inform the subsequent survey round and methodological decisions taken by the research team throughout the process.
Procedure: flow chart to illustrate the stages of the Delphi process, including a preparatory phase, the actual ‘Delphi rounds’, interim steps of data processing and analysis and concluding steps.
Definition and attainment of consensus: it needs to be comprehensible to the reader how consensus was achieved throughout the process, including strategies to deal with non-consensus.
Results: reporting of results for each round separately is highly advisable in order to make the evolving of consensus over the rounds transparent. This includes figures showing the average group response, changes between rounds as well as any modifications of the survey instrument such as deletion, addition or modification of survey items based on previous rounds.
Discussion of limitations: reporting should include a critical reflection of potential limitations and their impact of the resulting guidance.
Adequacy of conclusions: the conclusions should adequately reflect the outcomes of the Delphi study with a view to the scope and applicability of the resulting practice guidance.
Publication and dissemination: PRIUR-CCC will be published in peer-reviewed journals, with details on methodology (including CREDES checklist) in main text or supplementary material.

Open in a separate window

CREDES, Conducting and REporting of DElphi Studies; PRIOR, Preferred Reporting Items for Overviews of Reviews; PRIUR-CCC, Preferred Reporting Items for Umbrella Reviews for cross-sectional, case-control, cohort studies.

Discussion

This protocol for reporting guidance of umbrella reviews of epidemiological associations is needed to address specific reporting challenges of observational studies. PRIUR-CCC will provide a reporting framework to guide future umbrella reviews of observational studies assessing epidemiological associations, that can be used from researchers, reviewers, funders and editors to evaluate the transparency and quality of reporting of umbrella reviews, across different research questions. It will follow the guidance for reporting checklist, which are standard in the field. We propose that having reporting guidelines is of crucial relevance when included studies follow an observational design, with high baseline risk of confounding factors and numerous sources of bias potentially guiding misleading results.^{11 59} We aim to publish the report of this project including the final PRIUR-CCC guidelines. The main limitation of this study will be inherent in the Delphi study design, where only completers of all rounds fully contribute to the project.

Supplementary Material

Footnotes

References