Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Dec 10;19(24):16601.
doi: 10.3390/ijerph192416601.

Medical Experts' Agreement on Risk Assessment Based on All Possible Combinations of the COVID-19 Predictors-A Novel Approach for Public Health Screening and Surveillance

Mohd Salami Ibrahim  1 , Nyi Nyi Naing  1 , Aniza Abd Aziz  1 , Mokhairi Makhtar  2 , Harmy Mohamed Yusoff  1 , Nor Kamaruzaman Esa  1 , Nor Iza A Rahman  1 , Myat Moe Thwe Aung  1 , San San Oo  1 , Samhani Ismail  1 , Ras Azira Ramli  1
Affiliations

Medical Experts' Agreement on Risk Assessment Based on All Possible Combinations of the COVID-19 Predictors-A Novel Approach for Public Health Screening and Surveillance

Mohd Salami Ibrahim et al. Int J Environ Res Public Health. .

Abstract

During the initial phase of the coronavirus disease 2019 (COVID-19) pandemic, there was a critical need to create a valid and reliable screening and surveillance for university staff and students. Consequently, 11 medical experts participated in this cross-sectional study to judge three risk categories of either low, medium, or high, for all 1536 possible combinations of 11 key COVID-19 predictors. The independent experts' judgement on each combination was recorded via a novel dashboard-based rating method which presented combinations of these predictors in a dynamic display within Microsoft Excel. The validated instrument also incorporated an innovative algorithm-derived deduction for efficient rating tasks. The results of the study revealed an ordinal-weighted agreement coefficient of 0.81 (0.79 to 0.82, p-value < 0.001) that reached a substantial class of inferential benchmarking. Meanwhile, on average, the novel algorithm eliminated 76.0% of rating tasks by deducing risk categories based on experts' ratings for prior combinations. As a result, this study reported a valid, complete, practical, and efficient method for COVID-19 health screening via a reliable combinatorial-based experts' judgement. The new method to risk assessment may also prove applicable for wider fields of practice whenever a high-stakes decision-making relies on experts' agreement on combinations of important criteria.

Keywords: agreement; combinations; dashboard-based rating; novel method; public health screening; public health surveillance; risk assessment.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Figures

Figure 1
Figure 1
Conceptual framework for the overall methodology of a combinatorial-based prediction model for the COVID-19 Public Health Screening and Surveillance.
Figure 2
Figure 2
Determining and proving 2048 unique combinations of 11 predictors.
Figure 3
Figure 3
Turning unique bits into unique combinations of alphabetical code of predictors.
Figure 4
Figure 4
Removing the combinations which contain the mutually exclusive predictors.
Figure 5
Figure 5
Linking the table of unique combinations with the dynamic display of the rating dashboard.
Figure 6
Figure 6
The final layout of the dashboard-based rating.
Figure 7
Figure 7
Validation of a section of the dashboard-based rating via comparison and inspection with the manual record.
See this image and copyright information in PMC

Similar articles

  • Expert assessment of exposure to carcinogens in Norway's offshore petroleum industry.
    Steinsvåg K, Bråtveit M, Moen B, Austgulen LV, Hollund BE, Haaland IM, Naerheim J, Svendsen K, Kromhout H. Steinsvåg K, et al. J Expo Sci Environ Epidemiol. 2008 Mar;18(2):175-82. doi: 10.1038/sj.jes.7500578. Epub 2007 Apr 25. J Expo Sci Environ Epidemiol. 2008. PMID: 17457323
  • Methods of assessing categorical agreement between correlated screening tests in clinical studies.
    Zhou TJ, Raza S, Nelson KP. Zhou TJ, et al. J Appl Stat. 2021;48(10):1861-1881. doi: 10.1080/02664763.2020.1777394. Epub 2020 Jun 9. J Appl Stat. 2021. PMID: 34305250 Free PMC article.
  • Using the Data Agreement Criterion to Rank Experts' Beliefs.
    Veen D, Stoel D, Schalken N, Mulder K, Van de Schoot R. Veen D, et al. Entropy (Basel). 2018 Aug 9;20(8):592. doi: 10.3390/e20080592. Entropy (Basel). 2018. PMID: 33265681 Free PMC article.
  • Dental care during COVID-19 pandemic: Follow-up survey of experts' opinion.
    Brunello G, Gurzawska-Comis K, Becker K, Becker J, Sivolella S, Schwarz F, Klinge B. Brunello G, et al. Clin Oral Implants Res. 2021 Oct;32 Suppl 21(Suppl 21):342-352. doi: 10.1111/clr.13783. Clin Oral Implants Res. 2021. PMID: 34196051 Free PMC article. Review.
  • Updated diagnosis, treatment and prevention of COVID-19 in children: experts' consensus statement (condensed version of the second edition).
    Shen KL, Yang YH, Jiang RM, Wang TY, Zhao DC, Jiang Y, Lu XX, Jin RM, Zheng YJ, Xu BP, Xie ZD, Liu ZS, Li XW, Lin LK, Shang YX, Shu SN, Bai Y, Lu M, Lu G, Deng JK, Luo WJ, Xiong LJ, Liu M, Cui YX, Ye LP, Li JF, Shao JB, Gao LW, Wang YY, Wang XF; China National Clinical Research Center for Respiratory Diseases; National Center for Children’s Health, Beijing, China; Group of Respirology, Chinese Pediatric Society, Chinese Medical Association; Chinese Medical Doctor Association Committee on Respirology Pediatrics; China Medicine Education Association Committee on Pediatrics; Chinese Research Hospital Association Committee on Pediatrics; China Non-government Medical Institutions Association Committee on Pediatrics; China Association of Traditional Chinese Medicine, Committee on Children’s Health and Medicine Research; China News of Drug Information Association, Committee on Children’s Safety Medication; Global Pediatric Pulmonology Alliance. Shen KL, et al. World J Pediatr. 2020 Jun;16(3):232-239. doi: 10.1007/s12519-020-00362-4. Epub 2020 Apr 24. World J Pediatr. 2020. PMID: 32333248 Free PMC article. Review.
See all similar articles

References

    1. Rajani R., Klein J.L. Infective endocarditis: A contemporary update. Clin. Med. 2020;20:31–35. doi: 10.7861/clinmed.cme.20.1.1. - DOI - PMC - PubMed
    1. Kirsch B., Aziz M., Kumar S., Burke M., Webster T., Immadi A., Sam M., Lal A., Estrada-Y-Martin R.M., Cherian S., et al. Wells Score to Predict Pulmonary Embolism in Patients with Coronavirus Disease 2019. Am. J. Med. 2021;134:688–690. doi: 10.1016/j.amjmed.2020.10.044. - DOI - PMC - PubMed
    1. Patel S. Calculated decisions: CURB-65 score for pneumonia severity. Emerg. Med. 2021;23((Suppl. 2)):CD1–CD2. - PubMed
    1. Ismail A., Ibrahim M.S., Ismail S., Aziz A.A., Yusoff H.M., Mokhtar M., Juahir H. Development of COVID-19 Health-Risk Assessment and Self-Evaluation (CHaSe): A health screening system for university students and staff during the movement control order (MCO) Netw. Model. Anal. Health Inform. Bioinform. 2022;11:21. doi: 10.1007/s13721-022-00357-3. - DOI - PMC - PubMed
    1. Ibrahim M.S., Hassan N.M., Aziz A.A., Yusoff H.M., Esa N.K., Makhtar M., Rahman N.I.A., Ismail S., Ramli R.A., Naing N.N., et al. Clinical manifestations of COVID-19; what have we learned from the global database? Bangladesh J. Med. Sci. 2022;21:702–709. doi: 10.3329/bjms.v21i3.59588. - DOI

Publication types

Grants and funding

This research was funded by the Ministry of Science, Technology, and Innovation Malaysia (MOSTI), the Ministry of Plantation Industries And Commodities Malaysia (MPIC), and Universiti Sultan Zainal Abidin (UniSZA) internal fund with grant numbers UniSZA/202/PPL/MTC (008), R0270 & R0271, and the Ministry of Higher Education Malaysia (MOHE) with Fundamental Research Grant Scheme (FRGS) grant number FRGS/1/2020/ICT06/UNISZA/02/1.
-