Learn more: PMC Disclaimer | PMC Copyright Notice
Evidence-based databases versus primary medical literature: an in-house investigation on their optimal use*†
Abstract
Objective: The objective is to investigate the effectiveness of evidence-based medicine (EBM) resources in providing evidence for complex clinical questions versus general care management questions to identify situations for their optimal use.
Methods: In this investigation, forty complex clinical questions were randomly selected from an in-house archival database of questions received by librarians during clinical rounds. An additional forty questions were selected from a list of general care management questions received by the library from Pathways teams. To measure the effectiveness of resources in answering the questions, a team of librarians was asked to search UpToDate, the Cochrane Database of Systematic Reviews, and HealthGate Clinical Guidelines (formerly, EBM Solutions). The team then established consensus on whether a question was fully or partially answered by one of the above-mentioned EBM resources and was instructed to record the instances in which the primary literature needed to be used to answer the question completely.
Results: The study found that the EBM resources completely answered 20.0% of the 40 complex clinical questions and 47.5% of the 40 general care management questions, partially answered 40.0% of the complex clinical questions and 22.5% of the general care management questions, and did not answer 40.0% of the complex clinical questions and 30.0% of the general care management questions.
Conclusion: The pervasive use of EBM resources in answering clinical questions is making it imperative for information specialists to develop an expertise on their appropriate use. By exploring their use in answering complex clinical questions and general care management questions, this paper underlines the strengths and weakness of EBM resources and provides information specialists with some basic knowledge about how these resources can be combined with the primary literature to strengthen their effectiveness.
INTRODUCTION
Though the ideological principles that serve as the foundations of evidence-based medicine (EBM) were criticized during their development in the nineteenth and early twentieth centuries, interest in evidence-based medicine among health care professionals has been growing in the past two decades [1–3]. Due to an exponential growth in the medical literature, a recognized inadequacy of traditional information sources, increases in the number of unmet information needs during clinical encounters, and practitioner time constraints, the teaching and practice of EBM principles have become increasingly prevalent [4, 5].
The development of systems such as ILIAD, Internist-I/QMR, and DXplain represent early attempts by medical informaticians to address physicians' needs for synthesized medical literature to support decisions. These systems integrated medical information into knowledgebases created through extensive reading, filtering, and synthesis of the medical literature, as well as consultation from experts, and were designed to aid the human mind in the diagnostic decision process [6–8]. Evaluations of decision support systems have demonstrated their ability to improve clinical performance in aspects of medical care, yet they have not been widely employed [9, 10]. In one paper, Miller described a potential reason: the lack of explicitness; “Unintelligible to its user, the consultant program would employ seemingly superhuman reasoning capabilities (the ‘Greek Oracle’) to solve the physician's diagnostic problem” [11].
While the decision support systems mentioned above were developed in the early 1970s to late 1980s, not until the advent of resources such as the Cochrane Collaboration in the 1990s were evidence-based systems widely applied [12]. Developed to address the need for synthesized medical literature, current EBM resources facilitate the integration of evidence into practice by providing summarized recommendations for clinical care. Explicit presentation of evidence and targeted synthesis of the medical literature have contributed to their enthusiastic acceptance by health care professionals. The rapid adoption and incorporation of EBM resources requires information specialists to maintain a thorough understanding of optimal situations for their use in education and practice. This paper reports results from a study by the Annette and Irwin Eskind Biomedical Library (EBL) investigating the most appropriate use of EBM databases.
STUDY RATIONALE
EBL has seven years of experience providing tailored packets of clinical evidence via an innovative clinical informationist program, the Clinical Informatics Consult Service (CICS). CICS librarians currently collaborate with nine hospital units to provide synthesized evidence in answer to complex patient-care questions [13]. Complex CICS questions are internally defined as those that are patient-specific, are multifaceted, and, because of complexity, require access to a diverse set of resources. These questions typically require an average of two or more hours to be answered by a highly skilled librarian.
EBL also has recently begun providing evidence to support medical center Pathway development teams. Pathways detail specific patient goals, treatment procedures, lab tests, drug regimens, and additional assessments, with patients advancing through each phase of care as goals are accomplished [14]. Evidence provided to Pathway development teams typically supports general patient-care issues. These services have provided EBL with experience in answering complex patient-specific questions as well as general care management questions.
The familiarity with both complex patient-specific CICS questions and Pathways questions has, in turn, prompted the study team to investigate the belief that EBM resources are better suited for general care management questions than complex clinical questions. The central hypothesis for this investigation is that EBM resources provide effective evidence for general care management questions, whereas complex clinical questions require search and synthesis of the primary literature.
METHODOLOGY
Question selection
Eighty study questions were drawn from the CICS and the Pathways services. Of the 256 complex CICS questions stored in an in-house database, 40 were selected using a random number generator program (Minitab v.13). The CICS in-house database serves as an archive of questions received by the librarians while on rounds. The database is restricted in access to the Vanderbilt University Medical Center and allows health care teams to revisit answers to previously posed questions as they incorporate them into patient care and educational efforts. As fewer questions were answered for Pathways due to the shorter time the library has provided the service, the 40 most recent Pathways questions were selected for the study. Table 1 provides examples of CICS complex clinical questions and Pathways questions.
Table 1 Examples of complex Clinical Informatics Consult Service (CICS) and general care management pathways questions
Database selection
The databases selected for the study were UpToDate, the Cochrane Database of Systematic Reviews, and HealthGate Clinical Guidelines (formerly, EBM Solutions). UpToDate and Cochrane were selected based on their high use in the medical center. HealthGate Clinical Guidelines was chosen due to the institutions' partnering role as one of the developing consortium medical centers.
Review panel
The review panel included six librarians chosen based on interest and willingness to participate in the study. Each librarian independently analyzed the contents of the eighty information packets provided for each question and subsequently searched the EBM databases for answers. The panel then met as a group to form consensus on whether a question was fully or partially answered by one of the EBM databases and recorded the instances in which the primary literature needed to be used to answer the question completely.
RESULTS
Analysis of evidence-based medicine (EBM) resource use in filtered packets
Figure 1 represents a baseline analysis of the packets selected for the study. As the selected questions ranged from 1997 to present, not all of the EBM resources were available at the time the questions were initially asked. The results show 95% (38 of 40) of the complex clinical questions and 85% (34 of 40) of the general care management questions were answered exclusively by the primary literature. Packets contained information from both EBM resources and the primary literature in 5% (2 of 40) of the complex clinical questions and in 15% (6 of 40) of the general care management questions.
Questions answered by an EBM resource
Figure 2 shows the consensus data from the review panel on the ability of an EBM resource to answer the CICS and Pathways questions. Full answers were defined as a question where all facets of the question were answered by the resources. Partial answers were defined as a question where some, but not all, of the facets were answered by the resources. The results show the EBM resources provided a full answer to 20.0% (8 of 40) of the complex clinical questions and 47.5% (19 of 40) of the general care management questions. The EBM resources provided a partial answer to 40.0% (16 of 40) of the CICS questions and 22.5% (9 of 40) of the Pathways questions. In 40.0% (16 of 40) of the complex clinical questions and 30.0% (12 of 40) of the Pathways questions, the EBM resources provided no answers.
DISCUSSION
To facilitate optimal use of EBM databases that health care professionals frequently use, it is important to have data that can validate librarians' intuitions on resources' strengths and weaknesses. The outcomes from this study support the hypothesis that EBM resources provide answers for general patient care management questions more frequently than for complex clinical questions. The results also showed that EBM resources provide more partial answers for the complex CICS questions than for the Pathways questions. This finding may be explained by the nature of the CICS questions, which are often multifaceted and arise from patients in the intensive care unit with complex clinical presentations. During data analysis, the review panel noted instances when an EBM resource would provide an answer to one or two facets of a question and not the question in its entirety.
As the EBM resources do not provide information to answer 35% (28 of 80) of the questions overall, the study results also demonstrate the importance of the primary medical literature in answering patient care questions. As health care teams increasingly employ aggregated information sources, health sciences librarians may use these data to further inform and educate users on the appropriateness of these resources.
This study has limitations that merit discussion. First, the small sample size in the number of selected questions may limit generalizability of study results. Second, the inability of an EBM resource to answer a question may be due to lack of coverage in particular subject areas. However, the authors feel that this study models the situation that is likely to occur in practice, as health care teams are more likely to consult EBM resources because of lack of time and expertise in proper searching of the primary literature.
CONCLUSION
EBM resources offer an added value for health care professionals by strategically summarizing evidence to a variety of clinical questions. As these resources are getting to the point of being ubiquitously used by clinical and research teams, it is imperative that information specialists are aware of their strengths and weaknesses. The results provide data to support the crucial role of librarians as resource experts and educators, because users may discriminate among resources and ignore strengths and weaknesses in favor of convenience.
By exploring the use of EBM resources to answer complex clinical questions and general care management questions, this paper provides information specialists with some basic knowledge that can be leveraged to increase understanding for the optimal use of these reources. In particular, the results give some understanding of how EBM resources can be combined with the primary literature to strengthen their effectiveness. Continued research will aid information professionals in characterizing and determining the appropriate use of EBM resources for patient care and health care education.
Acknowledgments
The authors gratefully acknowledge the assistance of Jennifer Biddle, Garad M. Davis, Shannon Mueller, and Katherine Szigeti in conducting the study research and the assistance of Annette Williams and Nila Sathe in preparing this manuscript.
Footnotes
* Based on a presentation at MLA '03, the 103rd Annual Meeting of the Medical Library Association, San Diego, California; May 5, 2003. MLA '03 Research Award Recipient.
† This work supported in part by NIH Grant No. 1 R01 LM07849-01 from the National Library of Medicine.
REFERENCES
- Hill GB. Archie Cochrane and his legacy. an internal challenge to physicians' autonomy? J Clin Epidemiol. 2000 Dec; 53(12):1189–92. [PubMed] [Google Scholar]
- Rangachari PK. Evidence-based medicine: old French wine with a new Canadian label? J R Soc Med. 1997 May; 90(5):280–4. [PMC free article] [PubMed] [Google Scholar]
- Morabia A. P.C.A. Louis and the birth of clinical epidemiology. J Clin Epidemiol. 1996 Dec; 49(12):1327–33. [PubMed] [Google Scholar]
- Sackett DL, Straus SE, Richardson WS, Rosenburg W, and Haynes BR. Evidence-based medicine: how to practice and teach EBM. 2nd ed. Edinburgh, UK: Churchill Livingstone, 2000. [Google Scholar]
- Ely JW, Osheroff JA, Ebell MH, Chambliss ML, Vinson DC, Stevermer JJ, and Pifer EA. Obstacles to answering doctors' questions about patient care with evidence: qualitative study. BMJ. 2002 Mar 23; 324(7339):710. [PMC free article] [PubMed] [Google Scholar]
- Warner HR, Haug P, Bouhaddou O, Lincoln M, Warner H, Sorenson D, Williamson JW, and Fan C. ILIAD as an expert consultant to teach differential diagnosis. In: Proceedings of the Twelfth Annual Symposium on Computer Applications in Medical Care. New York, NY: IEEE Computer Society Press, 1988. [Google Scholar]
- Miller RA, Pople HE, and Myers JD. Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med. 1982 Aug 19; 307(8):468–76. [PubMed] [Google Scholar]
- Hupp JA, Cimino JJ, Hoffer EP, Lowe HJ, and Barnett O. DXPlain: a computer-based diagnostic knowledge base. In: Proceedings of the Fifth Conference on Medical Informatics (MEDINFO 86): Elsevier Science Publishers, 1986. [Google Scholar]
- Hunt DL, Haynes RB, Hanna SE, and Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998 Oct 21; 280(15):1339–46. [PubMed] [Google Scholar]
- Berner ES. Diagnostic decision support systems: how to determine the gold standard? J Am Med Inform Assoc. 2003 Nov–Dec; 10(6):608–10. [PMC free article] [PubMed] [Google Scholar]
- Miller RA, Masarie FE. The demise of the “Greek Oracle” model for medical diagnostic systems. Methods Inf Med. 1990 Jan; 29(1):1–2. [PubMed] [Google Scholar]
- Sackett DL, Straus SE, Richardson WS, Rosenburg W, and Haynes BR. Evidence-based medicine: how to practice and teach EBM. 2nd ed. Edinburgh, UK: Churchill Livingstone, 2000. [Google Scholar]
- Jerome RN, Giuse NB, Gish KW, Sathe NA, and Dietrich MS. Information needs of clinical teams: analysis of questions received by the Clinical Informatics Consult Service. Bull Med Libr Assoc. 2001 Apr; 89(2):177–84. [PMC free article] [PubMed] [Google Scholar]
- Ozbolt J, Brennan G, and Hatcher I. PathworX: an informatics tool for quality improvement. Proc AMIA Symp 2001; 518–22. [PMC free article] [PubMed] [Google Scholar]