Learn more: PMC Disclaimer | PMC Copyright Notice
Obstacles to answering doctors' questions about patient care with evidence: qualitative study
Abstract
Objective
To describe the obstacles encountered when attempting to answer doctors' questions with evidence.
Design
Qualitative study.
Setting
General practices in Iowa.
Participants
9 academic generalist doctors, 14 family doctors, and 2 medical librarians.
Main outcome measure
A taxonomy of obstacles encountered while searching for evidence based answers to doctors' questions.
Results
59 obstacles were encountered and organised according to the five steps in asking and answering questions: recognise a gap in knowledge, formulate a question, search for relevant information, formulate an answer, and use the answer to direct patient care. Six obstacles were considered particularly salient by the investigators and practising doctors: the excessive time required to find information; difficulty modifying the original question, which was often vague and open to interpretation; difficulty selecting an optimal strategy to search for information; failure of a seemingly appropriate resource to cover the topic; uncertainty about how to know when all the relevant evidence has been found so that the search can stop; and inadequate synthesis of multiple bits of evidence into a clinically useful statement.
Conclusions
Many obstacles are encountered when asking and answering questions about how to care for patients. Addressing these obstacles could lead to better patient care by improving clinically oriented information resources.
Introduction
Doctors are urged to practise evidence based medicine when faced with questions about how to care for their patients.1 They are advised to ask questions that can be answered with evidence and to evaluate the results of original research.1–3 But this advice may be difficult to follow in the pressurised atmosphere of a busy practice, where doctors are more likely to seek answers from readily available sources.4–6 Doctors are overwhelmed by the amount of information available, yet they often cannot answer their questions about specific clinical problems.5,7–9
Much has been written about the qualities of a good question but little about the qualities of a good answer.1–3 Traditionally, the burden has been placed on the practitioner, who is told to ask “well built clinical questions” and to find the “best available evidence” to answer them.1,3 We decided to shift the burden from the practitioner to the researcher and the author, who should address questions that occur in practice and synthesise original research so that it can be directly applied to patients.
We aimed to describe the range of obstacles that occur when trying to obtain evidence based answers to real clinical questions. We sought to build a taxonomy that characterises the problems that arise when searchers attempt to answer doctors' questions. This taxonomy could serve as a basis for better resources of knowledge and more accessible information within these resources. It could also guide strategies for finding relevant information in current resources. Doctors need up to date, high quality answers at the point of care within minutes.5 Before these objectives can be met with new information systems, the problems with current resources and search strategies need to be described.
Methods
Selection of questions
We collected 1101 questions from 103 family doctors in Iowa by using observations. The participants and procedures for data collection for this aspect of the study are described elsewhere.5 Briefly, after each consultation an observer asked the doctor to report any questions that occurred about how to care for the patient. We collected straightforward questions (“What is the dose of metformin?”) as well as vague uncertainties that would normally be kept to oneself (“I'm not sure what this rash is, but I'm going to call it a contact dermatitis for now.”). Using computer generated random numbers from a uniform distribution, we selected a random sample of 200 questions. Some of these questions were not amenable to evidence based answers (for example, “What is causing her abdominal pain?” “Is it ethical for me to take care of my own file clerk, who has back pain and wants a work excuse?”). Through an iterative process of reviewing questions, creating a classification scheme, coding questions, and revising the classification scheme, we developed a method of identifying questions that were potentially answerable with evidence. This iterative process, which has been described elsewhere, led to the development of an “evidence taxonomy” (box).10,11 Using this taxonomy, we found that 106 questions (53% of the original 200) could potentially be answered with evidence.
After listing these 106 questions in random order, one of us (JE) answered the first 10, the investigators answered two, and JE answered eight, totalling 20. We agreed on three criteria for selecting the two questions to be answered by all investigators: the question should be clearly stated, there should be a high likelihood of finding good quality evidence to answer it, and the answer should potentially have an impact on patient care. By using these criteria we selected “What is the proper treatment of gastro-oesophageal reflux disease (GERD)?” and “What should I use for atopic dermatitis?”
Answering questions
We did not follow a standardised search strategy because we wanted to study obstacles related to the strategy. We searched textbooks, journal articles, and various computer applications, but we did not seek individual consultations with humans. Working independently, the investigators completed searches that they thought were sufficient to avoid missing important evidence. While searching, the investigators used a modified “think aloud” method to write field notes that documented the obstacles they encountered.12,13
Development of the taxonomy
We used three data sources to develop the initial taxonomy. The primary source consisted of obstacles documented in field notes written by the investigators as they attempted to answer the questions. The second source comprised frustrations that the investigators had encountered while answering other clinical questions. The third source consisted of problems reported in the literature.1,14–16 The obstacles were described and organised into a taxonomy by using qualitative text analysis. The taxonomy was developed with an approach in which initial “codes” (obstacles described in the “think aloud” field notes) were augmented with obstacles described in the literature and previously encountered by the investigators.17
Validation of the taxonomy
To help validate the taxonomy, we first asked four volunteers (two medical librarians and two university family doctors) to answer four additional questions from the same dataset. Each volunteer coded their own field notes and identified obstacles that were not optimally characterised in the existing taxonomy. Secondly, we asked 21 practising doctors (purposively selected from a list of former trainees from practices in Iowa) to describe on paper the problems they encountered when attempting to answer one of their own questions. Thirdly, we completed 16 half day observation periods involving four randomly selected practising doctors in Iowa (four observation periods per doctor). We asked these doctors to “think aloud” as they attempted to answer their own questions. Based on these three additional sources of data, we added four obstacles to the taxonomy. The final version of the taxonomy was approved by all investigators.
Results
The box shows the taxonomy of obstacles, with descriptions of each obstacle. The taxonomy was organised according to the steps in asking and answering questions18–20: recognise a gap in knowledge, formulate a question, search for relevant information, formulate an answer, and use the answer to direct patient care. Most of the obstacles were supported by the data we obtained, but a few were primarily generated from the previous experiences of the investigators or from the literature. These distinctions are noted in the box.
Our methods did not allow a formal frequency analysis, but several obstacles seemed particularly salient because they recurred in the various procedures for data collection, and they were characterised as fundamental problems by the investigators and practising doctors. These were the excessive time required to find information, difficulty modifying the original question, which was often vague and open to interpretation, difficulty selecting an optimal search strategy, failure of a seemingly appropriate resource to cover the topic, uncertainty about how to know when all the relevant evidence has been found so that the search can stop, and inadequate synthesis of multiple bits of evidence into a clinically useful statement.
The obstacles related to evidence fell into two main categories. Firstly, the available evidence was inadequate to directly answer the question either because studies had not addressed the question (“Is smoking a risk factor for sinusitis?”) or because the studies that had addressed the question provided incomplete information. For example, when answering the question about treating gastro-oesophageal reflux disease, we found rigorous comparisons between lansoprazole and placebo and between omeprazole and placebo but the comparisons between lansoprazole and omeprazole were less definitive.23–25 Secondly, even when the evidence was adequate to answer the question, further obstacles hindered its use in the clinical setting. Available evidence often consisted of individual study results, which had not been synthesised or interpreted for clinicians. The following field notes were written by one of the investigators as he attempted to answer the question, “What should I use for atopic dermatitis?”
Therapies include: ciclosporin—possibly effective; borage oil—probably not effective; UVA1—works; primrose oil with water and oil emulsion—probably effective; topical doxepin—probably not effective; mite elimination—likely to be effective; topical cromolyn—likely to be effective; topical tacrolimus—probably effective; SEZ ASM 981—possibly effective.
Each investigator spent a median of 95 minutes (range 13 to 639 minutes) answering the question about gastro-oesophageal reflux and 45 minutes (range 17 to 374 minutes) answering the question about atopic dermatitis. For all 20 questions, JE spent a median of 327 minutes per question (range 90 to 1075 minutes). The question that took the shortest time was “Are there any drug interactions with St John's wort, specifically with SSRIs?” This question was highly specific, and it soon became apparent that there was almost no evidence to answer it. The question that took the longest time was “Why is there an increasing incidence of asthma, just in general?” The literature on this topic contained extensive speculation and many theories based on observational studies. It was difficult to know when to stop looking for a definitive study, but eventually it became clear that a definitive study did not exist. The highest level of evidence available was a randomised clinical trial for seven of the 20 questions, an observational study for eight questions, and an opinion for the remaining five questions.
The final version of the taxonomy comprised 59 obstacles. The four volunteer coders used 35 problems to code their field notes and made four suggestions to improve the taxonomy. For example, both librarians noted their lack of medical training as an obstacle to formulating an answer. Ten of the 21 practising doctors responded to our request to describe obstacles that arose as they answered one of their own questions. All of the obstacles reported by these doctors had been described in the existing taxonomy. Also, we collected 96 questions during 16 office observations from four additional practising doctors. These data led to the addition of four obstacles to the taxonomy: failure to initiate a search due to doubt about the existence of relevant information, ready availability of consultation, which leads to a referral rather than a search, uncertainty about the meaning of null search results, and resource not clinically oriented.
Discussion
Obstacles arise when searching for evidence based answers to doctors' questions: we identified 59. Among the most salient were inadequate time to search for information, failure of the resource to address the topic, and inadequate synthesis of multiple bits of evidence into a clinically useful statement. Practising doctors often decided not to pursue their questions because they doubted the existence of useful information in available resources.
Other studies
In a study of internists in Los Angeles, many questions were phrased in patient specific terms that would make it difficult to find answers from generally available resources.9 For example, a doctor would ask “Should I test the serum procainamide level in this patient?” rather than “What are the indications for measuring serum procainamide?” We excluded such patient specific questions, although we could have modified them into a more general form. In the study from Los Angeles, doctors reported lack of time as the most common barrier to finding information.9 Other investigators have identified obstacles involving computers and the internet.14,16 One study cautioned against the uncritical use of clinical trial results for direct patient care.15
Limitations
Although we sought to build a comprehensive list of problems, the taxonomy we developed was primarily based on only 20 questions, which came from a homogeneous group of doctors. However, we attempted to validate our findings by applying the original taxonomy to the obstacles encountered while answering an additional 106 questions, generated by 14 practising doctors (96 questions from consultations plus 10 mailed questions). Our taxonomy's framework was based on the steps in the process of asking and answering questions, but other frameworks could have been used. We tended to blame the author for difficulties the doctor might encounter when searching for information. While we believe that enhancing search skills could overcome some of these difficulties, we chose to focus on how resource developers could address the problems with retrieval of information we identified.
Implications
After quantifying and prioritising the obstacles we found, the taxonomy we developed could be used to write recommendations for authors as they attempt to produce clinically useful material. Authors will be most effective if they anticipate the needs of busy clinicians who often have only a minute or two to find information.5 For example, authors who name the drug of choice for a specific condition could include essential prescribing information (dosage, drug interactions, safety in pregnancy), so that the clinician does not waste time consulting another resource. Clinically oriented resources could be written in a question and answer style rather than a disease and topic style. The ongoing surveillance of doctors' changing questions could help keep resources current. Questions without adequate answers could help guide research and funding priorities. Until such research is completed, such questions may prompt the use of holistic clinical care and other alternatives. We often found it helpful to modify questions from the way they were originally stated by the doctor. Such modifications could be developed into recommendations for doctors, as they formulate their questions, and for intermediary searchers, who may play a larger part in the future, as they help doctors practise the best medicine.26
Conclusions
To meet the needs for clinical information, doctors must be aware of their gaps in knowledge and then formulate questions that can be addressed by available resources or patient specific consultations. When faced with a gap in knowledge, doctors must decide whether to do the best they can with their current knowledge or to expand that knowledge by formulating and answering a question. Practising doctors do not have time to search multiple sites or scroll through long text. Nor do they have time to search multiple textbooks or perform literature searches for most of their questions. They need to pick the right resource the first time, the information in that resource needs to be readily found, and all the information must be there. Although it remains to be shown, we believe that systems designed to overcome the obstacles we identified will improve the asking and answering of questions and potentially patient outcomes.
Acknowledgments
We thank Marcy E Rosenbaum and Toni Tripp-Reimer for their critical review and advice concerning the qualitative analysis, Susan Meadows, Dedra Diehl, Barcey Levy, and Robert Garrett for their help in verifying the final taxonomy, and the practising doctors who helped validate and refine the taxonomy.
Footnotes
Funding: This study was supported by grants from the American Academy of Family Physicians Foundation (G9518) and the National Library of Medicine (1R01LM07179-01).
Competing interests: JAO is an employee of Praxis Press, a company that produces evidence based clinical information resources for primary care doctors.