What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals?

Abstract

Background

Evidence on the journal-reading habits of clinicians comes from three separate groups of publications. First, several surveys have been used to ascertain the reading habits of physicians. Fafard and Snell [1] assessed house staff who reported reading an average of 8.7 hours per week, with about half of their time spent reading for specific patient situations. Reading time for family practice residents was more than three hours per week [2,3] and ranged from 1–12 hours. Dermatology residents averaged 4.2 hours reading per week and read an average of seven journals, four of which were peer reviewed [4]. Internists read an average of 4.4 hours per week [3], while surgeons reported an average reading time of 3.5 hours across 3–16 journals [5]. This average of three-four hours of reading time per week is quite consistent across disciplines, level of education, time, and nationality.

A second set of surveys and studies center on general information-seeking behaviors of clinicians. These studies show how journal reading fits in with the other types of information that clinicians use. Two systematic reviews have been done recently.

Researchers at the Australian National Institute of Clinical Studies [6] reviewed preferred information sources in many clinician groups, including physicians (primary care/general practice/family practice, hospitalists, rural physicians, diabetologists); nurses (hospital and occupational health nurses); physical therapists; dental hygienists; and policy-makers. They reviewed 34 studies and concluded that all groups used multiple information resources, with information needs answered most often by other people, followed by books and journals. Dawes and Sampson [7] evaluated 19 studies of physician information-seeking behavior. They placed books and journals in one category (print resources) and found this to be the most used information source, with colleagues being the second.

The third source of information on clinicians' use of information resources comes from marketing studies. The Association of Medical Publications [8] monitors physician use of printed journals and other information resources. Despite the rapid expansion of the Internet and all the information it contains, physicians continue to read and value journal articles, and their reliance on journals may be increasing. Data collected in 1983 and 1998 shows that physician reliance on journal literature as their main source of medical information increased from 61.8% to 76.3%, an absolute increase of 14.5% in 15 years.

The importance of reading journal articles for clinical care is evident. The increasing number of journals from which important and relevant articles are found, combined with the decreasing number of personal subscriptions [9], makes it more important than ever for physicians to choose carefully which journals to subscribe to and read. This decision should not be based on intuition alone, as shown in an important study by obstetricians and gynecologists. Weiner et al. [10] sought to determine which journals had published numerical data on the relation between oral contraceptive use and cancer–information they judged to be clinically important and readily available in their subspecialty journals. Assessing 3735 articles identified by MEDLINE searches, only 27 studies reported numerical data, of which 23 were published in mainstream general medical journals. Only four were published in obstetrics and gynecology journals.

Since the publication of the study by Weiner et al. [10], several groups have tried to determine targeted journal subsets that could provide the most important clinical information to physicians in different specialties. Birken and Parkin [11] assessed journals with pediatric content. Using data from pediatric-related systematic reviews in the Cochrane Database for Systematic Reviews for 1997, as well as policy statements from the American Academy of Pediatricians and the Canadian Paediatric Society, they determined that four general medical journals and three pediatric specialty journals provided access to most of the important advances: Archives of Diseases in Childhood, BMJ, JAMA, Journal of Pediatrics, Lancet, New England Journal of Medicine, and Pediatrics. Their results validate the findings of Weiner et al.–important studies in a discipline or specialty are often not published in specialty journals.

Gehanno and Thirion [12] used MEDLINE searches and the Science Citation Index (SCI) Impact Factors to identify journal subsets in occupational health. Eight journals provided coverage of 27% their discipline content; 38 journals increased this to 52%. Coverage needed to be expanded beyond their specialty journals for them to remain current in occupational health.

Lee et al. [13] sampled research articles from 30 randomly selected journals from a list of 107 general internal medicine journals defined by SCI. They found that journals with high citation rates, SCI Impact Factors, and circulation rates; low manuscript acceptance rates; and listing on the Brandon/Hill Library List [14] were predictive of higher article methodologic scores.

Ebell et al. [15] as well as our research group [16] present an alternative approach for clinicians to keep up to date with current literature. Both groups produce summaries of important advances in areas of clinical care so that individuals do not have to read primary journals and evaluate reports. Ebell et al. provided results of a hand search of 85 core journals of interest to family/general practice. Physicians read these journals for six months and identified articles that were considered to be POEMs (patient-oriented evidence that matters). A POEM addresses a clinical question encountered by a family physician at least once every two weeks, measures patient-oriented outcomes, and presents results that will likely affect practice. The report provides summaries of which journals publish important clinical advances for general/family practice.

In this article we report on a survey of the contents of 170 core clinical journals for the publishing year 2000 to assess which journals publish the highest number of methodologically sound and clinically relevant studies in the disciplines of internal medicine, general/family practice, general practice nursing, and mental health. In the "Methods" section we describe our two-step article selection process for clinical importance and methodologic rigor, which is very similar to that used by Ebell et al. [15]. The data we provide reflects the merit of individual journal titles from a clinical perspective; it may help clinicians to choose which journals to read, and health sciences libraries to include them in their collections.

Methods

The Health Information Research Unit of the Department of Clinical Epidemiology and Biostatistics at McMaster University, Ontario, Canada, publishes several secondary "evidence-based" journals, systematically selecting, summarizing and appraising articles in a broad range of primary clinical journals. In 2000 we prepared ACP Journal Club (ACP J Club) to support internal medicine , Evidence-Based Medicine (EBM) to support general/family practice, Evidence-Based Nursing (EBN) to support general care nursing, and Evidence-Based Mental Health (EBMH) to support mental health. To identify potential candidate articles for inclusion in these journals, six Masters-level trained staff read each article in the major general healthcare journals and those in the disciplines and subdisciplines related to the content of each abstract journal. The list of these journals (see Additional File 1) is comprised of titles suggested by librarians, clinicians, editors, and editorial staff; SCI Impact Factors; and systematic examination of the contents of each title for at least six months. More than 400 journal titles have been assessed since the abstract journals were started in 1991.

We consider the Cochrane Database of Systematic Reviews to be a separate journal that publishes systematic reviews of the literature on a quarterly basis. This is consistent with the U.S. National Library of Medicine's decision to index the Cochrane Database of Systematic Reviews as a separate journal. We evaluate only the new reviews and those that are substantially updated each quarter. We do not consider the rest of the database or protocols that describe reviews that are in progress or being planned.

Original and review articles are placed in one or more of seven categories of study type–therapy and prevention, screening and diagnosis, prognosis, etiology and harm, economics and cost, clinical prediction guides, and qualitative studies [16]. All categories have a set of pass/fail rules for selection (see: http://www.acpjc.org/shared/purpose_and_procedure.htm), except for qualitative and cost studies. Basic inclusion criteria are that the articles i) are about the healthcare of humans; ii) have at least one clinically important outcome; and iii) use appropriate statistical analyses. As an example of category-specific criteria, an article on screening or diagnosis must meet these additional criteria:

• a spectrum of participants were included, some with the disease or condition of interest and some without

• objective diagnoses were made using the "gold" standard or current clinical standard for diagnosis of the disease or condition

• participants received both the new test and some form of the diagnostic standard

• the diagnostic standard was interpreted without knowledge of the test result and vice versa.

The basic inclusion criteria are based on study design and methodology principles for evidence-based healthcare. Their use identifies studies that have data related to patients or those at risk of disease, diseases and conditions, and real-life clinical settings. Therefore, a study or review article that meets the criteria can be considered to be appropriate for possible use in patient care decision-making. The article readers are trained and retested annually so that they can reliably apply these selection rules for inclusion in our evidence-based journals (kappa measuring chance-adjusted agreement > 80% for all categories) [16].

With a research grant from the U.S. National Library of Medicine we intensified our data collection from the reading process related to the evidence-based journals for the publishing year 2000. All articles in 170 journals were classified as to whether they were "of interest" to the healthcare of humans and, if so, whether they reported original data or were systematic review articles. These original studies and reviews were classified into all possible categories (where more than one category could apply, for example, a therapy article that included economic data), and were then given a pass or fail methodologic designation for each category.

Articles passing methodologic criteria were assessed further for clinical interest by an editorial group of practicing clinicians for each abstract journal. These clinicians have expertise in methodology and specific areas of healthcare such as gastroenterology or neonatology nursing. At this point the clinician raters excluded all studies with preliminary results, interventions that were not readily available or proven useful, already known and applied findings, and topics addressing rare conditions or diseases. After review, often by a team of three-five clinicians (see http://hiru.mcmaster.ca/more/RatingFormSample.htm for a copy of the rating system that was used in paper format for this study) some articles were further processed. The editors chose articles to be abstracted that they considered to have the most important message for clinicians. The remaining pass articles were listed as "Other Articles Noted" if their content was of relevance to the disciplines covered by the abstract journals.

This dual selection process (methodologic rigor and clinical importance) provided insight into which journals yielded the highest numbers of pass articles. This had major implications for clinical practice at two levels of clinical relevance. The more stringent level includes articles that were summarized in each abstract journal. The second, less stringent level includes articles that are abstracted as well as those articles that are listed in the Other Articles Noted sections. Analysis was done by abstract journal title (ACP J Club, EBM, EBN, and EBMH) to ascertain which journal titles were most important to their target clinical audience (internal medicine, general/family practice, general practice nursing, and mental health, respectively). SCI Impact Factors were collected for each journal title for each discipline. If an SCI Impact Factor was not available we sought Social Science Index Impact Factors. These data were analyzed to determine if Impact Factors were related to yield of clinically important advances, as found by Lee et al. [13] and Gehanno and Thiron [12].

Results

For 2000, the 170 core journals we selected published 60,352 articles. The total number of pass articles was 3059 for original studies and 1073 for reviews. An article could be counted more than once if it passed for multiple categories. Six journals did not publish any pass articles. The complete list of journals and their yield appears in the Additional File 1.

The category breakdown of pass articles for original studies and review articles, respectively, was 1639 and 662 for therapy and prevention, 152 and 47 for screening and diagnosis, 195 and 22 for prognosis, 290 and 308 for etiology and harm, 35 and 10 for economics, 358 and 8 for qualitative studies, and 93 and 4 for clinical prediction guides.

The top 20 journals for yield of pass articles are included in Table ​Table1.1. The titles varied considerably in both the total number and proportion of clinically relevant articles that they published. For example, 95.0% of the articles (all reviews) in the Cochrane Database of Systematic Reviews passed our criteria, while only 2.8% of the articles in the American Journal of Gastroenterology met standards for clinically applicable studies. (The American Journal of Gastroenterology is a specialty journal and a substantial proportion of its content is preclinical. These preclinical articles, by definition, did not meet the clinical criteria in this study.) Generally, a clinical reader would need to read in the range of 13–14 articles from these top 20 journals to obtain one that is directly clinically important in any healthcare area, although the range is substantial (1.1 to 36.9). We call this number the "number of articles needed to be read" or NNR.

The number of pass articles did not correlate with SCI Impact Factors for the top 20 journals (correlation coefficient 0.29, P = 0.24). Analysis of the top 50 titles showed a weak correlation (correlation coefficient 0.41, P = 0.004) for the same analysis.

The breakdown by discipline was done using the total number of articles that were selected for inclusion in each of the four abstract journals–internal medicine, general/family practice, general care nursing, and mental health (Tables ​(Tables22,​,33,​,44,​,5).5). Both the total number of articles abstracted and the total number of articles in each journal (abstracted and "Other Articles Noted") are included in Tables ​Tables2,2, ​,4,4, and ​and55 giving a two-level assessment of "clinical worth". EBM does not publish an "Other Articles Noted" Section.

Conclusions

We found that the majority of articles for each discipline were sequestered in a small subset of journals. This is consistent with Bradford's Law of Scattering for journal subsets, which states that the important articles on any topic will be concentrated in a small subset of journals with exponential drop-off in numbers of relevant articles across journal titles [17]. Across disciplines and study areas, approximately 70% of articles are often found in 30% of journals in any given area of study.

Not surprisingly, for broad-based disciplines such as mental health and nursing, the number of titles was greater than for more focused disciplines such as internal medicine. SCI Impact Factors were highly correlated with the number of important clinical articles in separate titles for internal medicine and, to a lesser extent, for general/family practice, and mental health but not for general practice nursing. This likely reflects the volume of clinically important research activity in these fields–with especially high volumes in the disorders managed by internal medicine and its subspecialties–coupled with the avidity of authors from all disciplines to submit their best studies to the high-circulation general journals.

As found by Weiner et al. [10] and others, most of the important advances in any discipline are not published in specialty journals but in the more general healthcare journals such as JAMA, Lancet, BMJ, New England Journal of Medicine, and Cochrane Database of Systematic Reviews. Health professionals in all disciplines should be aware that major advances in any field will most likely be published in the main general medicine journals, while at the same time recognizing that specialty journals also publish important information. Much variation exists across journal titles in both the number and proportion of articles that are high quality, clinically important, and newsworthy. Variation also exists across disciplines. It is also interesting to note that all lists of important journals discussed in this report and also the one by Ebell et al. [15] include both North American and European titles. Reading choices for clinicians cannot be based on national or discipline boundaries alone.

Of the 45 titles that provided articles to EBM, 23 were on the list provided by Ebell et al. (POEM articles) [15]. Ebell et al. found common POEMs in 49 journals and any POEMs in 64 journals. POEMs and EBM cover the content of general/family practice by considering a similar number of journals, although both groups read approximately 50% unique titles. Ebell et al. read 85 titles for POEMs articles and we read 170 titles for this study. Our coverage of clinical content was broader and included internal medicine, general practice nursing, and mental health, but 53 titles were read by both groups. Correlational analysis for the ranking of each journal title according to the number of articles identified as clinically important showed a small but significant agreement (0.4397, P = 0.005) when comparing our list with the list by Ebell et al.

Consistent with the data from Weiner et al. [10], many advances important to general practice nursing are not published in nursing specialty or discipline-specific journals. Only four of the top 17 and eight of the top 41 journals in Table ​Table44 are considered nursing specialty titles. Overall 39 titles provided at least one article for abstraction and an additional 33 titles provided at least one article to the "Other Articles Noted" section, again showing the broader spectrum of journals that publish articles important to general care nursing.

Clinicians in the target disciplines described here could use our findings to focus their fulltext readings. For other disciplines, a similar audit of clinical yield would be needed, either from an appropriate secondary journal that systematically reviews specified journals, or an independent audit. Another approach to staying current may be to subscribe to one or more secondary journals that highlight important clinical advances. These secondary publications have not only selected the most appropriate studies for clinical consideration, they highlight important aspects of methodology and implementation. This assessment of studies before application can be time-consuming and difficult for many clinicians, and involves a certain amount of training and practice to become proficient. Many examples of secondary publications exist in various disciplines and include the four studied in this report, POEMs [15], and Journal Watch. Use of these summaries of studies and reviews can be supplemented by access to fulltext articles.

Many academic medical centers and hospitals provide good online access to major healthcare journals. For example, the Health Sciences Library of the University of Pittsburgh, PA, USA, provides online access to 24 of the top 25 journals in this study and all 25 of the journals identified as high yielders by Ebell et al. [15]. Specialized health libraries with limited budgets may wish to focus on the journals, either in paper or electronic format, with the highest yield for the disciplines they serve.

List of abbreviations

ACP J Club ACP Journal Club (journal)

AHCPR Agency for Health Care Policy and Research (now AHRQ)

AHRQ Agency for Healthcare Research and Quality (formerly AHCPR)

CCOHTA Canada Coordinating Office for Health Technology Assessment

EBM Evidence-Based Medicine (journal)

EBMH Evidence-Based Mental Health (journal)

EBN Evidence-Based Nursing (journal)

NNR Number of articles needed to read to obtain one high-quality and clinically relevant study or review

POEM Patient-oriented evidence that matters

SCI Science Citation Index

Competing interests

The authors all worked with ACP Journal Club, Evidence-Based Medicine, Evidence-Based Nursing, and Evidence-Based Mental Health at the time of this study, and were paid for this work, but the publishers of these journals were not involved in the study, which was funded externally. The authors do not hold stocks or shares in any company that may benefit from the publication of this paper.

Author contributions

NLW and RBH prepared grant submissions in relation to this project. All authors drafted and commented on the manuscript and approved the final manuscript, as well as supplied intellectual content to the collection and analysis of the data. NLW and KAM did data collection and analysis, and supervised research staff.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1741-7015/2/33/prepub

Supplementary Material

Acknowledgements

References