Impact of the Transparent Reporting of Evaluations With Nonrandomized Designs Reporting Guideline: Ten Years On

TREND Use by Authors of Research Articles

We identified, retrieved, and coded studies that cited the original TREND statement to examine how authors used TREND. We used both the Web of Science and Scopus databases “cited by” search functions to identify articles that cited TREND. We searched both databases because it was previously established that individual databases do not include all citations.^25,26

Adding Google Scholar to the repertoire of search engines could have yielded additional results, but Google Scholar was inefficient and less specific than the Web of Science and Scopus,²⁷ and it is likely that many irrelevant records would have had to be searched for little, if any, gain. We identified studies that used TREND to inform reporting within this sample through a manual search, and we extracted relevant details (e.g., study design).

TREND Use in Journals’ Instructions to Authors

We searched for references to the TREND guidelines in the instructions to authors information in journals that published articles that used TREND to inform reporting or were listed as TREND supporters on the CDC Web site (March 2013). Key words and phrases were copied verbatim when references were made to reporting guidelines (e.g., “authors should use TREND when reporting studies with nonrandomized designs”). In the event that 2 or more journals referred to the same instructions to authors, they were treated independently.²⁸ We also recorded if a journal’s instructions to authors had a requirement for completed checklists to be submitted with manuscripts or if it had links to the Enhancing the Quality and Transparency of Health Research (EQUATOR) and TREND websites.

We used a 1-way analysis of variance to check whether there were differences in journal status (as indicated by a proxy: impact factor) among journals that published articles that used TREND and those listed as supporters of the reporting guideline. (Note: impact factors are the ratio of the overall frequency with which articles are cited to the number of research articles, technical notes, and reviews [i.e., “citable” articles as defined by Institute for Scientific Information] within a journal from the preceding 2-year period.²⁹ Impact factors are not as objective as they might be seen as and used, because the method of calculating them is susceptible to biases [e.g., coverage and language preference of the Science Citation Index]²⁹; furthermore they are not an indication of study or article quality. Despite these issues, impact factors are known to influence authors’ and editors’ practices regarding manuscript submission and publication.)

TREND and the Reporting Completeness of Articles

We assessed the impact of TREND by comparing 2 samples of study reports—1 that used TREND and 1 that could have used TREND, but did not report doing so. We selected comparator studies by first identifying a pool of studies that could have used TREND, but did not report doing so, by using a focused search strategy (included as a supplement to this article at http://www.ajph.org) within MEDLINE, PsycINFO, and EMBASE. We then screened the search results (with 25% being randomly selected and double-screened by either J. P. or M. P.). Finally, we randomly selected an equal number of comparator articles (not citing TREND) to those that reported the use of TREND.²⁴ We calculated reporting completeness as the percentage of items identified in a published article that were listed in the TREND checklist or another quality appraisal tool.

There are multiple components (e.g., “clearly defined primary and secondary outcome measures”) within some items on the TREND checklist, and no instructions are given on how authors should comply with these components, that is, whether they need to report both the primary and secondary outcomes before they are judged as having have reported the item. Because an evaluation of TREND has not been undertaken previously, and the use of it is usually recommended rather than mandatory, we adopted a generous criterion when assessing reporting completeness. This meant that if 1 part of the item was reported (e.g., primary, but not secondary outcome, measures were reported), we coded it as reported.

We assessed the respective samples of articles for reporting completeness using the TREND checklist (the primary outcome measure), the Effective Public Health Practice Project-Quality Assessment tool (EPHPP-QA),³⁰ and the revised Graphical Appraisal Tool for Epidemiological studies (GATE).³¹ Reviewers were not blinded to which article sample a study was in (i.e., use or nonuse of TREND) because of limited resources. Also, article details, such as journal title and year of publication, were not concealed from reviewers because there was no strong evidence to indicate that this affected assessment of reporting completeness and study quality.²¹

T. F. assessed all the articles included in the study. A second reviewer assessed a random selection of 25% of both samples, and the interrater reliability between reviewers was checked. Results revealed good interrater reliability for reporting completeness with the TREND checklist (Cronbach α = 0.86), moderate reliability for the EPHPP-QA (Cronbach’ α = 0.49), and poor reliability for GATE (Cronbach α = 0.34). Because of the low level of interrater reliability with GATE, it was excluded from subsequent analyses.

To examine whether reporting completeness (as measured by the TREND checklist) was related to use of TREND, we conducted an analysis of covariance test by controlling for the year of publication and the impact factor. Because there were 2 outliers within the impact factor, the main analyses were run twice, once with and once without them, to gauge their effect. We used SPSS version 21 (IBM, Armonk, NY)³² to conduct all analyses.

We also conducted analyses to explore whether the stated use of TREND was associated with better reporting of particular aspects of studies, such as the theory used to inform interventions, discussion of fidelity of intervention, steps taken to minimize bias, blinding, and discussion of generalizability.

TREND use in Journals’ Instructions to Authors

TREND and the Reporting Completeness of Articles

Underutilization of TREND by Authors

We found evidence indicating that authors of primary research studies used TREND infrequently. This was both as a percentage of the number of times it was cited and compared with other reporting guidelines (e.g., CONSORT).³⁴ However, when authors of primary research articles used it, they used it as intended by TREND’s developers. This was in contrast with secondary researchers who used TREND as a study quality assessment tool, although it was not designed for this purpose.

Only 1 other study, to our knowledge, specifically examined how authors used reporting guidelines. In that instance, Da Costa et al. found that the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement was frequently misused as a quality assessment tool in systematic reviews.³⁵ Together with our results, this suggests that there is a need for developing specific study quality or risk of bias tools for nonrandomized study designs. Higgins et al. recently reached a similar conclusion in their article, which was based on a workshop organized by the Non-Randomized Studies Methods Group of the Cochrane Collaboration.³⁶

Variation and Ambiguity in the Promotion of TREND

Keeping in mind that instructions to authors are documents that change over time,³⁷ what we found reflected a “snapshot” of a sample of journals taken in 2013. Journal editors in journals’ instructions to authors used TREND relatively infrequently. Other studies that examined the frequency with which specific reporting guidelines were cited in instructions to authors found that, for example, CONSORT was referred to between 3%³⁸ and 53%³⁹ of the time.^28,40,41 Although we found that TREND was referred to in approximately 25% of the journals’ instructions to authors, this percentage was probably artificially inflated because the journals included claimed that they were TREND supporters or were journals that had published articles that used TREND.

Editorial endorsement of the use of a reporting guideline varies and ranges between weak (i.e., where a preference or suggestion for the use of the guideline is made) to strong (i.e., where the submission of a completed checklist with a manuscript is required and enforced by editorial staff). Although “should” is an indication of a strong preference for the use of reporting guidelines, it is still just that. The strength of editorial endorsement likely affects authors’ likelihood of using reporting guidelines,⁴² but further empirical research is required. Our finding that there was a large degree of heterogeneity and ambiguity in the choice of words used by editors in instructions to authors when referring to the requirement to use reporting guidelines was consistent with other results.^38,40,41,43 The ambiguity and relative weakness in instructions to authors might partially explain why the use of TREND was so low.

Taken together, the first 2 key findings suggested that there were failures in the dissemination and implementation strategy of the TREND guideline. Data from our study did not enable us to do anything more than speculate on the factors that affected the uptake and use of TREND. We recently conducted a second study using survey and interviews with editors and authors to explore this issue in depth (unpublished article).

Evidence That TREND Improves Reporting Completeness

Using 2 different measures of reporting completeness, we found that studies in which TREND was used to inform reporting were more likely to have greater reporting completeness than studies that did not use TREND. However, even when TREND was used to report studies, reporting completeness scores were only at 75%, indicating that there is considerable room for improvement and that use of TREND itself does not wholly achieve its intended purpose. In addition, the result informed us that differences exist in perceptions of what “reporting” constitutes between reviewers assessing reporting completeness and authors using reporting guidelines.

Making comparisons with other studies was problematic because of a lack of evaluations of reporting guidelines,^42,44 different methods used in evaluating impact, and sometimes an absence of detail regarding reporting completeness data.⁴⁵ Nevertheless, from the data that were available, our results showed reporting completeness to be within the range of that reported by other studies.^46,47 However, our estimates of reporting completeness were likely to be inflated because of the generous, rather than conservative, coding scheme we adopted when coding whether a particular item was reported.

Use of TREND Is Associated With Better Study Quality

The finding that use of TREND, when using EPHPP-QA, was associated with higher study quality (distinct from the impact factor) was unlikely to be solely because of better reported studies having more information. This was because the scoring of the EPHPP-QA does not always calculate nonreporting as a weakness of the study design. Thus, although the association between use of TREND and higher ratings of study quality appeared encouraging, it was not a causal link. Replication is required before stronger claims can be made.

Study Limitations

Three main limitations affected the strength of findings in our study. First, the main outcome measure of reporting completeness—the TREND checklist—was also the intervention of interest. To address this, we used a second measure of reporting completeness, the EPHPP-QA. It was chosen in the absence of other purposefully designed measures of reporting completeness and because of its relevance to nonrandomized study designs. Although it showed a similar direction in results, the differences were not statistically significant. It thus provided evidence that contrasts with that obtained when using the TREND checklist as the measure of reporting completeness. A second limitation of our study was that assessments of reporting completeness and study quality were not blinded. Despite having a second reviewer assess a random sample of 25% of articles, we could not completely discount the possibility that reviewer biases accounted for any differences in reporting completeness and study quality rating. Lastly, it was possible that the pool of studies randomly selected to be comparators were not representative of the population of articles that could have used the TREND guideline, but did not. A different sample could have yielded different reporting completeness results.

Conclusions and Implications

Without an ongoing dissemination and implementation strategy, TREND’s uptake and use has remained relatively low. Lack of widespread adoption of better reporting practices in public health risks potentially valid and relevant evidence being excluded from systematic reviews.

Replication of this study is required and could be done using purpose-made measures that assess reporting completeness or with blinded assessors. Similarly, further investigation is warranted regarding the association between study quality and use of TREND, possibly through examination of whether studies that use TREND are more likely to be included in systematic reviews.

If TREND is to have a role in the future, we expect that potential users would need to be convinced of the benefits of its use, and recommend that a new promotion and dissemination strategy be developed. Uptake of TREND could be increased if more journals required and enforced the use of TREND to report studies with nonrandomized designs. In addition, an updated TREND guideline could increase its perceived relevance. For example, simplifying certain items could improve ease of use and provide the focus for renewed promotion to opinion leaders and professional societies. As with other reporting guidelines, uptake could also be increased by improving the accessibility and usability of the TREND checklist (e.g., see the CONSORT checklist⁴⁸). We also suggest that there is potential for combining efforts with other initiatives to improve the quality of description of public health interventions. For example, the behavior change taxonomy of Michie et al. could form part of the foundation of a common language to be used within the framework provided by TREND.⁴⁹ Combining such initiatives with the continued support of the American Journal of Public Health for TREND and other reporting guidelines would have benefits for researchers and the community alike.