Evidence of bias in estimates of influenza vaccine effectiveness in seniors

Abstract

Background Numerous observational studies have reported that seniors who receive influenza vaccine are at substantially lower risk of death and hospitalization during the influenza season than unvaccinated seniors. These estimates could be influenced by differences in underlying health status between the vaccinated and unvaccinated groups. Since a protective effect of vaccination should be specific to influenza season, evaluation of non-influenza periods could indicate the possible contribution of bias to the estimates observed during influenza season.

Methods We evaluated a cohort of 72 527 persons 65 years of age and older followed during an 8 year period and assessed the risk of death from any cause, or hospitalization for pneumonia or influenza, in relation to influenza vaccination, in periods before, during, and after influenza seasons. Secondary models adjusted for covariates defined primarily by diagnosis codes assigned to medical encounters.

Results The relative risk of death for vaccinated persons compared with unvaccinated persons was 0.39 [95% confidence interval (95% CI), 0.33–0.47] before influenza season, 0.56 (0.52–0.61) during influenza season, and 0.74 (0.67–0.80) after influenza season. The relative risk of pneumonia hospitalization was 0.72 (0.59–0.89) before, 0.82 (0.75–0.89) during, and 0.95 (0.85–1.07) after influenza season. Adjustment for diagnosis code variables resulted in estimates that were further from the null, in all time periods.

Conclusions The reductions in risk before influenza season indicate preferential receipt of vaccine by relatively healthy seniors. Adjustment for diagnosis code variables did not control for this bias. In this study, the magnitude of the bias demonstrated by the associations before the influenza season was sufficient to account entirely for the associations observed during influenza season.

Numerous observational studies have reported that seniors who receive influenza vaccine are at substantially lower risk of death and hospitalization during influenza season than unvaccinated seniors.^1–24 The main issue in interpreting those findings is whether preferential receipt of vaccine by relatively healthy seniors could account for some or all of the observed reduction in the risk of health outcomes. Since influenza is a seasonal infection, a true protective effect of vaccination should be limited to periods of influenza viral circulation. Assessment of the vaccine association during influenza and non-influenza periods could therefore help to distinguish a true vaccine effect from an effect of bias due to differences in the underlying characteristics of the vaccinated and unvaccinated groups.

Several studies have assessed the seasonal specificity of estimates of influenza vaccine effectiveness by comparing the vaccine association during influenza season with that during a later time period, and some of those studies have reported a reduction in the risk of the non-specific outcomes of death or hospitalization in vaccinated persons compared with unvaccinated persons during influenza season but not during the later comparison period. Those findings have been interpreted as evidence for a true vaccine effect during influenza season. This interpretation assumes that any influence of bias due to differences in the underlying characteristics of the vaccinated and unvaccinated groups is constant over time.

We hypothesized that the magnitude of the underlying differences that predispose to death and hospitalization between vaccinated and unvaccinated groups may actually diminish over time. Our rationale was that if seriously ill seniors are less likely to receive the influenza vaccine there would be a higher short-term mortality risk in the unvaccinated group compared with the vaccinated group. As a consequence of the disproportionately greater loss of seriously ill persons from the unvaccinated group, over time, the two groups would become more similar. Other changes in health status, for better or worse, among members of both groups would also tend to lead to equilibration of underlying differences that predispose to death and hospitalization over time.

These effects could lead to the finding of a greater reduction in the relative risk of death or hospitalization in vaccinated persons compared with unvaccinated persons during influenza season than in a later comparison period, even in the absence of any true protective effect of vaccination against influenza infection. For this reason, evaluations of the seasonal specificity of the association of influenza vaccination and risk of death and hospitalization should include a pre-influenza season comparison period, when there is almost certainly no true vaccine effect, and when, unlike the more traditional post-influenza comparison period, the magnitude of the effect of bias due to differences in underlying characteristics between the vaccinated and unvaccinated groups is expected to be at least as strong as that present during influenza season.

To better assess the possible influence of bias due to confounding by health status on estimates of influenza vaccine effectiveness, we followed a large population-based cohort of seniors from September 1995 through August 2003. We estimated the relative risks of death, hospitalization for pneumonia or influenza, and other hospitalization outcomes in vaccinated versus unvaccinated persons, in periods before, during, and after influenza season. We also replicated methods of adjustment for covariates defined by diagnosis codes and indicators of medical utilization reported by previous influenza vaccine effectiveness studies to assess their ability to remove bias due to differences in health status.

Methods

Study population and setting

The study cohort included members of Group Health Cooperative, a health maintenance organization (HMO) in Washington State with an enrollment of ∼350 000 members. Group Health administrative data systems recorded information on enrollment, nursing home residence, immunizations, and International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) diagnosis codes assigned to inpatient and outpatient medical encounters. The cohort for the first study year of September 1995 through August 1996 consisted of persons who, as of September 1, 1995, were ≥65 years of age, had been enrolled in Group Health for at least 1 year, and were not residents of a nursing home. Additional Group Health members newly meeting the eligibility criteria as of September 1 of each subsequent year through 2002 entered the study cohort on that date. Study cohort members were followed from their date of study entry until death, disenrollment from Group Health, nursing home admission, or the study end date of August 31, 2003, whichever was the earliest.

Outcomes

The primary outcomes were all cause mortality and hospitalization with a discharge diagnosis of pneumonia or influenza (PI hospitalization), defined by ICD-9-CM codes 480 through 487. Secondary outcomes included hospitalization with a discharge diagnosis of cerebrovascular disease (ICD-9-CM codes 431–437), ischaemic heart disease (410–414), congestive heart failure (428), and injury or trauma (800–904 and 910–959).

Disease covariates

To assess the effect of adjustment for health status covariates defined according to methods used in previous influenza vaccine effectiveness studies in HMO populations^11,14 on the associations of influenza vaccination with risk of death and hospitalization, we defined the covariates of heart disease, lung disease, diabetes mellitus, renal disease, cancer, vasculitis and rheumatologic disease, dementia, hypertension, atrial fibrillation, lipid disorders, hospitalization for pneumonia in the prior year, and 12 or more outpatient visits in the prior year by those methods. Covariate values were updated on September 1 of each study year and were based on information recorded in Group Health administrative data systems in the 12 months prior to that date.

Time periods

We categorized each September through August study year into periods before, during, and after influenza season based on estimated dates of the start and end of the influenza season. We used national influenza viral surveillance data^25–29 to define the onset and end of each influenza season as the first and last weeks with at least 50 influenza isolates reported. For comparison, we also assessed local influenza viral surveillance data reported by Public Health-Seattle and King County, and defined the onset and end of influenza season by the first and last occurrences of at least two consecutive weeks with two or more influenza isolates reported (Table 1). Since the results of analyses based on either local or national surveillance data were very similar, unless otherwise noted, the results presented are based on influenza periods defined by published national data.

To further differentiate risk over time, we also defined intervals within the before, during, and after influenza periods. We defined a ‘high vaccination’ before influenza period as the interval from the date by which 50% of cohort vaccinations had been administered (Table 1) to the onset of influenza season. Within the influenza season, we defined early (the onset of influenza season to the peak influenza period), peak (the 5 weeks that spanned the 2 weeks before and after the week of peak viral circulation), and late (the week after the peak period through the end of influenza season) influenza periods. We also divided the after influenza period into a post-influenza period (end of influenza season through May 31) and summer (June 1 through August 31).

Statistical analysis

We used Cox proportional hazards regression to estimate the relative risk of the primary and secondary outcomes for vaccinated cohort members compared with unvaccinated cohort members during time periods defined by influenza surveillance.³⁰ In analyses of each of the hospitalization outcomes, we allowed for recurrent events by using a multiple failure time-proportional hazards model based on counting processes.³¹

To account for changing vaccination status during the period of vaccine availability, we incorporated a time-varying vaccination status variable into the Cox models.^32,33 At the September 1 start of each study year, all cohort members were defined as unvaccinated. Persons who were vaccinated during the study year then changed to vaccinated status on the day following vaccination and retained that status through the August 31 end of the study year. The estimated relative risk was therefore based on the number of events, the number at risk, and the vaccination status of each study participant on the exact day at which each event occurred during the study period, and thus accounted for differences in cohort vaccination coverage over time. In addition, models included an interaction term between vaccination status and each time period to allow the association between influenza vaccination and risk of the outcome to vary in periods before, during, and after influenza season.

The primary models were adjusted for sex and age (5 year age-groups through age ≥85 years). For comparison, secondary models were also adjusted for disease covariates. We fit models that combined data across all study years, and we also examined single year models to assess variability by study year. In the analyses across all study years, age and disease covariates were time-varying variables updated annually on the September 1 start date of each study year. Analyses were conducted using Proc PHREG from SAS Version 8.2 (Cary, NC), using the Breslow method for tied follow-up times.

Results

The study cohort included 72 527 seniors, who contributed a total of 338 264 person-years of observation during the 8 year study period. During each year, ∼44 000 seniors were evaluated, and influenza vaccine coverage ranged from 68 to 74% during the study years (Table 1). Across the study period, persons who had been assigned diagnosis codes indicative of chronic conditions, with the exception of dementia, contributed a greater proportion of vaccinated than unvaccinated person-time (Table 2).

In analyses across all years, the relative risks of the primary outcomes of death and PI hospitalization were lowest in the period before influenza season (Figure 1), and increased progressively in the influenza and post-influenza periods. These results, which were based on time intervals defined by national influenza surveillance data, did not differ substantively from the results of analyses based on time intervals defined by local influenza surveillance data. For example, in the analyses based on intervals defined by local surveillance, the relative risk of death was 0.41 before influenza season, 0.54 during influenza season, and 0.74 after influenza season, compared with estimates of 0.39, 0.56, and 0.74, respectively, for analyses based on national surveillance.

In analyses of the secondary hospitalization outcomes, a similar temporal trend was found, with the lowest point estimates of the relative risk in the before influenza period (Table 3). Of the secondary hospitalization outcomes, the lowest estimates of the relative risk for vaccinated persons compared with unvaccinated persons were reported for the outcome of injury or trauma hospitalization, which was a control outcome selected because it should be unrelated to influenza infection.

Adjustment for the disease covariates defined by diagnosis codes consistently resulted in lower estimates of the relative risk compared with the estimates derived from the primary age- and sex-adjusted models, across all outcomes and in all time periods (Table 3).

Estimates of the relative risk of death and PI hospitalization in the before influenza period were robust to alteration of the date of onset of that period. In analyses of the interval defined as starting on the date by which 50% of cohort vaccinations had been distributed and ending at the onset of influenza season, the relative risk of death was 0.38 and of PI hospitalization was 0.70 (Table 4).

Within the influenza season, the point estimate of the relative risk of death for vaccinated persons compared with unvaccinated persons was lowest in the early influenza period (0.46) and then increased progressively through the peak (0.50) and late (0.69) influenza periods. For the outcome of PI hospitalization, the relative risk estimates varied somewhat between the early (0.82), peak (0.74), and late (0.89) influenza periods, but the confidence intervals overlapped, and all of the within-influenza season point estimates were higher than the before influenza season point estimate of 0.72.

The results of the analyses of individual study years, though more variable, were consistent with the results of the analyses across the entire study period (Table 5). In every study year, except 2002, the relative risk of all cause mortality was lowest in the period before influenza season and increased progressively in the periods during and after influenza season. Due to the lower number of events, estimates of the relative risk of PI hospitalization and injury or trauma hospitalization were more variable, and in general the confidence intervals of the before influenza and during influenza estimates overlapped substantially.

Discussion

In this study, the reductions in risk observed in the before influenza period suggest the presence of bias due to preferential receipt of vaccine by relatively healthy seniors on the estimates of influenza vaccine effectiveness observed during influenza season. Among this large cohort of Group Health seniors, we found reductions in risk of all cause mortality and of PI hospitalization in vaccinated persons compared with unvaccinated persons during influenza season that were consistent with estimates reported by previous observational studies. For example, our age- and sex-adjusted estimate of the relative risk of death during influenza season of 0.56 is identical to the corresponding estimate reported for the 1999/2000 influenza season in a large cohort study of seniors in three HMO populations,¹¹ and our estimate of the relative risk of PI hospitalization of 0.82 is very similar to the corresponding estimate of 0.80 reported in that study.

In contrast to previous cohort studies, we also evaluated the period before influenza season and found the greatest reductions in risk of death and PI hospitalization during that period. Those estimates were robust to variation in the definition of the onset of the pre-influenza period, and the survival analysis methods we used accounted for changes in the vaccination status of individuals during each study year. The reductions in risk observed before influenza season almost certainly could not be due to a true vaccine effect, and most likely were due to underlying differences between the vaccinated and unvaccinated groups. The magnitude of the bias demonstrated by the associations of vaccination with risk of all cause mortality and PI hospitalization before influenza season was sufficient to account entirely for the associations observed during influenza season. The movement of the measures of the vaccine association towards the null in later time periods is compatible with a reduction in the differences of the outcome risk between the vaccinated and unvaccinated groups over time.

Nearly all of the previous observational studies that addressed the seasonal specificity of estimates of influenza vaccine effectiveness did so by comparing differences in risk between vaccinated and unvaccinated persons during influenza season with differences in risk during post-influenza periods or during peri-influenza periods that included post-influenza intervals.^{1,6,9–11,24,34} The results of our study suggest that the observation of a greater reduction in risk during influenza season compared with a later period could be due to a decrease in the magnitude of the differences between vaccinated and unvaccinated persons over time, and so is not necessarily evidence of a true vaccine effect during influenza season. Therefore, evaluation of a before influenza period is needed in order to appropriately interpret the relative risk estimates observed in the influenza and post-influenza periods.

Only one prior study reported comparison of an influenza period with a pre-influenza period. That case-control study, of influenza vaccination and risk of hospitalization for pneumonia in the elderly, reported a reduction in the risk of pneumonia during the influenza season [odds ratio (OR), 0.69; 95% CI, 0.49–0.96] but not before influenza season (OR, 0.98; 95% CI, 0.69–1.39).⁷ While these findings are consistent with a seasonal specificity of the vaccine effect for the outcome of hospitalized pneumonia, the study was subject to some limitations. Case patients were recruited retrospectively, after the end of influenza season, and so the survival and health status of participants at the time of recruitment could have influenced the likelihood that they would be included in the study population. Vaccination status was also assessed after the end of influenza season, and was defined by proxy report for 50% of cases, and so may have been misclassified. While there is no evidence that the magnitude of the selection bias or information bias potentially present in this study differed for subjects with index dates before or during influenza season, the possibility of differential bias argues for a cautious interpretation of the time period comparisons.

In our study, we replicated methods of adjustment for covariates defined by indicators of medical utilization and by groupings of diagnosis codes that have been used in previous studies of HMO populations. We expected that, in analyses of the before influenza period, when any difference in risk between vaccinated and unvaccinated persons is presumably due to bias, proper adjustment would produce relative risk estimates close to the null value of 1.0. Instead, we found that adjustment for the covariates we defined led to relative risk estimates for death and PI hospitalization in the before influenza period that were, if anything, further from the null than the unadjusted estimates.

This same effect of adjustment that we observed, leading to lower estimates of the relative risk and therefore greater estimates of vaccine effectiveness, has been consistently documented in other vaccine effectiveness studies that adjusted for health status covariates defined by similar methods.^{1,11–14,21,22} For example, in the cohort study of three HMO populations, adjustment moved the estimate of the relative risk of all cause mortality in the 1999/2000 influenza season from 0.56 (age- and sex-adjusted) to 0.50 (age-, sex- and covariate-adjusted), and moved the estimate of the relative risk of PI hospitalization from 0.80 to 0.71.¹¹ Failure of this method to adjust for bias may be the result of the fact that diagnosis codes assigned at medical encounters are not measures of frailty or disease severity, which are likely influential factors in the association of influenza vaccination and the risk of serious health outcomes.

It is important to note that, like other observational studies, we did not evaluate outcomes specifically due to influenza infection, because influenza infections are rarely documented by laboratory testing. The limitation of this approach is that prevention of influenza-related complications may have relatively little impact on the broader, non-specific study outcomes. This problem can be illustrated by considering the possible effect of influenza vaccination on the risk of all cause mortality. Assuming, for example, that influenza vaccine reduces the risk of fatal influenza infection by 58%, which is the level of efficacy against serologically confirmed influenza infection reported by a randomized trial of older adults,³⁵ and that influenza infection accounts for 10% of all deaths during influenza season,³⁶ then influenza vaccination would be expected to reduce all cause mortality during influenza season by 5.8%. The corresponding estimate of the relative risk of all cause mortality for vaccinated persons compared with unvaccinated persons, in the absence of bias, would be ∼0.94.

For this reason, our finding that differences in health status between vaccinated and unvaccinated groups leads to bias in estimates of influenza vaccine effectiveness against all cause mortality and other non-specific outcomes does not mean that there is no effect of vaccination against serious complications of influenza infection. Our results do suggest, however, that other methods for evaluations of influenza vaccine effectiveness should be explored. These methods could include prospective ascertainment of influenza-specific outcomes, to improve study sensitivity to detect a true vaccine effect, as well as more accurate characterization of disease severity and functional status, to allow better adjustment for confounding. In future studies, assessment of the effect of adjustment in the before influenza period may assist in evaluating the degree to which influential differences between vaccinated and unvaccinated persons are controlled for in analyses of events during influenza season.

References