Learn more: PMC Disclaimer | PMC Copyright Notice
Modification of a vaccine hesitancy scale for use in adult vaccinations in the United States and China
ABSTRACT
Most research on vaccine hesitancy has focused on parental attitudes toward childhood vaccination, but it will be important to understand dimensions of vaccine hesitancy in the adult population as more adult vaccines are introduced in the future. We modified the Vaccine Hesitancy Scale to target adult vaccines and provide measures of its reliability and validity relative to influenza vaccine uptake and COVID-19 vaccination acceptance in cross-sectional internet surveys in the United States and in China. We assessed the impact of vaccine hesitancy on influenza and COVID-19 vaccination using multivariable regression modeling, which informed concurrent validity of the adult Vaccine Hesitancy Scale (aVHS). Among 1103 participants in the March 2020 China survey, 5.4% would not accept a COVID-19 vaccine, whereas this figure was 18.8% for the March 2020 US survey and 27.3% for the June 2020 US survey. The aVHS exhibits good internal consistency in all three surveys. Models adjusted for age, gender and income level show that prevalence of COVID-19 vaccine acceptance was a fraction as high in those who scored higher on the VHS than those who scored lower on all three surveys. Prevalence of past and future flu vaccine acceptance was a fraction as high in those with higher aVHS scores than those with lower scores. Prevalence of COVID-19 vaccine acceptance is lower in those with higher vaccine hesitancy scores, which supports the scale’s concurrent validity. The aVHS exhibits good internal consistency, making it a valid and reliable tool for measuring vaccination uptake.
Introduction
Diminishing community spread of COVID-19 with a vaccine is crucial both for public and economic health. However, accomplishing this depends on not just vaccine safety and efficacy, but also the strength of public trust in the technology. Vaccine hesitancy is no new phenomenon, stretching as far back as the 19th century when Edward Jenner and his colleagues faced suspicion and outright refusal toward the smallpox vaccine in England. The political and social dialogue surrounding the vaccine in this period culminated into organized groups, such as the National Anti-Vaccination League, which lobbied against mandatory smallpox vaccination policies in Britain in mid-to-late 19th century.1 Moreover, the League, though spearheaded by scholars, socialites, and scientists, garnered additional support from working-class families, who did not want to face fines or imprisonment for refusing to vaccinate their children.2
Despite its immense historical presence, vaccine hesitancy has become an increasingly pressing issue in the 21st century, threatening existent herd immunity to once highly prevalent illnesses and halting progress toward ongoing disease prevention measures. According to the World Health Organization’s Strategic Advisory Group of Experts (WHO SAGE), vaccine hesitancy is now defined as “the delay in acceptance or refusal of vaccination despite availability of vaccine services. Vaccine hesitancy is complex and context-specific, varying across time, place and vaccines. It is influenced by factors such as complacency, convenience, and confidence.”3 It prevails in the time since the coronavirus pandemic’s start for a number of reasons. Several leading vaccine producers project that there will be a safe and effective contender available as early as the end of 2020, making it one of the swiftest-made vaccines for an emergent illness to date. For many members of the public, the COVID-19 vaccine’s truncated manufacture timeline leaves room for doubt. A study comparing opinions between Israeli healthcare workers and members of the public found that even medical personnel were hesitant, because the vaccine is being manufactured so quickly, indicating the need for educational intervention earlier on in the timeline so as to minimize confusion and doubt.4 Another major source of hesitancy stems from rapid spread of conspiratorial or manipulated information regarding the vaccine. In a scoping review of English language coronavirus reports published between January and March 2020, one group found that the majority of the misinformation circulating was not completely fabricated, but rather altered stories meant to mislead or manipulate consumers.5 This creates an environment that leaves the public wary and skeptical of coronavirus information that is truthful or attempts to debunk the myths and conspiracies that circulate regarding the virus and its potential therapeutics.
There are a handful of different scales and indices available to measure hesitancy in different targets, such as parents, healthcare workers, or caregivers. One commonly used scale is the Parent Attitudes about Childhood Vaccines (PACV) survey, which is primarily used to measure childhood vaccine hesitancy in parents or guardians.6 The WHO SAGE Working Group on Vaccine Hesitancy has a ten-item Vaccine Hesitancy Scale (VHS) that is widely used in different countries and settings.7 Another, the Vaccination Confidence Scale, was developed and tested using parental attitudes toward adolescent vaccines.8 Lastly, Betsch et al.’s 5 C scale expands upon the constructs named within the WHO SAGE definition, to further specify vaccine hesitancy, through assessing confidence, complacency, constraints, calculation, and collective responsibility of getting vaccinated.9
Most vaccine hesitancy-related scales are used to measure parental attitudes toward vaccination for their children. For instance, even though the VHS is general enough to be applied to many contexts, it has been tested and psychometrically evaluated using survey data taken from parents who are asked about childhood vaccines.10,11 Studies regarding adult vaccines, such as those for influenza or shingles are typically performed in high-income countries and use surveys that take from model frameworks or scoping reviews rather than scales.12–14 While this is simply due to methodological preference, there remains insufficient evidence on adult vaccine hesitancy, even with scales that are adaptable to such situations.
Furthermore, preliminary studies already predict vaccine hesitancy may present a significant challenge for COVID-19 vaccine rollout. One study in 735 university students in Italy found that while the majority of students surveyed would take the COVID-19 vaccine, more than 10% were hesitant.13,14,15 Another study focused on Chinese healthcare workers found that 76.4% of workers were highly accepting of a COVID-19 vaccine.16 Another study conducted in Pakistan describes the serious threat that conspiracy theories had posed against vaccine hesitancy in the past for polio elimination and how similar theories could threaten acceptance of a vaccine for COVID-19.17 Thus, while the descriptive literature of COVID-19 vaccine hesitancy grows, more investigation is necessary. Understanding the association between vaccine hesitancy and COVID-19 acceptance using scales that are applicable to adult immunizations (such as the VHS) can shed light on targets that would improve vaccine uptake across the world. Additionally, if the VHS is valid and reliable for assessing attitudes toward COVID-19 and influenza vaccination, it bears potential use for investigating attitudes toward other adult vaccinations. In this study, we modified the Vaccine Hesitancy Scale to target adult vaccines and provide measures of its reliability and validity relative to influenza vaccine uptake and COVID-19 vaccination acceptance in the United States and in China.
Methods
Study population
This study used a convenient, opt-in, internet-based sample of participants who were recruited by the survey research firm Dynata through social media and advertisements. The sample comprised participants who are at least 18 years old. We administered three surveys: two waves of cross-sectional surveys in the United States in March and June 2020, and one wave in China in March 2020. For each wave, we used quota sampling, whereby the number invited into the study was roughly proportional to the age/gender distribution of the adult population. We eliminated individuals who took under 180 seconds to complete the survey (which we judged to be the minimal legitimate time to thoughtfully complete the questionnaire).
Questionnaire
The questionnaire asked about COVID-19 vaccine acceptance, demographic characteristics, COVID-19 risk perceptions, prevention behaviors, and included the 10-item adult Vaccine Hesitancy Scale (aVHS) to assess vaccine hesitancy. The aVHS had a 5-point Likert scale as answer choices, ranging from least hesitant (1) to most hesitant (5).
The original aVHS has three items which are negatively worded, and seven which are positively worded. To test the effect of negatively versus positively worded questions on the scale’s validity and reliability, the June survey in the United States added a modified aVHS to compare against the original scale. For instance, where item 9 in the original scale reads, “I am concerned about serious adverse effects of vaccines,” the modified item reads, “I am NOT concerned about serious adverse effects of vaccines.” Participants in June 2020 received a unique, randomized set of 10 questions that were a mix of the original and modified items.
In the United States, the questionnaire was administered in English, while the survey administered in China was translated into Chinese and back translated into English. The surveys were created using Qualtrics. The survey items are available at: https://doi.org/10.6084/m9.figshare.13207145 .
Statistical analyses
Validation of the aVHS
Cronbach’s alpha was computed to measure internal consistency of the scale items across the three surveys. We subtracted out items from the scale based on prior research on vaccines to compute the alpha under different circumstances.10,11
For the data collected in June, we compared items on the original aVHS to modified items. The purpose of this analysis was to investigate whether a modified scale functioned similarly as the original scale, as previous researchers have found that the wording of certain items influenced the scale’s internal consistency.10,11 We used this modified scale in the US June survey only because our primary goal was to compare the scales’ wording, regardless of location. After the appropriate items in each scale were reverse-coded, we calculated the mean and standard deviation (sd), as well as tallied the percentage of individuals whose answers corresponded to vaccine-hesitant behaviors. Additionally, to assess any statistical differences in item answers by scale type, we conducted Kruskal-Wallis tests, because the distribution of answers for each item was discrete and often not normally distributed. Any significant results from these tests would indicate that the pair of items are ill-perceived as direct opposite questions, making the original and modified scales less comparable.
Analytical framework
In addition to validating the aVHS, we aimed to investigate factors associated with COVID-19 vaccine acceptance. The survey administered in June in the United States investigated two additional binary outcome variables: whether or not the participant received an influenza vaccine in the past year and whether or not they would get an influenza vaccine in the upcoming fall.
The key independent variable is vaccine hesitancy. Seven of the items were reverse-coded so that higher numbers represent higher vaccine hesitancy. We used two measures for vaccine hesitancy. One was a summary score of the ten aVHS items (range = 10–50) and the other was a dichotomous variable using the cutoff points at 25. The cutoff point was decided by plotting the continuous scores against COVID-19 vaccine acceptance (Figure 1) and influenza vaccination (Figure 2). The relationship between the aVHS score and influenza vaccination was relatively monotonic, but for COVID-19 vaccine acceptance there was a slight inflection at a score of 25. Thus, those who scored at or above 25 were encoded as vaccine hesitant.
Association between aVHS score and vaccination
The relationship between aVHS and vaccination (COVID-19 vaccine acceptance or influenza vaccination) was assessed with a Poisson regression model with robust variance estimates, which output prevalence ratios and 95% confidence intervals (CIs). These models adjusted for age, gender, and monthly household income. For each wave, we assessed aVHS as a continuous variable and dichotomized at 25. We compared fitness between the models with continuous vs dichotomous variables with quasi-likelihood information criteria (QIC) statistics.
All statistical analyses were completed using SAS version 9.4 (SAS Institute, Cary, North Carolina). Plots showing the association between vaccine hesitancy and COVID-19 vaccine acceptance in the United States and China, as well as the relationship between vaccine hesitancy score and past or future flu vaccination decisions used R version 3.6.1. For the China survey, we limited the graph to showing aVHS to a ceiling of 40 (instead of 50) because of a limited number of data points between 40 and 50.
Ethical statement
This study was deemed exempt by the University of Michigan Institutional Review Board (#HUM00180096) and was reviewed and approved by the ethical review committee at Fudan University (#IRB00002408). Before starting the survey, participants read an informed consent form, which they could download as a PDF, and clicked a button to agree.
Results
Characteristics of the study populations
In the March 2020 China survey, 1,822 initially viewed the survey, and 1103 agreed to take the survey. In the US surveys, 692 of 1,069 agreed in March 2020 and 657 of 1,674 agreed in June 2020. Table 1 describes the socio-demographic characteristics of our study populations. The percentage of participants in the March survey in China who would not accept a vaccine was 5.4%, whereas 18.8% of the US participants in the same survey timeline would not accept a vaccine for COVID-19. In the June survey, this number increased to 27.3%.
Table 1.
Characteristics of the study population
| China, March 2020 | USA, March 2020 | USA, June 2020 | |
|---|---|---|---|
| count (%) | count (%) | count (%) | |
| Total count | 1103 | 692 | 657 |
| Age in years | |||
| 18–24 | 154 (14%) | 77 (11%) | 54 (8%) |
| 25–34 | 233 (22%) | 101 (15%) | 116 (18%) |
| 35–44 | 257 (24%) | 127 (18%) | 120 (18%) |
| 45–54 | 189 (18%) | 144 (21%) | 134 (20%) |
| 55–64 | 139 (14%) | 113 (16%) | 114 (17%) |
| 65–99 | 104 (10%) | 127 (18%) | 119 (18%) |
| Gender | |||
| Male | 550 (50%) | 317 (46%) | 304 (46%) |
| Female | 546 (50%) | 375 (54%) | 353 (54%) |
| Rural Residence | |||
| Yes | 766 (78%) | 222 (32%) | 202 (31%) |
| No | 216 (22%) | 470 (68%) | 455 (69%) |
| Monthly Income | |||
| less than $1,999 | – | 137 (20%) | 121 (18%) |
| $2,000 – $4,999 | – | 195 (28%) | 183 (28%) |
| $5,000 – $9,999 | – | 212 (31%) | 204 (31%) |
| more than $10,000 | – | 144 (21%) | 149 (23%) |
| Monthly Income | |||
| less than ¥4,999 | 52 (5%) | – | – |
| ¥5,000 – ¥9,999 | 181 (17%) | – | – |
| ¥10,000 – ¥19,999 | 601 (56%) | – | – |
| more than ¥20,000 | 235 (22%) | – | – |
| Race/ethnicity | |||
| Non-Hispanic White | – | 527 (76%) | 378 (58%) |
| Non-Hispanic Black | – | 50 (7%) | 125 (19%) |
| Hispanic | – | 52 (8%) | 98 (15%) |
| Other | – | 61 (9%) | 56 (8%) |
| Healthcare Coverage | |||
| Yes | – | 612 (89%) | 562 (86%) |
| No | – | 75 (11%) | 93 (14%) |
| Political Affiliation | |||
| Republican | – | 215 (32%) | 190 (29%) |
| Democrat | – | 259 (38%) | 285 (44%) |
| Independent | – | 215 (30%) | 178 (27%) |
| Employment Status | |||
| Employed or Self-Employed | – | 392 (57%) | 359 (55%) |
| Unemployed | – | 294 (43%) | 298 (45%) |
Survey evaluation
Cronbach’s alpha
In general, the aVHS exhibited good internal consistency across all three waves (Table 2). The item sequence that resulted in the highest Cronbach’s alpha values for all three waves was items L1, L2, L3, L4, L6, L7, and L8, with the highest value out of this sequence resulting from the US survey administered in March (Cronbach’s alpha = 0.939). The Cronbach’s alpha values for the other iterations with this sequence were at least 0.8, indicating good internal consistency.18 Subsequently, when items 1 (“Vaccines are important for my health.”) and 3 (“Being vaccinated is important for the health of others in my community.”) were removed from the 7-item sequence, Cronbach’s alpha values declined slightly. However, the values still indicate comparable reliability to the 7-item and 10-item sequences. Lastly, when items 5, 9, and 10 were assessed for internal consistency, the resultant Cronbach’s alpha values indicated poor internal consistency at best, and unacceptable at worst. However, this is likely due to the fact that only three items were assessed, despite having similar wording and tone.
Table 2.
Cronbach’s alpha values by survey iteration and item sequence
| Item Sequence | China, March 2020 | USA, March 2020 | USA, June 2020 |
|---|---|---|---|
| All items (L1-L10) | 0.729 | 0.893 | 0.813 |
| L1-L4, L6-L8 | 0.812 | 0.939 | 0.826 |
| L2, L4, L6, L7, L8 | 0.759 | 0.913 | 0.757 |
| L5, L9, L10 | 0.514 | 0.694 | 0.312 |
Comparatively, the aVHS items from the June wave had lower Cronbach’s alpha values compared to the March surveys because each participant had a mixed assortment of questions that were either from the original aVHS or the modified, obverse version, resulting in a lower, yet still acceptable, internal consistency between items overall. The patterns observed in March data are observed similarly in the June survey.
Scale comparison using june data
To assess the wording of the original aVHS, we compared it against a modified version of the scale, where the statements were obversely worded (Table 3). We performed Kruskal-Wallis tests to identify statistical differences in mean answers that are attributable to differences in wording between pairs of items on each scale that might affect how participants answer the questions. Item pairs 3, 4, 6, 7, and 8 are statistically different from one another, given that their respective test p-values are each lower than an alpha level of 0.05, indicating that the wording of the item pairs are not addressing the exact same construct. This is also evident by the statistically different distributions of the participants’ answers, which are depicted by mean answers and proportional differences in vaccine-hesitant answers for each of the pairs. For instance, while 11.8% of individuals disagreed with original item 4 (“All routine vaccines recommended by the CDC are beneficial.”), 28.2% of individuals presented with obverse item 4 (“Not all routine vaccines recommended by the CDC are beneficial.”) agreed or strongly agreed. Additionally, original item 4 had a mean answer closer to 2, indicating that most people agreed with the statement, whereas obverse item 4 had a mean answer approximating 3, meaning most people neither agreed nor disagreed with that item. The evidence suggests that the items are not direct opposites in wording, despite whichever question they are randomized to.
Table 3.
Comparison of mean values and proportion hesitant based on wording of question in a June 2020 survey using the adult Vaccine Hesitancy Scale
| Item | Question Coding | Mean (sd) | Percentage Hesitant (n = 657) | p-value* |
|---|---|---|---|---|
| L1 | Original | 1.98 (1.09) | 9.8 | 0.8050 |
| Modified | 2.04 (1.25) | 15.0 | ||
| L2 | Original | 1.96 (0.89) | 4.9 | 0.8485 |
| Modified | 2.08 (1.22) | 14.0 | ||
| L3 | Original | 1.88 (1.03) | 7.7 | 0.0436 |
| Modified | 2.09 (1.21) | 14.7 | ||
| L4 | Original | 2.23 (1.08) | 11.8 | <0.0001 |
| Modified | 2.83 (1.19) | 28.2 | ||
| L5 | Original | 3.13 (1.08) | 32.7 | 0.8355 |
| Modified | 3.17 (0.98) | 30.6 | ||
| L6 | Original | 2.29 (0.88) | 6.8 | 0.0494 |
| Modified | 2.48 (1.13) | 17.4 | ||
| L7 | Original | 1.93 (0.92) | 5.5 | 0.0130 |
| Modified | 2.18 (1.15) | 12.3 | ||
| L8 | Original | 1.99 (0.95) | 7.0 | 0.0188 |
| Modified | 2.28 (1.26) | 19.0 | ||
| L9 | Original | 3.18 (1.26) | 43.9 | 0.0680 |
| Modified | 3.03 (1.19) | 35.9 | ||
| L10 | Original | 2.42 (1.19) | 18.0 | 0.3431 |
| Modified | 2.31 (1.05) | 12.2 |
sd, standard deviation
*Kruskal-Wallis test
Concurrent validity
Table 4 describes the prevalence ratios and 95% confidence intervals describing the association between vaccine hesitancy score and vaccination.
Table 4.
Prevalence ratios measuring the association between adult Vaccine Hesitancy Score (aVHS) and COVID-19 vaccine acceptance, as well as past and future influenza vaccination
| aVHS as a continuous predictor | aVHS as a dichotomous predictor | ||||||
|---|---|---|---|---|---|---|---|
| Date & Location | Outcome | Unadjusted Analyses | Adjusted Analyses* | aQIC | Unadjusted Analyses | Adjusted Analyses* | aQIC |
| China, March 2020 | COVID-19 vaccine acceptance | 0.985 [0.980, 0.990] | 0.987 [0.982, 0.992] | 35070.1 | 0.856 [0.807, 0.907] | 0.870 [0.822, 0.921] | 35038.8 |
| US, March 2020 | COVID-19 vaccine acceptance | 0.976 [0.970, 0.982] | 0.976 [0.970, 0.982] | 7021.2 | 0.731 [0.662, 0.808] | 0.737 [0.665, 0.817] | 6976.6 |
| US, June 2020 | COVID-19 vaccine acceptance | 0.971 [0.964, 0.978] | 0.970 [0.962, 0.976] | 4556.9 | 0.721 [0.651, 0.799] | 0.696 [0.629, 0.771] | 4515.7 |
| US, June 2020 | Past flu vaccination | 0.963 [0.954, 0.972] | 0.963 [0.954, 0.972] | 3070.1 | 0.634 [0.552, 0.728] | 0.639 [0.554, 0.737] | 3049.1 |
| US, June 2020 | Future flu vaccination | 0.961 [0.953, 0.969] | 0.961 [0.953, 0.969] | 3749.4 | 0.604 [0.532, 0.685] | 0.604 [0.531, 0.686] | 3706.4 |
*adjusted for age, gender, and income
China, March 2020
Using a continuous version of the vaccine hesitancy score, the prevalence of COVID-19 vaccine acceptance in this study sample, adjusting for age, gender, and household income is 0.985 [95% CI: 0.980, 0.990] times as high in those with higher vaccine hesitancy scores. When vaccine hesitancy score was dichotomized at cutoff score 25, the adjusted prevalence of COVID-19 vaccine acceptance was 0.870 [95% CI: 0.822, 0.921] times as high in those reporting high vaccine hesitancy than in those with low vaccine hesitancy. The QIC statistic for the dichotomous model was lower than the models utilizing the continuous predictor, indicating that the dichotomous model is a better fit.
United States, March 2020
Using the continuous version of the vaccine hesitancy score, the prevalence of COVID-19 vaccine acceptance in this study sample, adjusting for age, gender, and household income is 0.976 [95% CI: 0.970, 0.982] times as high in those with continuously higher vaccine hesitancy scores. Using the dichotomized predictor, the adjusted prevalence of COVID-19 vaccine acceptance scaled back to 0.737 [95% CI: 0.665, 0.817]. The prevalence of vaccine acceptance is 0.737 times as high in those reporting high vaccine hesitancy than in those with low vaccine hesitancy. The QIC indicated that the dichotomous predictor was a better fit.
United States, June 2020
Under the continuous predictor model, the prevalence of COVID-19 vaccine acceptance in this study sample, adjusting for age, gender, and household income is 0.970 [95% CI: 0.962, 0.976] times as high in those with higher vaccine hesitancy scores. Using the dichotomized predictor, the adjusted prevalence of COVID-19 vaccine acceptance was 0.696 [95% CI: 0.629, 0.771] times as high in those reporting high vaccine hesitancy than in those with low vaccine hesitancy.
Additionally, using past influenza vaccination as the outcome, the adjusted prevalence is 0.963 [95% CI: 0.954, 0.972] times higher with increased vaccine hesitancy scores compared to those with lower hesitancy scores. Using the dichotomous predictor, the prevalence of past influenza vaccination is 0.639 [95% CI: 0.554, 0.737] times as high for those reporting high vaccine hesitancy than for those with low vaccine hesitancy. For future flu vaccination, the adjusted prevalence of the outcome increases by 0.961 [95% CI: 0.953, 0.969] for every unit increase in continuous aVHS. On the dichotomous scale, the adjusted prevalence of future flu vaccination is 0.604 [95% CI: 0.531, 0.686] times as high for those reporting high vaccine hesitancy than for those with low vaccine hesitancy. Like the previous models discussed, the lower QIC for the dichotomous, adjusted model indicates a better fit than the continuous, adjusted model.
Discussion
This study is part of a growing body of descriptive research illustrating vaccine hesitancy in the coronavirus pandemic era. Our study showed that COVID-19 vaccine acceptance differed dramatically between China and the United States. The percentage of participants in the March survey in China who would not accept a vaccine was 5.4%, whereas 18.8% of the US participants in the same survey timeline would not accept a vaccine for COVID-19. In the June survey, this number spiked to 27.3%. Additionally, the prevalence of COVID-19 vaccine acceptance in China was higher compared to both surveys conducted in the United States, adjusted for age, sex and income.
The inter-country political and cultural differences may reflect the extent of approval and trust in each country’s respective government’s COVID-19 response. The United States’ hyperpolarized political culture as well as the circulation of countlessly debunked conspiracy theories on social media regarding the pandemic – on the origins of the virus, how it spreads, false assertions about COVID-19 vaccine composition – contributes to a culture of mistrust in the scientific process that is further amplified when those in higher offices support such conspiracies.19,20 One group noted that while acceptance of misinformation amongst the American public did not depend on political party, right-wing news media conjugated the spread of it more than other outlets, contributing to the mistrust expressed toward prominent scientific voices and organizations, such as those at the Centers for Disease Control or National Institute of Allergy and Infectious Diseases.21
China’s stricter response and lockdown in early 2020 – the setting in which our survey was conducted – likely contributed to cultural perceptions of the pandemic and shaped opinions on response. The Chinese government led a centralized, coordinated response to the virus shortly after reporting unexplained pneumonia clusters the WHO on December 31.22 By the end of January, the government had allocated nearly 10 billion CNY to pandemic response efforts.23 Universal healthcare coverage in China distributed according to geographic region, has generally allowed for secure access to care for COVID-19 patients as well and reached vulnerable populations with higher need for care, such as those with disabilities or the elderly.24 Most Chinese citizens expressed satisfaction with their country’s response, although younger and more educated people were more suspicious of the response.25
Internal consistency
In all three surveys, the scale exhibits at least acceptable internal consistency, making the aVHS reliable as a tool for measuring vaccine hesitancy. Moreover, while using a randomized assignment of original and modified scale items, as is the case for June’s survey, produced lower Cronbach’s alphas than the United States’ March survey, the mixing of questions still generates sufficient internal consistency metrics to indicate that this scale remains reliable for studying vaccine hesitancy.
However, the fact that items 5, 9, and 10 exhibit poor internal consistency altogether suggests that they may contribute to lower Cronbach’s alpha values when they are included with the other items. Moreover, the low Cronbach’s alpha for these three items in the June survey is predictable, given that the items comprise opposing versions of the questions. These findings concur with other VHS validation studies. For instance, item 9 (“I am concerned about serious adverse effects of vaccines.”) and item 10 (“I do not need vaccines for diseases that are not common anymore.”) in particular have been poorly associated with other items in the scale in the literature.10,11 Despite the incongruencies that these items contribute, the VHS exhibits good internal consistency, regardless of wording, and thus is a reliable tool to measure vaccine hesitancy in adults.
Validity
We find that this scale exhibits both concurrent and content validity as well. While there is no standard method to measure vaccine hesitancy, this study also illustrates that the WHO SAGE Working Group’s scale exhibits concurrent validity, particularly when the aVHS score is used dichotomously, where the nuances within cross-sectional settings are best depicted. The prevalence of vaccine acceptance, be it for influenza or COVID-19, is much lower in vaccine-hesitant individuals. This pattern is observed in all cases using the dichotomous predictor. Between the March surveys, respondents in China had higher vaccine acceptance than those in the United States, evident by the prevalence ratio being closer to 1 for China than it is for the United States. This reflects the proportion of vaccine-hesitant individuals identified in each survey, where there were more hesitant Americans than Chinese. The preference of the dichotomous aVHS score rather than the continuous aVHS score was decided by comparing the QIC statistics between the models – the lower values indicated that the dichotomous models had the best fit, adding to the evidence of concurrent validity, in other words, the scale is valid when being used to predict a related outcome.
Moreover, because the scale is so widely used in the literature in many different contexts, the scale exhibits content validity. The aVHS has been used in multiple countries and undergone several psychometric evaluations that draw similar conclusions regarding the scale’s properties. Domek et al., who field-tested the scale in Guatemala, found their EFA model was best fit with a 7-item scale (without L3, L6, L9) rather than a 10-item scale.10 Some of the scale’s earlier evaluations also found that item 10 did not agree with the other factors and was thus excluded from any final analyses.11,26 While our study included all scale items in the analyses, we also found that item 10s absence produced higher internal consistency in the scale. In another study on HPV vaccine acceptance in the United States, Szilagyi et al. eliminated item 4 (“All vaccines offered by the government program in my community are beneficial.”) altogether because it did not fit the context of their study in the United States.27 In our study, the question was modified for the United States surveys (“All routine vaccinations recommended by the CDC are beneficial.”) but remained the same in China’s survey, and its presence does not appear to sacrifice internal consistency within each iteration. In sum, because our study draws similar parallels to others in the literature, the scale exhibits content validity.
To sum, the Vaccine Hesitancy Scale both a valid and reliable tool for assessing vaccine hesitancy toward adult vaccinations. We recommend this scale’s use in future investigations that expand the discourse on attitudes and perceptions of adult immunizations. During such a pivotal time as the ongoing coronavirus pandemic, this is especially crucial; not only because it is important to assess how adult attitudes change over the course of the forthcoming COVID-19 vaccination campaigns, but it is equally, if not more important to understand how this pandemic impacts future vaccine decision-making. We find the VHS a sufficient tool to answer such a question.
Study limitations
This is a conveniently sampled, internet-based survey, and thus, there are associated limitations. While we eliminated participants who took a short amount of time to complete the survey, there remains the potential inclusion of individuals who did not complete the survey honestly or as focused as others. Combined with removing fast survey-takers, the three iterations have smaller sample sizes compared to the pre-analysis populations. More data collected either cross-sectionally or longitudinally that utilize larger study populations are needed in order to verify and support the conclusions drawn in this study.
Additionally, we observed more Americans becoming hesitant in the time between March (n = 692) and June (n = 657), but we have yet to observe if such a change also occurred in China. Perhaps our results were affected by the timeliness of the surveys. As information on the coronavirus vaccine appeared over the time between March and June, including information on the truncated timeline on which it is being developed, opinions on the vaccine could have swayed amongst Americans, reinforced by the aforementioned politicization of scientific guidelines on COVID-19 prevention. As we used cross-sectional data, it is unlikely to investigate how people changed and why. Nevertheless, more data are necessary to make such a judgment on China’s state of COVID-19 vaccine acceptance in the months since its lockdown lifted in April; we cannot infer simply from one snapshot in time or from a narrative within another country.
Lastly, this study occurred within one high-income country and one upper middle-income country, each with very different governing styles. In line with similar pitfalls in the general discourse on vaccine hesitancy, evidence on COVID-19 vaccine acceptance is severely lacking in lower income countries. More evidence is required from a diverse array of nations to more accurately assess not only the state of coronavirus vaccine hesitancy, but also how to structure immunization programs that will effectively reach hesitant communities across the globe.
Conclusion
The WHO SAGE Working Group’s Vaccine Hesitancy Scale modified for adults is both a valid and reliable tool in measuring COVID-19 vaccine acceptance, as well as both past and future flu vaccination. While it has its documented drawbacks with inter-item consistency, the aVHS has demonstrated its usefulness in multivariable analysis of COVID-19 and influenza vaccine acceptance in adults. More attention to the reasons for hesitancy in target populations is necessary to design precise interventions.
Acknowledgments
We appreciate the timely responses to our applications to both ethical review boards.
Funding Statement
This project was supported by the National Institute Of Allergy And Infectious Diseases of the National Institutes of Health under Award Number [K01AI137123] and by the National Science Foundation under Award Number [2027836]. The content is solely the responsibility of the authors and does not necessarily represent the official views of the study funders.
Disclosure of potential conflicts of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.