Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Feb;13(2):e230089.
doi: 10.57264/cer-2023-0089. Epub 2024 Jan 23.

Visualizing the target estimand in comparative effectiveness studies with multiple treatments

Gabrielle Simoneau  1 , Marian Mitroiu  2 , Thomas Pa Debray  3   4 , Wei Wei  2 , Stan Rw Wijn  4   5 , Joana Caldas Magalhães  4 , Justin Bohn  6 , Changyu Shen  7 , Fabio Pellegrini  8 , Carl de Moor  7
Affiliations

Visualizing the target estimand in comparative effectiveness studies with multiple treatments

Gabrielle Simoneau et al. J Comp Eff Res. 2024 Feb.

Abstract

Aim: Comparative effectiveness research using real-world data often involves pairwise propensity score matching to adjust for confounding bias. We show that corresponding treatment effect estimates may have limited external validity, and propose two visualization tools to clarify the target estimand. Materials & methods: We conduct a simulation study to demonstrate, with bivariate ellipses and joy plots, that differences in covariate distributions across treatment groups may affect the external validity of treatment effect estimates. We showcase how these visualization tools can facilitate the interpretation of target estimands in a case study comparing the effectiveness of teriflunomide (TERI), dimethyl fumarate (DMF) and natalizumab (NAT) on manual dexterity in patients with multiple sclerosis. Results: In the simulation study, estimates of the treatment effect greatly differed depending on the target population. For example, when comparing treatment B with C, the estimated treatment effect (and respective standard error) varied from -0.27 (0.03) to -0.37 (0.04) in the type of patients initially receiving treatment B and C, respectively. Visualization of the matched samples revealed that covariate distributions vary for each comparison and cannot be used to target one common treatment effect for the three treatment comparisons. In the case study, the bivariate distribution of age and disease duration varied across the population of patients receiving TERI, DMF or NAT. Although results suggest that DMF and NAT improve manual dexterity at 1 year compared with TERI, the effectiveness of DMF versus NAT differs depending on which target estimand is used. Conclusion: Visualization tools may help to clarify the target population in comparative effectiveness studies and resolve ambiguity about the interpretation of estimated treatment effects.

Keywords: comparative effectiveness; matching; multiple sclerosis; propensity score; visualization.

PubMed Disclaimer

Conflict of interest statement

Competing interests disclosure

M Mitroiu, W Wei, C Shen, F Pellegrini, are employees of Biogen and hold stocks or stock options. At the time of conceiving this manuscript, J Bohn, G Simoneau and C de Moor were employees of Biogen and held stocks or stock options. TPA Debray, SRW Wijn and JC Magalhães received consulting fees from Biogen. The authors have no other competing interests or relevant affiliations with any organization or entity with the subject matter or materials discussed in the manuscript apart from those disclosed.

Figures

Figure 1.
Figure 1.. Conceptual illustration of the bivariate covariate space for three treatments A (green), B (orange) and C (purple).
Various target populations are illustrated for comparing three treatments with shaded gray areas: (i) the entire population, (ii) the population of individuals that are eligible to receive A, (iii) the population of individuals that are eligible to receive B, (iv) the population of individuals resembling those who received B and are eligible to receive A, (v) the population of individuals resembling those who received C and are eligible to receive A, and (vi) the population of individuals that are eligible to receive all three treatments A, B, and C.
Figure 2.
Figure 2.. Scatter plot and bivariate ellipses representing the distribution of two covariates x1 and x2 in the super-population of n = 120,000 individuals.
The larger points with a black outline represent the mean of the bivariate distribution, by treatment. A random sample of individual observations are represented with lighter points. The ellipses represent the 95th percentile of the bivariate distribution of x1 and x2 by treatment based on all individuals.
Figure 3.
Figure 3.. Bivariate ellipses representing the distribution of x1 and x2 before matching (first column) and in the matched sample by target ATT (columns ATT-A, ATT-B and ATT-C) by pairwise treatment comparison in the simulation study.
The larger points with a black outline represent the mean of the bivariate distribution, by treatment. A random sample of individual observations are represented with lighter points. The ellipses represent the 95th percentile of the bivariate distribution of x1 and x2 by treatment based on all individuals. Grayed-out areas are empty because they correspond to an ATT that cannot be targeted for each pairwise treatment comparison. ATT: Average treatment effect in the treated.
Figure 4.
Figure 4.. Joy plot for the univariate distributions of x1 and x2 in the original sample and in ATT-A and ATT-B for individuals receiving treatment A (green) or B (orange).
ATT: Average treatment effect in the treated.
Figure 5.
Figure 5.. Bivariate ellipses representing the distribution of age and disease duration before matching (first column) and in the matched sample by target ATT (columns ATT-TERI, ATT-DMF, and ATT-NAT) by pairwise treatment comparison in MS PATHS, USA and Europe, 2015–2022.
The larger points with a black outline represent the mean of the bivariate distribution, by treatment. Individuals are represented with smaller points, where points for control individuals selected multiple times in the matched sample are bolder. The ellipses represent the 95th percentile of the bivariate distribution of age and disease duration, by treatment. Grayed-out areas are empty because they correspond to an ATT that cannot be targeted for each pairwise treatment comparison. ATT: Average treatment effect in the treated; DMF: Dimethyl fumarate; MS PATHS: Multiple Sclerosis Partners Advancing Technology and Health Solutions; NAT: Natalizumab; TERI: Teriflunomide.
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. European Medicines Agency. Data Analysis and Real World Interrogation Network (DARWIN EU) (2021). https://www.ema.europa.eu/en/about-us/how-we-work/big-data/data-analysis...
    1. United States Food and Drug Administration. Real-World Evidence (2022). https://www.fda.gov/science-research/science-and-research-special-topics...
    1. Cui ZL, Hess LM, Goodloe R, Faries D. Application and comparison of generalized propensity score matching versus pairwise propensity score matching. J. Comp. Eff. Res. 7(9), 923–934 (2018). - PubMed
    1. Samuel M, Batomen B, Rouette J et al. Evaluation of propensity score used in cardiovascular research: a cross-sectional survey and guidance document. BMJ Open 10(8), e036961 (2020). - PMC - PubMed
    1. Yao XI, Wang X, Speicher PJ et al. Reporting and guidelines in propensity score analysis: a systematic review of cancer and cancer surgical studies. J. Natl Cancer Inst. 109(8), djw323 (2017). - PMC - PubMed

Publication types

-