Designing a Consumer-centric Care Management Program by Prioritizing Interventions Using Deep Learning Causal Inference

Abstract

Introduction

Care management is a set of healthcare activities designed to help patients and their support systems effectively manage medical conditions and related psychosocial problems¹. The aim of care management includes improving patients’ health status, enhancing coordination of care, and reducing the need for expensive medical services². Care management programs for high-risk consumers in health are covered by government health insurance, including Medicare and Medicaid, as well as most commercial health insurance. These programs provide consumers access to care management teams, which connect consumers with providers and community resources, and also provide consumers with social determinants of health (SDoH) barrier assistance and self-management knowledge education. Consumer engagement is one of the most critical factors that reflect the success of care management programs³. Better engagement also improves consumer outcomes and experiences of care⁴. However, low engagement rates are found in phone-based care management programs⁵. Therefore, low consumer engagement is an urgent problem to be solved to improve the quality of care management programs.

Previous work found multiple factors that could influence consumer engagement in care management programs, including patient-related factors, health professional-related factors, organization-related factors, and lay community-related factors⁶. While these factors are mainly inspired by sociological and psychological perspectives, few works studied how to improve consumer engagement from a data perspective. In this work, we explore improving consumer engagement from a data perspective by analyzing historical program data in the following care management program.

Post Discharge Management (PDM) is an important care management program in Elevance Health, a health insurance provider. The program is aimed at preventing 30-day readmission for those people who have recently been discharged from hospitals, which corresponds to the Hospital Readmissions Reduction Program (HRRP) by Centers for Medicare & Medicaid Services (CMS). The program lasts 4 weeks. For each enrolled member, case managers make outbound calls once every week to conduct interventions to ensure medication adherence and compliance with follow-up appointments, as well as to refer members to other programs and community resources. From Elevance Health internal report, although the PDM program has reduced potential 30-day readmission by a large amount, the program still suffers from low member engagement.

The current PDM program does not provide case managers with instructions on the interventions to be conducted on each call. Case managers will choose the interventions based on their experience to optimize member engagement. Intuitively, we hypothesize that conducting some interventions on the first call will lead to better engagement with members. To achieve this goal, we use historical user interaction data to analyze member’s engagement by Average Treatment Effect (ATE) with different interventions conducted on the first call. Considering each member’s health condition is different, causal inference methods are used to analyze the data. If the hypothesis is verified, it not only provides a reliable reference to case managers to decide which interventions should be conducted on the first call but also improves the program’s effectiveness by enhancing member engagement.

Causal inference is now widely used in healthcare with the emergence of large amounts of observational data from medical insurance companies and widespread adoption of electronic health records⁷. While the large amounts of data provide probability to calculate and analyze ATE for the tasks that cannot be performed by Randomized Controlled Trials (RCT), the effects of interventions may be confounded by variables that affect both treatment assignment as well as treatment outcome⁸. To correctly calculate the ATE using observational data, lots of causal inference methods have been developed in the past decades. Bayesian Additive Regression Trees (BART)^9,10 is a prevailing machine learning model for causal inference and has demonstrated superior performance in a causal inference competition¹¹. However, BART runs relatively slow in using Markov Chain Monte Carlo (MCMC)¹², which makes it hard to process industry-scale data. With the development of artificial intelligence in recent years, a lot of deep learning methods are also proposed for causal inference^13,14,15. Balancing Covariates Automatically Using Supervision (BCAUS)⁸ is a deep learning model for causal inference that uses neural networks to remove covariate imbalance by propensity modeling. It is much faster than BART and shows comparable performance with BART on a benchmark dataset⁸, which provides possibilities to analyze industry-scale data.

In this work, we use BCAUS as a causal inference model to prioritize the interventions that should be conducted on the first call to optimize member engagement in the PDM program. Our data is from the Elevance Health database, which contains covariates, interventions, and outcomes of each member enrolled in the PDM program. Experiments from two settings of BCAUS and one setting of BART demonstrate the robustness and correctness of our research, which identifies that interactive interventions, specifically Medication Reconciliation and Follow up Appointment, if conducted on the first call, will lead to better member engagement than other interventions. In summary, this work has the following contributions: 1) We explore improving consumer engagement from a data approach using causal inference in care management. 2) Our prioritization of interventions provides a reliable reference for case managers to optimize member engagement in the PDM program. 3) Implementation of BCAUS in our work verifies the feasibility of using deep learning for causal inference in large-scale healthcare data.

Methods

The Post Discharge Management (PDM) program is a 4-week care management program in Elevance Health. Members recently discharged from hospitals are selected for the program to prevent 30-day readmission. Each week, case managers will call the enrolled member and choose one or multiple interventions from all seven interventions to conduct. If a member is engaged with the program and closes all of the seven interventions in four weeks, the member is considered to successfully graduate from the program.

The seven interventions are Medication Reconciliation, Problem List, Problem Root Cause, Follow up Appointment, Appointment Scheduled, Health Record, and Referral. Medication reconciliation is an intervention in which the case manager review and compare patient’s current medications with medication history, to identify potential prescription errors; Problem list is an intervention for case manager to note down list of problems reported by provider; In problem root cause intervention, the case manager dig into patient’s medical record and discuss with patient to identify root cause of the problems; Follow up appointment is for the case manager to review all follow up doctor appointments that are required; Appointment scheduled is an intervention to help patient scheduling doctor appointments; Health record is an intervention in which the case manager review and update patient’s health record; Referral is an intervention in which the case manager provider patient with doctor referral. In this work, we use graduation as the metric to indicate a member’s engagement.

To find whether conducting some interventions on the first call will lead to better member engagement, we use causal inference models to calculate the ATE of members’ graduation for each intervention or each intervention pair. If the covariates are correctly balanced, the intervention with a higher ATE will lead to better outcomes in the total population if conducted on the first call.

Causal Inference Model Balancing Covariates Automatically Using Supervision (BCAUS)⁸ model for causal inference is applied in this work. It is a deep learning model that predicts the intervention and balances the covariates at the same time by using a joint loss penalizing both the incorrect prediction of the assigned treatment and the imbalance of covariates calculated by inverse probability of treatment weighting (IPTW). Figure 1 is the workflow of the BCAUS model. The model uses all of the covariates as input and predicts the intervention assignment. Covariates with IPTW are used to calculate the covariate bias between treatment group and control group. The model is trained by minimizing the binary cross-entropy (BCE) loss of intervention prediction and mean squared error (MSE) loss of the covariate bias. After training, propensity score together with outcome is used to calculate the ATE^16,17. To verify the performance of BCAUS model on balancing the covariates, we calculate the standardized difference of covariates between treatment group and control group with and without IPTW. We use 0.1 as a commonly used threshold for ensuring the removal of covariate bias between the two groups^16,17. We also include another commonly used machine learning model for causal inference, Bayesian Additive Regression Trees (BART)⁹, to cross-validate the performance of BCAUS model. Instead of using propensity score, BART is a model using counterfactual results to calculate ATE. The two models with different methods to calculate ATE will cross-validate their performance if both models produce similar results.

Open in a separate window

Figure 1:

The workflow of BCAUS model. BCE loss for minimizing intervention assignment error and MSE loss for minimizing covariate bias are used together to optimize the model.

Experiment Setting In this work, we use two experiment settings to get the ranking of interventions to be conducted on the first call for better member engagement. In the first setting, we use the cases who had a specific intervention on the first call as the treatment group and the cases who did not have the same intervention on the first call as the control group, regardless of whether they had other interventions or not on the first call. Instead, we add the information of whether they had other interventions or not into the covariates, which are supposed to be balanced by propensity modeling. The ranking of the interventions is based on the ATE calculated for each intervention. Interventions with higher ATE are better for member engagement in the total population if conducted on the first call. In the second setting, we calculate the ATE between intervention pairs. For every two interventions, we select the cases that had only one specific intervention on the first call as the treatment group and the cases that only had another intervention as the control group. By calculating the ATE of the two groups, we can get a relative ranking of these two interventions. After all the intervention pairs are calculated, we will get a 7 × 7 matrix, in which each value measures the ATE of one pair of two interventions, with one intervention representing the row of the value and another intervention representing the column of the value. By comparing these ATE values, we can get the whole ranking of the 7 interventions. To distinguish the two settings, we call the first setting Single Intervention and the second setting Intervention Pair.

Results

Data The PDM program data is from the Elevance Health database. The data has been subjected to the company’s deidentification processes to ensure the removal or modification of personal identifiers, thus minimizing the risk of re-identification and ensuring compliance with privacy regulations and standards. Data with the start date from 01/01/2021 to 05/05/2022 is used in this work, with 27,300 cases in total. For each case, interventions conducted on the first call and graduation status are extracted from the database with related members’ data. Member’s data including member’s health conditions, medications, health service usages, UM frequencies, risk scores, and comorbidities are used as covariates in the causal inference models with 75 covariates in total. The number of cases who had a specific intervention in the first week ranges from 8,465 to 16,385. Table 1 lists the number of cases that had or did not have a specific intervention for each intervention. The percentages of cases that are conducted with a specific intervention on the first call among all the cases are from 30.9% to 60.0%. These numbers indicate that case managers select the interventions on the first call neither completely randomly nor similarly among all the case managers, which also suggests the necessity of providing a reference for case managers to choose interventions to conduct. Using this data, we conduct experiments of both settings using BCAUS, and an extra experiment using BART to cross-validate the results.

BCAUS Single Intervention In this setting, we use the cases who had one specific intervention as the treatment group and cases who did not have that intervention as the control group. We add the other interventions to the covariates and train the BCAUS model. ATE is calculated to measure how the specific intervention influences member engagement in the total population. Using Elevance Health internal server, it takes less than a minute to train a BCAUS model, which is the most time-consuming part of getting the ATE from BCAUS. Table 2 shows the result of ATE for each intervention using BCAUS. In the table, Medication Reconciliation has the highest ATE among all the interventions. Based on the results, all the interventions are recommended in the following order.

Medication Reconciliation > Follow up Appointment > Appointment Scheduled > Referral > Health Record > Problem List > Problem Root Cause

To learn the performance of how BCAUS balances covariates in this setting, we calculate the standardized difference of covariates of raw data and covariates balanced with IPTW from BCAUS separately. Figure 2 is the distribution of standardized differences of all member’s data covariates from raw data and BCAUS IPTW. X axis is the standardized difference. Y axis is the index of covariates. Whiskers are at 5 and 95 percentiles. Covariates with standardized differences smaller than 0.1 are treated as balanced covariates. In this setting, all the covariates are balanced after BCAUS IPTW, while lots of covariates are not balanced in raw data. Therefore, BCAUS has a high performance in balancing the covariates, which also suggests that our result of intervention prioritization is reliable.

Open in a separate window

Figure 2:

Distributions of standardized difference of all the member’s data covariates from raw data (left) and from BCAUS IPTW (right). All the covariates are balanced after BCAUS IPTW.

BCAUS Intervention Pair In this setting, for every two interventions, we select the cases that only had one intervention as treatment group and that only had the other intervention as control group. For each intervention pair, we train a BCAUS model and calculate the ATE. ATE for each intervention pair represents the impact on member engagement if conducting the intervention of the treatment group on the first call instead of the intervention of the control group. Table 3 shows the result of ATE for each intervention pair using BCAUS. The indices of row and column from 1 to 7 represent Medication Reconciliation, Problem List, Problem Root Cause, Follow up Appointment, Appointment Scheduled, Health Record, and Referral. Columns represent interventions of the treatment group and rows represent interventions of the control group. Based on the results, all the interventions are recommended in the following order.

Table 3:

ATE for each intervention pair in the Intervention Pair setting calculated by BCAUS.

	1	2	3	4	5	6	7
1		-0.383	-0.319	-0.140	-0.222	-0.355	-0.130
2	0.381		-0.069	0.195	0.158	-0.047	0.062
3	0.313	0.071		0.246	0.155	0.027	0.148
4	0.150	-0.195	-0.246		-0.00	-0.172	-0.023
5	0.220	-0.157	-0.155	0.017		-0.124	-0.015
6	0.351	0.043	-0.025	0.177	0.123		0.074
7	0.131	-0.063	-0.148	0.027	0.012	-0.078

Open in a separate window

Discussion

This study explores the impact of interventions conducted on the first call on consumer engagement in a care management program from a data perspective. We use a deep learning causal inference model BCAUS to analyze how the interventions conducted on the first call influence member engagement, specifically member’s graduation, in the PDM program, and cross-validate the results using another prevailing machine learning model BART. Our results show that the two interventions that require interaction between member and case manager, Medication Reconciliation and Follow up Appointment, continuously get the highest ATE, which indicates that member’s engagement will be improved in the total population if these two interventions are conducted on the first call. On the other hand, Problem Root Cause, which is a relative technical intervention with a structured process¹⁸, continuously gets the lowest ATE, which suggests that case managers may consider conducting this intervention later from the perspective of improving member engagement. However, Problem Root Cause has the highest percentage 60.0% to be conducted on the first call from statistics of the data in Table 1. Medication Reconciliation and Follow up Appointment have relatively low percentages, 32.2% and 36.1%, separately. The inconsistency of our prioritization and current program situation emphasizes the importance of this work in providing a reference of intervention conduction for improving member engagement.

Our results come from two experiments using BCAUS and one experiment using BART. In the two BCAUS experiments, there is only one minor difference, where two adjacent interventions are reversed in the two results. Considering one experiment using the whole data and the other using the subgroup of the data, and the ATE of the two reversed interventions are relatively close, these two experiments demonstrate the robustness of BCAUS method. BART experiment cross-validates the results of BCAUS. Only one intervention’s order from BART is different from BCAUS with two places advanced. Since BCAUS and BART use two different causal methods to calculate the ATE, all three experiments demonstrate the correctness of our results. Moreover, BCAUS is an order of magnitude faster than BART in Elevance Health internal server. In this way, our work not only provides a reliable reference for case managers to improve members’ engagement and promote consumer health, but also demonstrates the ability of BCAUS as a deep learning causal inference model to fast and accurately process large-scale healthcare data.

Experienced nurses in Elevance Health found our results intuitive. Medication Reconciliation is an effective intervention in readmission prevention¹⁹. From the nurses’ perspective, it is an educational topic and a very engaging intervention. The conversation would start with ”Do you understand your medication?” Many members were sick and just got out of the hospital, so they did not get a chance to grab the understanding. By helping members understand their medications and concerns on the first call, case managers will achieve better engagement with members throughout the entire program period. On the other hand, Problem Root Cause is an intervention where case managers go through the member’s records and summarize the root cause, which does not require the member’s participation. Case managers conducting this intervention miss the opportunity to connect with members, which may lead to relatively poor engagement. Our results are also consistent with previous studies on patient engagement in hospital settings, which show that interactive interplay between the patient and nurse helps promote patient engagement²⁰. However, documentation work from nurses does not necessarily reflect patient engagement²¹. Nurses can be task-oriented in some activities, gaining information from patients without sufficient patient engagement²⁰. These studies may explain our findings in care management. Interactive interventions such as Medication Reconciliation promote consumer engagement, while documentation work such as Problem Root Cause contributes less to improving consumer engagement.

Conclusion

In this work, we explore the impact of interventions conducted on the first call on consumer engagement in a care management program of Elevance Health. Our results show that more interactive interventions such as Medication Reconciliation and Follow up Appointment help promote consumer engagement, while less interactive and more technical interventions such as Problem Root Cause may not contribute to consumer engagement. Our experiments demonstrate the robustness of the deep learning causal inference model we use and the correctness of our results and suggest the feasibility of using deep learning causal inference to process large-scale healthcare data. Our intervention prioritization provides a reliable reference for case managers to select the interventions to be conducted on the first call to improve member engagement. However, our pipeline has limitations. For each intervention, a BCAUS model needs to be trained to get the ATE for prioritizing. When the number of interventions is large, it will be difficult to calculate ATEs for all the interventions. For example, if we want to prioritize intervention combinations in the same setting, there will be 49 intervention combinations containing 2 interventions and 343 intervention combinations containing 3 interventions. Calculating all ATEs will be difficult and prioritizing intervention combinations with this large amount of numbers will be inaccurate. In future work, we intend to implement the proposed pipeline in other care management programs in Elevance Health to promote consumer health. We are also building efficient deep learning causal inference pipelines for complex healthcare programs dealing with more interventions to promote outcomes and benefit consumers.

Acknowledgments

References