External validation of a proprietary risk model for 1-year mortality in community-dwelling adults aged 65 years or older

Abstract

Objective

To examine the discrimination, calibration, and algorithmic fairness of the Epic End of Life Care Index (EOL-CI).

Materials and Methods

We assessed the EOL-CI’s performance by estimating area under the receiver operating characteristic curve (AUC), sensitivity, and positive and negative predictive values in community-dwelling adults ≥65 years of age in a single health system in the Southeastern United States. Algorithmic fairness was examined by comparing the model’s performance across sex, race, and ethnicity subgroups. Using a machine learning approach, we also explored local re-calibration of the EOL-CI considering additional information on past hospitalizations and frailty.

Results

Among 215 731 patients (median age = 74 years, 57% female, 12% of Black race), 10% were classified as medium risk (15-44) and 3% as high risk (≥45) by the EOL-CI. The observed 1-year mortality rate was 3%. The EOL-CI had an AUC 0.82 for 1-year mortality, with a positive predictive value of 22%. Predictive performance was generally similar across sex and race subgroups, though the EOL-CI displayed better performance with increasing age and in older adults with 2 or more outpatient encounters in the past 24 months. Local re-calibration of the EOL-CI was required to provide absolute estimates of mortality risk, and calibration was further improved when the EOL-CI was augmented with data on inpatient hospitalizations and frailty.

Discussion

The EOL-CI demonstrates reasonable discrimination, albeit with better performance in older adults and in those with greater health system contact.

Conclusion

Local refinement and calibration of the EOL-CI score is required to provide direct estimates of prognosis, with the goal of making the EOL-CI a more a valuable tool at the point of care for identifying patients who would benefit from targeted palliative care interventions and proactive care planning.

Introduction

Healthcare utilization and spending sharply increase in the last years of life.^1,2 End-of-life care frequently involves repeated hospitalizations and procedures that may neither extend life nor alleviate symptoms, often misaligned with patients’ goals.³ This disconnect has driven significant interest in promoting advance care planning (ACP) discussions, as well as the development of prediction models and care pathways that identify patients most likely to benefit from interventions through screening, prognostication, and decisional support.⁴ The combination of modern electronic health record (EHR) systems with advances in clinical prediction modeling has led to a proliferation of risk stratification tools designed to serve as personalized assessments of prognosis.⁵ Notably, there have been numerous prediction models developed for mortality (within 1 to 3 years) or other undesirable outcomes with a goal of improving the allocation of limited palliative care resources to those with serious illness most likely to benefit from targeted interventions.^6–9 While many models that predict mortality exhibit reasonable discrimination,¹⁰ the generalizability and calibration (ie, how much predicted risk aligns with actual observed risk) tend to be insufficient for clinical deployment, at least for the allocation of limited human resources such as geriatrics or palliative care consults.¹¹ This underscores the importance of external validation not only for assessing model performance but also for informing best practices in prediction model implementation within clinical workflows, ensuring that such tools are both accurate and actionable in real-world settings.^12,13

Another limitation of many current prognostic models is that they lack automated integration into the EHR, which hampers their effectiveness at the point of care in informing prognosis-based decision-making. One exception is the End-of-Life Care Index (EOL-CI), a proprietary model for predicting 1-year mortality developed by Epic Systems Corporation (Epic, Verona, WI).¹⁴ The EOL-CI offers pragmatic advantages by being automatically integrated into the EHR, enabling real-time risk stratification. Developed with data from over 550 000 patients across 3 health systems using penalized logistic regression, the EOL-CI leverages structured EHR data on demographics (race and/or ethnicity are not model inputs), labs, medications, and comorbidities. The EOL-CI had an estimated area under the receiver operator curve (AUC) of 0.90 (0.83 in patients 65 years or older) and positive predictive values in the range of 0.20-0.40 depending on the score threshold and health system. There has been only one external validation study of the EOL-CI, which showed moderate agreement with physician impressions of short-term prognosis.¹⁵ As physician opinions regarding prognosis have generally been shown to be subjective and inaccurate,^16,17 there is a clear need for independent, external, and ongoing quantitative validation of the EOL-CI leveraging prospective follow-up for mortality.¹⁸ This is particularly relevant given recent concerns raised about another proprietary, EHR-integrated prediction model by Epic, designed for early sepsis detection.^19,20

We aimed to externally validate the EOL-CI in a large primary care population of older adults aged 65 or older. Beyond assessing overall performance, we evaluated the algorithmic fairness of the EOL-CI, defined as the model's ability to predict outcomes consistently across different demographic groups without exacerbating existing health disparities.^21,22 We assessed whether including past inpatient hospitalizations and an automated electronic measure of frailty could improve the EOL-CI's prognostic performance.²³ Additionally, we examined one potential clinical application of the EOL-CI related to the under-utilization of ACP discussions in outpatient settings.²⁴ To gauge the potential for the EOL-CI to be used as a prioritization tool for targeted outreach, we examined the association of the EOL-CI with incident outpatient encounters that utilized billing codes for ACP.

Methods

This prognostic study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD+AI) reporting guideline.²⁵ The study was approved by the institutional review board at Wake Forest University Health Sciences (IRB00109077) under a waiver of HIPAA authorization.

Study population

We extracted EHR data for adults aged 65 years or older (as of July 22, 2023 [index date]) in the Atrium Health-Wake Forest Baptist (AH-WFB) Health System in Winston-Salem, North Carolina. AH-WFB is a 6-hospital health system with a 24-county catchment area covering Western North Carolina and Southern Virginia. Primary care is organized through an integrated network of over 165 practices and more than 1000 primary care providers. AH-WFB has used Epic as its EHR across ambulatory and inpatient settings since 2012. Epic is one of the largest and leading EHR providers, with the largest hospital EHR market share in the world and 39.1% of the hospitals in the United States.²⁶ In July 2023, we implemented a process to store monthly snapshots of all available EOL-CI scores, as historical values of the EOL-CI were not previously stored in our local Epic databases, nor was it possible to retrospectively calculate the EOL-CI given its proprietary nature.

The Epic End of Life Care Index

The penalized logistic regression model for the EOL-CI was developed using data from 2014 to 2018 from 3 anonymous healthcare organizations. The model has 46 features including demographic information (not including race or ethnicity), medications, labs, and diagnoses and produces a score from 0 to 100. We utilized the score thresholds suggested by Epic: low risk (<15), medium risk (15-45), and high risk (45 or greater). While the EOL-CI was activated and available in our EHR, there were no formal clinical care protocols that were triggered based on the score during the study timeframe.

Additional data from the EHR

For patients aged 65 years or older, we extracted demographic information, information on connectivity to primary care, ICD-10 diagnosis codes (from encounter diagnoses and the problem list), past outpatient encounters, and past inpatient hospitalizations over a 2-year look-back window relative to index date of July 22, 2023. Self-reported race and ethnicity was extracted from the EHR based on over 40 race categories and over 15 categories for ethnicity. Race was then categorized as White, Black, or Other (inclusive of patients that reported multiple races or declined to respond), with Ethnicity categorized as Hispanic or Latino or Not (including patients that declined to report their ethnicity). We summarized multimorbidity using the weighted Charlson Comorbidity Index.²⁷ In addition, we leveraged scores from the electronic Frailty Index (eFI), which quantifies the proportion of age-related deficits present for an individual, based on the theory of deficit accumulation.²³ The eFI has been integrated into the instance of Epic at AH-WFB since October 2019, calculated on a rolling basis for all patients 55 years or older with 2 or more outpatient visits with a measured blood pressure in the past 2 years.²⁸

Mortality and other outcomes

Mortality information in our EHR is supplemented through a monthly deterministic linkage (based on the last 4 digits of the social security number, name, date of birth, and sex) to the North Carolina State Center for Health Statistics death index, which captures deaths for North Carolina residents. We additionally extracted information on outpatient encounters in the year following the study index date that utilized Current Procedure Terminology billing codes for ACP (99497 and 99498).

Sample size requirements

We used a simulation approach to approximate the sample size required for external validation.²⁹ These calculations aimed to provide a pre-specified level of precision in estimating confidence interval width for the calibration slope in an external dataset. As we expected to have a large population available, the intent with these calculations was to estimate our ability to validate the EOL-CI within smaller population subgroups. Assuming a C-statistic of 0.90 and a 1-year mortality incidence of 2%, we estimated that 20 000 patients (∼420 events) would be required to have an estimated calibration slope standard error of 0.05 (ie, a confidence interval width from 0.90 to 1.10). With 10 000 patients (∼210 events), we estimated that the standard error for the calibration slope would be 0.07 (confidence interval width of 0.86-1.14).

Statistical analysis

Validation metrics

We used time to event models and performance metrics that account for right-censoring. We assessed the performance of the EOL-CI score with respect to discrimination and calibration. One challenge with evaluating the EOL-CI is that the score itself is not an estimated probability. Not only is this a critical limitation of the tool as it does not map to an absolute quantification of prognosis, it also does not provide a means to assess calibration. Despite this issue, we evaluated the EOL-CI as if it represented an estimated probability, as we believe this interpretation, albeit incorrect, is prevalent with real-world implementation, ie a provider interpreting an EOL-CI score of 50 as representing a 50% chance of death in the next year.

Algorithmic fairness

We examined fairness of the EOL-CI based on 2 common measures: equalized odds and equal opportunity.³⁰ Equal opportunity is defined as equal sensitivity or positive predictive values across all levels of the domain of interest, ie comparing men to women. Equalized odds requires that a predictive model exhibit equal opportunity and equal specificity or false-positive rates. We compared predictive metrics across subgroups using bootstrap resampling (1000 bootstrap replicates). We did not consider a third common measure, demographic parity, as men tend to exhibit lower life expectancy compared to women with increasing age, and members of racial and ethnic minority groups also tend to exhibit different mortality rates compared to White older adults.³¹ Therefore, there is not an expectation that the EOL-CI would exhibit the same distribution of predicted risk across these population groups.

Recalibration and augmentation of the EOL-CI. We considered 2 enhancements to the EOL-CI and evaluated each of them using 10-fold cross-validation. First, we re-calibrated the EOL-CI to our population by fitting a Cox proportional hazards model with time to death as the outcome and the EOL-CI as the single predictor, and then used this model to generate 1-year predicted probabilities of mortality for observations in the held-out fold, repeating the process 10 times holding out each fold once. Second, using the same folds, we evaluated the combination of re-calibrating the EOL-CI and augmenting it with the eFI and the number of inpatient hospitalizations within the past 24 months. The augmented model was developed using an oblique random survival forest (ORSF), a flexible machine learning technique for right-censored survival data.^32,33

Associations with Incident Advance Care Planning Encounters. Separately for each EOL-CI score category, we estimated the cumulative incidence over 1 year of outpatient encounters that involved billing codes for ACP, accounting for the competing risk of death.³⁴ All analyses were performed using the R Statistical Computing Environment.

Results

The population included 215 731 adults aged 65 years or older (Table 1). Patients were a median age of 74 years [interquartile range, 69-80 years], 122 235 were female (57%), and 25 297 (12%) self-identified as of Black race. Using the Epic-recommended thresholds, 22 197 (10%) adults were classified as medium risk, while 6108 were classified as high risk (3%). Several factors demonstrated expected associations with increasing scores on the EOL-CI. As the score accounts for age, adults in the medium-risk and high-risk categories were 7 years older on average than those categorized as low risk (P < .001). In addition, with increasing scores on the EOL-CI, we observed a higher prevalence of comorbidity, higher levels of frailty based on the eFI, and greater prevalence of past outpatient encounters and inpatient hospitalizations.

Over 1 year, we observed an increasing risk of mortality with higher scores on the EOL-CI, with 1-year mortality rates of 1%, 9%, and 22% for low-, medium-, and high-risk adults, respectively (Table 1 and Figure 1). The EOL-CI demonstrated reasonable discrimination, with an AUC of 0.82 (Figure 2). Using the high-risk score threshold, the sensitivity for 1-year mortality was 0.23 while the positive predictive value was 0.22 (Table 2). The medium-risk threshold had higher sensitivity (0.57), though lower positive predictive value (0.12). Overall, negative predictive values were high, estimated to be 0.99 and 0.98 for the medium-risk and high-risk thresholds, respectively. Given the large available sample size, we observed several statistically significant differences in the performance of the EOL-CI across the characteristics we considered, although many of these differences were small in magnitude, especially for specificity and negative predictive value (Table 2, Tables S1 and S2). In terms of sensitivity and positive predictive value, we observed larger differences as a function of age, connectivity to primary care, and the number of past outpatient encounters. For example, we observed significantly worse predictive performance for the EOL-CI in older adults with less than 2 outpatient encounters in the past 24 months compared to those with 2 or more, in terms of lower sensitivity (difference for high-risk threshold = -0.17; 95% CI, −0.19 to −0.15) and lower positive predictive value (difference for high-risk threshold = -0.08; 95% CI, −0.10 to −0.06; Table S2). These differences are reflected in the calculated equal opportunity measure (which maximizes differences in sensitivity across subgroups of a particular characteristic), with past outpatient encounters (0.17; 95% CI:, 0.15-0.19), connectivity to primary care (0.15; 95% CI, 0.07-0.22), and age (0.12; 95% CI, 0.09-0.15) having the largest magnitude differences for the high-risk threshold (Table 3).

When we examined the calibration of the EOL-CI treating it as an estimated probability (Figure 2), we observed that the EOL-CI score will overestimate observed risk. For example, for an older adult with an EOL-CI score of 75, the estimated 1-year mortality rate in our population was approximately 25%. However, calibration can be significantly improved leveraging an augmented model that is locally calibrated (directly provided survival probability estimates) and flexibly incorporates additional predictors that are not a component of the EOL-CI (Figure 2). While we observed excellent calibration of the re-calibrated index overall (Table S3), we did observe some calibration slopes statistically different from 1 related to the same characteristics discussed above (age and connectivity to primary care).

Figure 3 shows the cumulative incidence of outpatient encounters with an ACP billing code restricted to adults with a primary care provider affiliated with our health system. The incidence estimates account for the competing risk of death and are stratified by EOL-CI category. Overall, the incidence of ACP encounters was generally low (<10%), though there was an increasing relationship with the EOL-CI score, with adults categorized as medium risk or high risk exhibiting a higher incidence of ACP encounters over 1 year of follow-up.

Discussion

This study underscores the critical role of external validation in assessing the performance of proprietary models like the EOL-CI in diverse healthcare settings to allow for broader applicability. We found that the discrimination of the EOL-CI (AUC = 0.82) was close to Epic’s reported AUC of 0.83 in patients 65 years or older and consistent with the performance of other risk models for shorter term mortality.¹⁰ This level of discrimination should be viewed favorably, as there is an upper limit of what can be expected from any risk model for shorter-term mortality.³⁵ When we examined the performance of the EOL-CI across subgroups, there were generally small differences in performance, with some exceptions. The EOL-CI, even when re-calibrated, performed better with increasing age, performed worse in older adults with limited contact with our health system for outpatient care, and there was a non-significant indication of lower predictive performance in older adults of Hispanic or Latino ethnicity. It is not surprising that the EOL-CI performs better with increasing age and in adults with more utilization within a particular health system. Short-term deaths in adults closer to middle age (ie, 65-75 years) are inherently less predictable (ie, more random) than deaths in adults 85 years or older. There is also correspondingly more complete, though potentially biased, information in patients with more health system contact.³⁶ Both results highlight that one should exercise caution in defining the population to which the EOL-CI is applied. The result for adults of Hispanic ethnicity requires further investigation in larger cohorts with greater representation of this population. These findings highlight the importance of evaluating algorithmic fairness in predictive models, especially in underrepresented populations.

Although the observed performance of the EOL-CI is encouraging, our work highlights a fundamental problem based on the process used to develop the risk score. While the population used to derive the EOL-CI was large, it reflected community-dwelling populations with generally low 1-year mortality risk (<2% to 3%). Epic utilized a down-sampling technique to artificially produce a 1-year mortality rate of 5%. Data augmentation through down-sampling, over-sampling, or techniques such as synthetic minority oversampling³⁷ is thought to provide gains in the ability to uncover discriminatory features. However, evaluations of these techniques have shown that this is often not the case.^38,39 Additionally, the use of these sampling approaches complicates the estimation of absolute risk, making calibration difficult, if not impossible. This contributes to the limitations of proprietary, non-transparent AI models, which impact broader governance considerations. While best practice recommendations for the integration and monitoring of proprietary models in clinical settings call for reproducibility, the lack of knowledge of the model’s source code hamper transparent reporting guidelines.^13,40 Epic is aware of this issue, as their documentation indicates that the EOL-CI is not an estimated probability and will over-estimate risk if it is treated as such, consistent with the results presented here. Despite this acknowledgment, we do not think it is appropriate to then ignore calibration with the EOL-CI, given that this can often be the most important measure of performance in clinical situations.⁴¹

There are important trade-offs regarding the clinical utility of the EOL-CI with respect to prognosis and calibration. On one hand, if the EOL-CI were used to prompt a low-resource intervention, such as automated outreach to schedule a provider visit or prompt telehealth interventions for ACP,^42,43 calibration might be less critical. Even if a patient lives for several more years, initiating an ACP discussion earlier could be seen as a proactive approach within the continuum of care.⁴⁴ Our data certainly indicate an opportunity for such interventions to further increase the occurrence of ACP discussions,⁴⁵ given the current low rate of these encounters within our health system. However, a key component of these discussions is assessing the patient’s disease understanding and prognostic awareness.⁴⁶ Clinicians, who are often inaccurate in their prognostic estimates, frequently rely on general or disease-specific mortality risk tools to guide conversations and support shared decision-making.¹⁶ In these cases, accuracy and the proper quantification of uncertainty are essential. Unfortunately, since it was not designed to provide an estimate of mortality risk, the EOL-CI may not be the best tool for this purpose. Clinicians might need to switch to a properly calibrated prognostic index, which is likely not integrated into the EHR, to provide a more accurate prognosis.

The issue becomes more critical when considering the allocation of limited human resources, especially given the ongoing shortage in the palliative care workforce.⁴⁷ If the EOL-CI were used as the default trigger for palliative care consultations in our health system, it could overwhelm services by prioritizing a population in which the majority would not die within the year. While palliative care is not exclusively for those nearing the end of life, as it can support symptom management and goals of care at any stage of serious illness, health systems must ensure they allocate resources effectively. The EOL-CI’s default thresholds would likely strain both inpatient and outpatient palliative care programs in our system, which already operate at capacity. A promising aspect of our results is that we were able to locally re-calibrate the EOL-CI, which could address this concern and provide the benefit of direct estimates of prognosis. Additionally, we improved calibration by incorporating data on past healthcare utilization and a measure of frailty. This observation is not surprising, as both factors are known to be strongly associated with mortality.^48,49 Accounting for frailty and comorbidity has also been shown to improve performance of predictive models for life expectancy.⁵⁰ While these modifications should be prospectively evaluated, including thorough evaluation for fairness, they suggest a potential to improve the accuracy and clinical applicability of the EOL-CI through straightforward local modifications. This reinforces the importance of adaptability and context-specific refinements in pragmatic real-world implementation, whether a risk tool is developed through statistical approaches or leverages artificial intelligence.¹⁸

There are limitations with our work. First, the external validation was conducted within a single academic health system, which may limit generalizability. We have planned future validation efforts across our entire health system, Advocate Health, which spans 6 states and serves over 6 million patients. Second, while our cohort was diverse, certain populations were not well-represented. Third, while we assessed algorithmic fairness across sex, race, and ethnicity, other factors such as socioeconomic status, access to care, or geographic measures of social disadvantage were not evaluated. Future research should explore these dimensions of fairness in our augmented and recalibrated model and compare the EOL-CI to other externally validated mortality prediction models.¹⁰ An important aspect of these evaluations will be to determine whether the inclusion of healthcare utilization as an input has unintended consequences with respect to health equity.⁵¹ Fourth, another potential limitation is the risk of data drift. Given the dynamic nature of healthcare, evaluations to mitigate reduced model performance over time must be implemented to align with the changing landscape of clinical workflows and population demographics. Finally, we evaluated the EOL-CI as a risk tool for an ambulatory, community-dwelling population. A future direction is to assess the EOL-CI’s utility as a trigger in specific clinical contexts,⁵² such as during inpatient hospitalization or before major surgery.

Conclusion

Our external validation of the EOL-CI highlights its strengths in predicting 1-year mortality, demonstrating reasonable discrimination. However, challenges remain in clinical application due to the need for local calibration to quantify and accurately estimate mortality risk over 1 year. By incorporating local data for calibration, the EOL-CI has the potential to more reliably guide decision-making and resource allocation, supporting targeted interventions in serious illness care. These findings emphasize the importance of locally validating and adapting predictive models, as well as continuously evaluating them within regulatory and governance frameworks to ensure optimal performance across diverse clinical settings.^12,13

Acknowledgments

The authors would like to thank Andrew McWilliams, MD, for reviewing a previous draft of this work.

Author contributions

Erica Frechman (Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Writing—original draft, Writing—review & editing), Byron Jaeger (Data curation, Formal analysis, Methodology, Validation, Writing—original draft, Writing—review & editing), Marc A. Kowalkowski (Writing—review & editing), Jeff Williamson (Conceptualization, Writing—review & editing), Kristin Macfarlane Lenoir(Writing—review & editing), Jessica A. Palakshappa (Writing—review & editing), Brian J Wells (Writing—review & editing), Kathryn Callahan (Writing—review & editing), and Nicholas Matthew Pajewski (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Writing—original draft, Writing—review & editing), and Jennifer L. Gabbard (Conceptualization, Investigation, Methodology, Resources, Writing—original draft, Writing—review & editing)

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was funded by the National Institute on Aging (NIA) of the National Institutes of Health (NIH) under Award Number U54AG063546, which funds the NIA Imbedded Pragmatic Alzheimer's Disease and AD-Related Dementias Clinical Trials Collaboratory (NIA IMPACT Collaboratory). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The authors gratefully acknowledge the use of data extraction services by the Office of Informatics, funded by the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health, through Grant Award Number UL1TR004929. J.L.G. was supported by NIA/NIH under Award Number K23AG070234 and J.A.P. by NIH/NIA under Award Number K23AG073529. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Conflicts of interest

The authors have no competing interests to declare.

Data availability

The data underlying this article cannot be shared publicly due to being protected electronic health record data. Researchers interested in collaborative research should contact the corresponding author.

References

Author notes