A machine learning algorithm-based risk prediction score for in-hospital/30-day mortality after adult cardiac surgery

Abstract

OBJECTIVES

A study of the performance of in-hospital/30-day mortality risk prediction models using an alternative machine learning algorithm (XGBoost) in adults undergoing cardiac surgery.

METHODS

Retrospective analyses of prospectively routinely collected data on adult patients undergoing cardiac surgery in the UK from January 2012 to March 2019. Data were temporally split 70:30 into training and validation subsets. Independent mortality prediction models were created using sequential backward floating selection starting with 61 variables. Assessments of discrimination, calibration, and clinical utility of the resultant XGBoost model with 23 variables were then conducted.

RESULTS

A total of 224,318 adults underwent cardiac surgery during the study period with a 2.76% (N = 6,100) mortality. In the testing cohort, there was good discrimination (area under the receiver operator curve 0.846, F1 0.277) and calibration (especially in high-risk patients). Decision curve analysis showed XGBoost-23 had a net benefit till a threshold probability of 60%. The most important variables were the type of operation, age, creatinine clearance, urgency of the procedure and the New York Heart Association score.

CONCLUSIONS

Feature-selected XGBoost showed good discrimination, calibration and clinical benefit when predicting mortality post-cardiac surgery. Prospective external validation of a XGBoost-derived model performance is warranted.

Open in new tabDownload slide

INTRODUCTION

Prediction models are used in international guidelines [1, 2] to determine the most appropriate treatments and enable clinicians to counsel patients. With reductions in mortality, the emergence of minimally invasive surgery and interventional procedures, updated and more accurate models are needed. Accurate predictions also help in benchmarking individual surgical and institutional results. Older models, such as the logistic European System for Cardiac Operative Risk Evaluation (EuroSCORE) [3] and EuroSCOREII [4], need to be updated. The primary limitation of previous models is poor calibration with over-estimation of risk in the highest risk group [5–10]. Their performance varies by era, centre [4, 11–13] and inherent procedural risk [5, 14, 15], and the application of risk-adjusted mortality ratios is recommended [4, 12].

Current models utilize logistic regression (LR). Complex interactions may exist between the input variables and would need to be accounted for by the model developer. Alternative machine learning (ML) models use algorithms to interpret these interactions using large volumes of data. We have previously shown that alternative ML-based models had statistically better discrimination and clinical utility than a retrained-LR model but similar calibration when using the EuroSCOREII [11] variables. The most consistent improvement in performance was with a XGBoost-based model.

Herein, we have built upon our previous work by utilising all available variables routinely collected in the UK National Adult Cardiac Surgery Audit (NACSA). It is the largest dataset reviewed and utilizes an exhaustive variable selection approach in order to create a novel XGBoost-based model.

METHODS

Ethics statement

This register-based cohort study is part of research approved by the Health Research Authority and Health and Care Research Wales and a need for patients consent was waived (IRAS ID: 257758, 23.7.2019). Reporting of results follows the TRIPOD statement.

Data extraction

A complete extract of prospectively collected data from the NACSA was obtained from the National Institute of Cardiovascular Outcomes Research (NICOR), containing data on all adults undergoing cardiac surgery in England and Wales between January 2012 and March 2019. Patients undergoing transplant, congenital surgery or isolated mechanical support insertion were excluded. The outcome measure was death within the same admission or 30 days of the index operation. Cases where the outcome measure was missing were excluded. Data processing and imputation of missing data were as previously described using R v4.0.2 [16]: ‘age and weight were imputed using the median and the absence of other variables was presumed to be an absence of that risk factor, a method previously established by NACSA’ [17, 18]. Assumptions and pre-processing of data were in line with previously published NACSA guidelines [11, 17, 18]. Missing data were <1%.

The variables routinely collected by NACSA were reviewed by 2 clinicians and a data analyst. Based on published literature, clinical experience and degree of missingness of the data 71 variables were initially reviewed. Those variables representing singular episodes were removed. The variables for pre-existing cardiac rhythm and the need for mechanical support were combined as shown in Supplementary Material, Fig. S1. Ultimately 61 variables were utilized (Supplementary Material, Fig. S1). These variables are defined in the NACSA database v5.1.0 [19].

Statistical analysis

Categorical variables were summarized as counts and percentages. Continuous variables were summarized as mean and standard deviation.

Model development

Both models were developed with Python [20] Scikit-learn v0.23.1 and Keras v2.4.0. Categorical variables’ values were converted to either binary values for dichotomous categories or the ordinal scale for multi-category variables. For reasons stated previously [11], we temporally split the data—70% of records (1 January 2012 to 31 December 2016) for training/validation and 30% (1 January 2017 to 31 March 2019) for testing. Data standardization was performed by subtracting variable mean and dividing by the standard deviation.

The XGBoost model is a combination of multiple short decision tree-based predictions. At each iteration, the performance of the model and the difference from the observed outcome is assessed. A weak decision tree model is then generated to predict this residual. This weaker tree is weighted by its predictive accuracy and added to the pre-existing model. Following multiple iterations, a final model is developed and used for onward assessment. Thus, it is an ensemble model that does not rely solely on one tree. It is a form of gradient boosting that incorporates a regularization term and cross-validation. The combination of these features in such a large dataset as ours reduces the risk of over-fitting. The final hyperparameters are listed in Supplementary Material, Table S2.The XGBoost-based model was further refined using a wrapper-based greedy algorithm for feature selection called sequential backward floating selection, using the mlxtend [21, 22] package in python. A model was developed using all 61 variables. Thereafter, sequentially one variable is removed at a time and the performance of the model assessed, in this case by measuring the area under the receiver operator curve (AUC). The ‘floating’ component allows the inclusion of variables even after the initial exclusion phase so long as the final combination of variables has an improved overall performance. This continues till the optimal combination of variables is reached.

Model validation and evaluation were conducted as before [11]. The primary outcomes were model discrimination (AUC, F1 score), calibration [visual inspection of a plot of observed versus expected outcomes and quantification of the observed minus expected outcomes (O-E), expected calibration error (ECE)], Brier score and clinical utility [decision curve analysis (DCA)]. 5-fold cross-validation was applied for evaluation of the training/validation set for AUC, F1, O-E and net benefit (NB) with the corresponding 95% confidence intervals (CIs) calculated using bootstrap-t sampling (n = 100) with replacement. The latter emphasizes the true positives and penalizes a high type II error rate. The calculation and interpretation of these are stated in our previous study [11]. An ideal model has a high AUC and F1 score, be on the bisector of the observed-to-expected curve, have a high adjusted ECE/Brier score (1−ECE and 1−Brier) and a high overall NB (i.e. the higher the curve on the DCA plot the better the clinical performance).

Variable importance

The SHapley Adaptive exPlanations (SHAP) method [23] was used to understand variable importance in the final model. SHAP values [24] are derived from game theory and assign an importance value to each variable. In the final model, a prediction is made for an individual in the presence or absence of a given variable. The difference between the predictions is the marginal contribution of the variable to the prediction. Differing combinations of variables are utilized and averaged to determine the final SHAP value of a given variable and hence its importance to the final model.

RESULTS

Demographics

A total of 224 318 adults underwent cardiac surgery across 42 centres during the study period. There were 6100 deaths (2.72%) (Fig. 1). Baseline differences in the variables between survivors and non-survivors are shown in Table 1 and Supplementary Material, Table S1.

Final model

The final model consisted of the variables in Table 2.

XGBoost-23

The XGBoost model with the greatest training set AUC had 27 variables—AUC 0.837 (95% CI: 0.837–0.838) and F1 score 0.277 (95% CI: 0.276–0.279) (Supplementary Material, Table S3). However, visual inspection of the AUC curve (Supplementary Material, Fig. S4) and output values (Supplementary Material, Table S3) showed that there was minimal loss in discrimination with less variables. Hence, after discussion and in the interest of parsimony, a ‘sensitivity’ analysis was performed to assess the alteration in model performance with a lower number of variables (20, 23, and 25) (Supplementary Material, Table S3). There was a marked drop in NB (treated) when going from 23 to 20 variables. There was risk over-estimation at 20 variables and under-estimation at 23 and 25 variables, with minimal differences in clinical utility between the 23 and 25 variable models. Good performance was achieved in terms of AUC, F1 score, Adjusted Brier Score, Adjusted ECE score and clinical NB when benchmarked against EuroSCORE II [25, 26] (Supplementary Material, Table S3B). On balance we opted to move forward with a XGBoost model with 23 variables (i.e. XGBoost-23)—AUC 0.846 (95% CI: 0.845–0.846) and F1 score 0.288 (95% CI: 0.287–0.290) (Fig. 2A), which showed good calibration even with those at high predicted risk (Fig. 2B). The number of patients in each risk centile, as predicted by XGBoost-23, are shown in Table 3 and Supplementary Material, Fig. S5. There are very few patients with a predicted risk above 30% that have undergone surgery, as would be expected. The DCA (Fig. 2C) showed an NB in those treated at all threshold probabilities below 60%.

Variable importance

Variable importance of the XGBoost-23 model conducted on the training/validation set is shown in Fig. 3 and Supplementary Material, Table S4.The most influential factors are the type of operation, age, creatinine clearance, urgency and New York Heart Association scores.

DISCUSSION

This study is an example of how using a feature-selected gradient boosting ML algorithm (XGBoost) can produce a mortality risk prediction model with good discrimination, calibration and clinical utility. This is the largest national study and builds on our previous work [11].

We began with 61 variables and used automated feature selection to develop the final model. It was developed with a view to optimize discrimination and indeed had not only a higher AUC and F1 score compared to EuroSCORE II and previous studies but also showed improved calibration and clinical utility [26]. It showed good calibration across all deciles of risk. Most patients had risks less than 10%, as would be expected of those undergoing surgery. However, in the higher-risk patients, XGBoost-23 continued to perform well. DCA showed that with probability thresholds under 60% there is a clear NB when using XGBoost-23 suggesting patients would be appropriately operated upon if using this model. Improved performance in the few high-risk patients becomes important when considering operations on patients with few alternatives (e.g. aortic dissections) or in units routinely operating on high-risk cases. It would be interesting to review model performance in those turned down for surgery but this data is not routinely collected.

Notably, we have not sought any information that is not already routinely gathered though many additional risk factors have been proposed previously [27, 28]. We also rationalized the final XGBoost model in the name of parsimony and to encourage its use by clinicians going forward.

No matter which modelling tool is utilized the relationship between patient, perioperative factors and death is inherently complex. Alternative ML models, such as XGBoost, offer the benefit of requiring little input from the model developer and having innate additive properties such that additional information can be used to adapt models without re-running the algorithm from the beginning. However, it is hard to depict as a standard formula and is less trusted by some. With the best performing model (XGBoost-23) we have described variable importance, which should also aid in greater understanding and acceptance in the wider surgical community. We are developing a website to host the model and allow prospective evaluation by others. LR has the advantage that it is well understood by the scientific community and publishing of coefficients allows external validation. However, modelling of interactions and non-linear relationships between the explanatory and response variables will be needed to optimize a LR model. The addition of new results would also require re-running the model development process.

Ensemble tree based, particularly gradient boosting, algorithms have been used with increasing frequency recently. Ours is the largest database studied, with only 3 other papers having more than 10 000 patients [30–32]. Most studies have shown marginal improvements in model performance with the use of alternative ML algorithms compared to commonly used models and retrained-LR models [31]. Interestingly Ong et al. [13] also showed improvements in discrimination when training models on local data. Although intuitive it reinforces the need to use either local variables or, as we have done, use hospital codes. Previous reviews have shown variable model performance on the basis of the hospital [4, 11–13] and operation undertaken [5, 14, 15]. Though the team behind the EuroSCORE advocated using hospital risk-adjusted mortality ratios, we felt that the hospital code could be incorporated into the final model from the outset and would increase ease of use. Zeng et al. [29] also performed a very thorough comparative study of different gradient boosting algorithms with a variety of feature selection methods but found only minor differences in performance between them and a retrained-LR model. Our study is unique in performing DCA in addition to discrimination and calibration. We feel this is crucial as improved clinical utility is the ultimate aim for prediction modelling.

Some variables are similar to those already taken into account in published scoring systems–age, creatinine clearance, New York Heart Association, urgency, critical pre-operative state, previous surgery, left ventricular function, peripheral vascular disease and pulmonary disease. Others have been replaced with similar variables of greater granularity, such as all diabetes management strategies, types of operations performed and distinguishing between mitral valve repair and replacement. New risk factors include the presence of a previous myocardial infarction (MI) not just recent MI, number of grafts inserted, pre-operative cardiac rhythm, first operator grade, basal metabolic index, pre-operative stroke, aortic arch procedure and need for mechanical support. Furthermore, differences in risk between those with previous valve surgery and previous coronary artery bypass surgery are also accounted for.

With the advent of more artificial intelligence-based search engines, there is increased public awareness and acceptance of these processes. We believe this should extend to the cardiac surgical community. With increasing server capacity and ultimately prospective data analysis we hope to be able to assess and fine tune this model further. The importance of risk prediction for morbidities (e.g. stroke, MI), resource utilization (e.g. length of stay, cost of intervention [33]) and interventional procedures [34] also has clear benefits for patient counselling and resource planning. Any future model utilized needs to be accurate, easy to use, avoid over-fitting and be recalibrated periodically.

Limitations and future work

The current model was created using a historic database. With the advent of more interventional strategies and the interruption of services secondary to the pandemic we should explore temporal differences in model performance. Prospective analysis is warranted before routine adoption of any model. Mortality following cardiac surgery in the UK is low [35] and strategies for managing class imbalance are needed. Future work will also include the development of an optimized LR-based model in order to perform a comparative analysis.

CONCLUSIONS

A feature-selected XGBoost-based model for 30 day or in-hospital mortality in adult cardiac surgery showed good discrimination, calibration and clinical benefit. Prospective assessment of a XGBoost-derived model in an external dataset is warranted.

SUPPLEMENTARY MATERIAL

Supplementary material is available at EJCTS online.

FUNDING

This study was supported by a British Heart Foundation-Turing research grant HF-Turing-19_2_1020.AJ was supported by the National Institute for Health and Care Research Bristol Biomedical Research Centre, University Hospitals Bristol and Weston NHS Foundation Trust and University of Bristol(NIHR203315).

Conflict of interest: none declared.

DATA AVAILABILITY

The data underlying this article were provided by NICOR/HQIP under licence/by permission. Data will be shared on request to the corresponding author with permission of NICOR/HQIP.

ETHICS STATEMENT

This register-based cohort study is part of research approved by the Health Research Authority (HRA) and Health and Care Research Wales and a need for patients consent was waived (HCRW)(IRAS ID: 257758, 23.7.2019).

Author contributions

Shubhra Sinha: Conceptualization; Data curation; Formal analysis; Methodology; Supervision; Validation; Writing—original draft; Writing—review & editing. Tim Dong: Conceptualization; Formal analysis; Methodology; Software; Visualization; Writing—review & editing. Arnaldo Dimagli: Conceptualization; Methodology; Writing—review & editing. Andrew Judge: Supervision; Writing—review & editing. Gianni D. Angelini: Funding acquisition; Methodology; Project administration; Supervision; Writing—review & editing.

Reviewer information

Reviewer information European Journal of Cardio-Thoracic Surgery thanks the anonymous reviewers for their contribution to the peer review process of this article.

REFERENCES

ABBREVATIONS

 
• AUC

  Area under the receiver operator curve

 
• CI

  Confidence interval

 
• DCA

  Decision curve analysis

 
• ECE

  Expected calibration error

 
• EuroSCORE

  European System for Cardiac Operative Risk Evaluation

 
• LR

  Logistic regression

 
• MI

  Myocardial infarction

 
• ML

  Machine learning

 
• NACSA

  National Adult Cardiac Surgery Audit

 
• NB

  Net benefit

 
• SHAP

  SHapley Adaptive exPlanations

 
• XGBoost

  Extreme gradient boosting machine

Author notes