Derivation and validation of an artificial intelligence-based plaque burden safety cut-off for long-term acute coronary syndrome from coronary computed tomography angiography

Abstract

Aims

Artificial intelligence (AI) has enabled accurate and fast plaque quantification from coronary computed tomography angiography (CCTA). However, AI detects any coronary plaque in up to 97% of patients. To avoid overdiagnosis, a plaque burden safety cut-off for future coronary events is needed.

Methods and results

Percent atheroma volume (PAV) was quantified with AI-guided quantitative computed tomography in a blinded fashion. Safety cut-off derivation was performed in the Turku CCTA registry (Finland), and pre-defined as ≥90% sensitivity for acute coronary syndrome (ACS). External validation was performed in the Amsterdam CCTA registry (the Netherlands). In the derivation cohort, 100/2271 (4.4%) patients experienced ACS (median follow-up 6.9 years). A threshold of PAV ≥ 2.6% was derived with 90.0% sensitivity and negative predictive value (NPV) of 99.0%. In the validation cohort 27/568 (4.8%) experienced ACS (median follow-up 6.7 years) with PAV ≥ 2.6% showing 92.6% sensitivity and 99.0% NPV for ACS. In the derivation cohort, 45.2% of patients had PAV < 2.6 vs. 4.3% with PAV 0% (no plaque) (P < 0.001) (validation cohort: 34.3% PAV < 2.6 vs. 2.6% PAV 0%; P < 0.001). Patients with PAV ≥ 2.6% had higher adjusted ACS rates in the derivation [Hazard ratio (HR) 4.65, 95% confidence interval (CI) 2.33–9.28, P < 0.001] and validation cohort (HR 7.31, 95% CI 1.62–33.08, P = 0.010), respectively.

Conclusion

This study suggests that PAV up to 2.6% quantified by AI is associated with low-ACS risk in two independent patient cohorts. This cut-off may be helpful for clinical application of AI-guided CCTA analysis, which detects any plaque in up to 96–97% of patients.

Graphical Abstract

Open in new tabDownload slide

Introduction

Coronary computed tomography angiography (CCTA) has emerged as one of the first-line non-invasive imaging methods for patients with the suspicion of coronary artery disease (CAD).^1,2 Recent advances in CCTA using artificial intelligence (AI) have enabled accurate, automated, and fast plaque quantification and characterization.^3–6 Therefore, the derivation of robust quantitative plaque cut-offs, potentially useful to inform clinical decision making, may have become feasible. Plaque burden quantified by AI has also proved prognostic value with continuously increasing event risk with increasing plaque burden.^7,8 However, up to 97% of patients with a current indication for CCTA may present with any coronary plaque detected by AI.⁷ In order to avoid overdiagnosis, unnecessary downstream diagnostic testing, and implementation of advanced therapies in the broad range of patients undergoing CCTA, a clinically reasonable plaque burden safety cut-off for future coronary events is needed. Therefore, we aimed to derive and externally validate a plaque burden safety cut-off pre-defined to have ≥90% sensitivity for future acute coronary syndrome (ACS) among patients referred for CCTA using a previously validated AI-based plaque quantification and characterization tool with clearance from the US Food and Drug Administration.^3,4,9

Methods

Derivation cohort

For derivation, a consecutive cohort of 2411 symptomatic patients with suspected CAD having undergone CCTA at Turku University Hospital, Finland, from February 2007 to December 2016, was used. Out of these, 137 had non-retrievable CCTA images, and 3 patients were lost to follow-up, resulting in a final cohort of 2271 patients (Figure 1). Patients with previous coronary revascularization or known obstructive CAD were not considered for inclusion. Data on clinical characteristics, symptoms, and medication were retrospectively collected from electronic medical records.

External validation cohort

Seven hundred and forty-two consecutive patients undergoing clinically indicated PET-CT at Amsterdam University Medical Center, the Netherlands, from September 2008 to November 2014 were used for validation. Of these, 58 had non-retrievable CCTA images, 70 were lost to follow-up, and 46 patients were excluded due to prior myocardial infarction (MI), percutaneous coronary intervention (PCI), or coronary artery bypass grafting, resulting in a total of 568 patients (Figure 1).

Patient consent

The study complied with the Declaration of Helsinki. The Ethics Committees of the Hospital District of Southwest Finland and Amsterdam University Medical Center approved the study protocol and waived the need for informed consent.

Clinical event adjudication

Comprehensive data on outcomes were collected using electronic medical records, standardized telephonic follow-up, and national registry databases. The identified events were confirmed by the investigators. ACS included MI or unstable angina pectoris (uAP) in accordance with the latest recommendations.¹⁰

CCTA acquisition

CCTA imaging was performed as described previously.^11–13 In brief, patients were targeted at a stable heart rate targeted below 65 bpm (either spontaneous or after administration of oral and/or intravenous metoprolol). Sublingual/oral nitrate was administered immediately before CCTA. Intravenously administered low-osmolal iodine contrast agents were used. Prospectively triggered CCTA acquisition was applied whenever feasible. At Turku University Hospital, a 64-row hybrid PET-CT scanner (GE Discovery VCT or GE D690, GE Healthcare, Waukesha, WI, USA) was used. The CCTA scans were performed with a collimation of 64 × 0.625 mm and gantry rotation time of 350 ms. The tube current was 600–750 mA and voltage 100–120 kV according to patient size. Iterative CCTA reconstruction with standard Kernel recommended by the vendor was applied. At Amsterdam University Medical Center, a 64-row hybrid PET-CT scanner (Philips Gemini TF 64, Philips Healthcare, Best, The Netherlands) or a 256-slice CT scanner (Philips Brilliance iCT, Philips Healthcare, Best, The Netherlands) was used. The CCTA scans were performed with the collimation of 64 × 0.625 mm, a gantry rotation time of 420 ms, a tube voltage of 120 kV, and a tube current of 800–1000 mA. Iterative CCTA reconstruction with standard Kernel recommended by the vendor was used.

AI-guided quantitative computed tomography analysis

Artificial intelligence-guided quantitative computed tomography (AI-QCT) is a US Food and Drug Administration-cleared software service that uses a series of validated convolutional neural networks for image quality assessment, coronary segmentation and labelling, lumen wall evaluation, vessel contour determination, and plaque characterization. AI-QCT analysis was performed as described previously.^3,4 In brief, coronary segments with a diameter ≥1.5 mm were included in the analysis using the 18-segment Society of Cardiovascular Computed Tomography model.¹⁴ Plaque volumes (cubic millimeter) were calculated for each coronary lesion and then summated to compute the total plaque volume at the patient level. Coronary plaque volume was normalized to the total per-patient vessel volume to account for variation in coronary artery volume, calculated as plaque volume (mm³)/vessel volume (mm³) × 100%, referred to as percent atheroma volume (PAV) (per cent). Plaque types were categorized using Hounsfield unit (HU) ranges, with non-calcified plaque volume (NCPV) defined as plaques with any component on a pixel-level basis with HU between −30 and +350, and calcified plaque volume (CPV) defined as >350 HU. Low-density non-calcified plaque volume (LD-NCPV) was defined as HU between −30 and +30.¹⁵ Owing to the low volume of LD-NCPV detected in these cohorts, consistent with previous observations,⁷ total NCPV was used for the prognostic models with continuous plaque burden. The prognostic value of LD-NCPV was assessed together with the presence of positive remodelling (i.e. lesion diameter/reference diameter ≥1.1), referred to as two feature positive plaque (two FPP).¹⁵ AI-QCT diameter stenosis was categorized by the Coronary Artery Disease-Reporting and Data System (CAD-RADS)¹⁶ and ≥50% stenosis classified as obstructive disease.

Statistical analysis

Continuous variables are shown as mean ± standard deviation (SD) or median [IQR (25th–75th percentile)]. Categorical variables are shown as numbers with percentages. Mann–Whitney U test was used to compare continuous variables and two-sided χ² test was used for categorical variables. The association between PAV, NCPV, and CPV per 1% increase and ACS was assessed with multivariable Cox regressions. Adjusting covariates were chosen based on clinical reasoning and consisted of age, sex, hypertension, diabetes mellitus, smoking, dyslipidaemia, family history of CAD, typical angina, and early elective coronary revascularization within 6 months from CCTA. Variables with a significant association in univariable models (i.e. age, sex, hypertension, typical angina, and early revascularization) were used as adjusting covariates (see Supplementary data online, Table S1). We constructed four multivariable reference models: clinical characteristics (Reference 1), CAD-RADS + Clinical characteristics (Reference 2), AI-QCT diameter stenosis ≥50% + Clinical characteristics, (Reference 3), and two FPP + Clinical characteristics (Reference 4). PAV, NCPV, and CPV per 1% were added to the reference models and the models’ performance was assessed and compared using Harrell’s C. For the model with FPP, only NCPV (%) without LD-NCPV (%) was added on top, since LD-NCPV (%) is part of the two FPP definitions. The binary plaque burden cut-off to predict ACS was chosen from receiver operating characteristic (ROC) curve analysis to have a sensitivity of ≥90% for ACS in the derivation cohort. A threshold selection according to the method of Youden, which gives equal weight to sensitivity and specificity¹⁷ is not suitable for derivation of a safety cut-off for adverse patient outcome, where sensitivity must be emphasized.¹⁸ However, for conceptual validation and comparison to previous work,⁵ we also tested a threshold according to Youden in total plaque volume. We then calculated sensitivity, negative predictive value (NPV), specificity, positive predictive value (PPV), and area under the curve (AUC) for the binary plaque burden cut-off. We additionally assessed the test characteristics for ACS of the plaque burden cut-off at various timepoints during follow-up [1, 2, 3, 5, 6.9 (median follow-up), and 10 years]. Kaplan–Meier curves were created and compared with the log-rank test. Additionally, subgroup analyses for age (below vs. above median age of 63 years) and sex were performed. The derived binary plaque burden cut-off was then tested in the external validation cohort. Analyses were two-tailed and a P-value of <0.05 was considered statistically significant. All analyses were performed with Stata version 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC) and R version 4.3.2 (packages ‘survival’, ‘timeROC’, and ‘DTComPair’). S.B. and T.M. had full access to all the data in the study and take responsibility for its integrity and the data analysis.

Results

Patient characteristics and AI-QCT analysis in the derivation cohort

Hundred out of 2271 (4.4%) of patients experienced ACS (67 MI, 33 uAP) throughout a median follow-up of 6.9 (IQR 4.8–9.1) years. Patient characteristics and AI-QCT analysis of patients with vs. without ACS are reported in Table 1. Median time between CCTA and ACS was 1352 (IQR 502–2493) days. Overall, patients with vs. without ACS had higher plaque volumes and burden of all types (P < 0.001 for all) (Table 1).

Predictive value of plaque burden

The predictive value of continuous PAV, NCPV, and CPV per 1% increase on top of clinical characteristics (Reference 1), CAD-RADS (Reference 2), AI-QCT diameter stenosis ≥50%, (Reference 3), and two FPP (Reference 4) is shown in Table 2. For the model with two FPP, the added NCPV (%) did not include LD-NCPV (%), since this is already part of the two FPP definition. All plaque burden types remained independent predictors of ACS when added on top of all four reference models (P ≤ 0.006), except for NCPV (%) on top of two FPP (P = 0.093). According to the C-indexes, the addition of PAV or NCPV (%) significantly improved ACS prediction in all models (P ≤ 0.046), whereas CPV (%) did not further improve the prediction of ACS on top of CAD-RADS (P = 0.126) and AI-QCT diameter stenosis ≥50% (P = 0.324). The models including PAV showed the numerically highest C-indexes and performed in general similarly to models including only NCPV (%). Therefore, due to previously reported heterogeneity in NCPV (%) quantification across various software platforms (although predating AI-QCT),¹⁹ to improve generalizability of the study findings, PAV was chosen for cut-off derivation.

Cut-off selection for PAV

The AUC from ROC analysis of continuous PAV was 0.782 [95% confidence interval (CI) 0.743–0.823] (Figure 2). The cut-off with pre-defined ≥90% sensitivity for ACS was ≥2.6%. This cut-off resulted in a sensitivity of 90.0%, NPV of 99.0%, specificity of 46.8%, PPV of 7.2%, and AUC of 0.68 for ACS in the derivation cohort (Graphical Abstract, Table 3). Test characteristics of PAV 2.6% over time [at 1, 2, 3, 5, 6.9 (median follow-up), and 10] are shown in Supplementary data online, Table S2. A threshold selection according to the method of Youden in total plaque volume for comparison with previous work,⁵ resulted in 254.4 mm³. The test characteristics of this cut-off are shown in Supplementary data online, Table S3.

Predictive value of PAV 2.6% in the derivation cohort

Thousand hundred and forty-five out of 2271 patients (54.8%) had PAV ≥ 2.6% and 1026/2271 (45.2%) had PAV < 2.6%, which was markedly higher as compared with the 97/2271 (4.3%) patients with no plaque (PAV 0%) (P < 0.001). Ninety patients (7.2%) with PAV ≥ 2.6% as compared with 10 patients (1.0%) with PAV < 2.6% experienced ACS [Hazard ratio (HR) 7.68, 95% CI 4.00–14.77, P < 0.001] (Graphical Abstract, Figure 3, Table 4). This result was consistent after adjusting for clinical characteristics and early revascularization (HR 4.65, 95% CI 2.33–9.28, P < 0.001) (Table 4). Among the patients with PAV 0%, no ACS occurred. Subgroup analyses for age and sex showed consistent results (no interaction) with similar prognostic value of PAV 2.6% in women and men, however numerically higher prognostic value among patients of ≥63 years (median), even though confidence intervals (CI) were wide (see Supplementary data online, Figure S1).

Patient characteristics in the external validation cohort

In the external validation cohort, 27/568 (4.8%) of patients experienced ACS (19 MI, 8 uAP) throughout a median follow-up of 6.7 (IQR 4.7–7.8) years. Median time between CCTA and ACS was 282 (IQR 51–1473) days. Patient characteristics and AI-QCT analysis of patients with vs. without ACS are shown in Table 1.

External validation of PAV ≥ 2.6%

In the external validation cohort, 373/568 patients (65.7%) had PAV ≥ 2.6% and 195/568 (34.3%) had PAV < 2.6%, which was significantly higher as compared with the 15/568 (2.6%) patients with no plaque (PAV 0%) (P < 0.001). The diagnostic test characteristics of PAV ≥ 2.6% to detect ACS were sensitivity 92.6%, NPV 99.0%, specificity 35.7%, PPV 6.7%, and AUC 0.64 (Graphical Abstract, Table 3).

Twenty-five patients (6.7%) with PAV ≥ 2.6% as compared with 2 patients (1.0%) with PAV <2.6% experienced ACS (HR 7.94, 95% CI 1.87–33.62, P = 0.005) (Graphical Abstract, Figure 3, Table 4). This result remained consistent after adjusting for typical angina and early revascularization (HR 7.31, 95% CI 1.62–33.08, P = 0.010) (Table 4). Other variables were not included due to the limited numbers of events. No ACS occurred among patients with PAV 0%.

Sensitivity analyses

A sensitivity analysis excluding patients who underwent early elective revascularization showed consistent results (see Supplementary data online, Tables S3 and S4). Test characteristics and prognostic value of PAV 2.6% for MI alone as well as the composite of ACS or all-cause death are reported in Supplementary data online, Tables S5 and S6.

Discussion

In this observational cohort study on patients who underwent CCTA for suspected CAD, we assessed the association between AI-based plaque burden quantification and ACS, and aimed to derive and validate a plaque burden safety threshold for long-term ACS. We found that AI-QCT plaque burden showed a consistent and continuous association with ACS throughout a median follow-up of 7 years on top of clinical predictors, CAD-RADS, obstructive stenosis, and two FPP. A threshold of PAV 2.6% was derived to identify future ACS patients with 90.0% sensitivity in the derivation cohort. This cut-off showed a sensitivity of 92.6% in the validation cohort and excellent NPV of 99.0% in both the derivation and validation cohort. Importantly, 43% of the patients could be identified as having low long-term risk of ACS based on PAV < 2.6% in both cohorts, as compared with only 4% of patients showing absence of plaque (PAV 0%) based on AI-QCT. PAV ≥ 2.6% showed strong prognostic value with five- to seven-fold increased adjusted rates of ACS in both cohorts.

Plaque burden and the risk of ACS

Numerous prior studies have demonstrated the association between plaque burden on CCTA and ACS.^20–22 However, lesion-based studies have generally shown poor PPV for ACS, since a majority of vulnerable plaques will rupture silently without causing a direct ischaemic event for a given patient.²³ Consequently, the concept of the vulnerable patient has evolved.²⁴ Therefore, in this study, we aimed for a patient-level plaque burden cut-off that takes into account the totality of coronary atherosclerosis on CCTA. In line with the previous findings using major cardiovascular events as endpoint,⁷ we found independent prognostic value of plaque burden quantified by AI-QCT on top of clinical characteristics, CAD-RADS, obstructive stenosis, and two FPP. Among the different plaque types, PAV showed the numerically highest C-indexes and performed similarly to NCPV (%), whereas CPV (%) did not consistently improve event prediction. Therefore, and since NCPV (%) quantification was reported to vary across software platforms (although predating AI-QCT),¹⁹ to improve generalizability of the study findings, PAV was chosen for cut-off derivation.

Conventional clinical CCTA analysis is based on semi-quantitative measures of plaque burden and manual QCT has been available for many years, but remained restricted to research situations, because it is time-consuming. Technological developments during recent years with AI can finally make plaque quantification feasible and widely available for clinical use. Complete quantification and characterization of the whole coronary tree are now possible within ∼10 min, and together with final human quality control and report generation a total time of ∼25–30 min per scan are needed.⁴ Together with its potential to improve risk stratification,^7,8 as also evidenced in this study, and diagnostic accuracy comparable to the consensus of three Level III readers,⁴ AI-guided CCTA analysis represents a promising next step in CCTA technology. However, whereas conventional CCTA represents the current state-of-the art imaging modality to rule-out significant CAD,^1,2 AI-guided analysis detects any plaque (PAV > 0%) in up to 96–97% of patients.⁷ The clinical implications of these observations were unknown so far. Therefore, we aimed to derive and validate a clinically useful safety cut-off for AI-based PAV regarding long-term coronary events. However, our current study does not aim to prove superiority of AI- vs. non-AI-based CCTA analysis.

A binary plaque burden cut-off for ACS risk stratification

In a substudy of the SCOT-HEART trial (Scottish COmputed Tomography of the HEART Trial), a cut-off for low-attenuation plaque burden of 4% was identified, which was associated with 4.7-fold elevated adjusted risk of future MI throughout 5 years.²¹ More recently, using a deep-learning approach for coronary plaque quantification, a cut-off for total plaque volume of 238.5 mm³ was derived and shown to be associated with a 7.3-fold higher crude risk of MI.⁵ However, for both cut-off derivations, the method of Youden was applied, which is not suitable for ruling out adverse patient outcome, where more weight to sensitivity must be given.

In the current study, we aimed to determine a patient-level safety cut-off for AI-based plaque burden, to predict future ACS with a pre-defined sensitivity of ≥90%. This resulted in a cut-off of PAV 2.6%. Since we used this sensitivity-focussed approach, it was expected that the cut-off would be lower as previously suggested.²¹ For conceptual validation with previous work,⁵ we also tested a threshold according to Youden in total plaque volume resulting in a cut-off of 254.4 mm³, close to the Youden-based cut-off of 238.5 mm³ determined to detect MI with another AI-based plaque quantification tool.⁵ For our cut-off selection, to account for different plaque volumes across different patient sizes, also related to patients’ sex, we had chosen plaque burden normalized to the total vessel volume, instead of absolute plaque volume. Subgroup analyses yielded indeed similar progntaostic value among women vs. men, however markedly higher prognostic value for patients >63 years, potentially related to their higher event rate in general, although CI were wide and the result may be interpreted with caution.

Safety cut-off for ACS

The main aim of this cut-off derivation was to identify patients at low risk for ACS. Patients below the cut-off had indeed good prognosis with only 1% cumulative incidence of ACS over a median follow-up of 7 years. Forty-five per cent of patients in the derivation and 34% of patients in the validation cohort were classified at low risk according to this cut-off (PAV < 2.6%), whereas the vast majority of patients (96%) had any plaque detected by AI-QCT (PAV > 0%). Our analyses may thus suggest that a plaque burden up to the derived and validated threshold of PAV 2.6% is associated with low rates of ACS, and may therefore not justify intensified downstream testing or allocation to advanced atherosclerotic medication. This information may be helpful for the reasonable clinical application of AI-guided plaque analysis, which detects any plaque in up to 96–97% of patients. Furthermore, in relation to a previously validated plaque staging system based on PAV,⁷ it may allow for further subdivision of stage 1 (PAV > 0–5%) into patients with low event risk with PAV < 2.6% vs. those with relatively increased risk with PAV ≥ 2.6%.

Limitations

The results of this study must be considered in the light of several limitations. It represents an observational study with all the inherent limitations of a non-randomized investigation. However, this was in part mitigated by the use of two independent patient cohorts with large to moderate sample size and long follow-up. Data collection was performed in part retrospectively, however CCTA scans were re-analysed by AI-QCT in 2022–23 blinded to clinical data and outcomes, and events adjudicated appropriately. The CCTA scans were performed with relatively old CT scanners and newer generation scanners might provide better image quality. However, analysis of such older scans allowed for long-term follow-up, a key issue in the current analysis. For this analysis, the prescription rate based upon the CCTA findings of antiplatelet agents, statins, and other medication with potential outcome benefit were not available. However, for the derivation cohort from Turku, statin initiation and its association with clinical outcomes has been reported previously.²⁵ Data on risk factor management after CCTA (e.g. lipid levels) were not available. With the focus on mechanistic links between predictors and outcome, our study endpoint included only acute coronary events and no other events related to atherosclerosis. This ACS endpoint was limited to non-fatal events, since data on cardiac causes of death was not available. However, focussing on less severe (i.e. non-fatal) events could potentially have rendered an even lower (safer) threshold to classify patients at low risk for ACS. Furthermore, we have addressed this by performing a sensitivity analysis on the composite of ACS or all-cause death. The derivation and validation cohort differed with respect to the occurrence of ACS after CCTA with later occurrence in the Turku and earlier occurrence in the Amsterdam cohort. However, given the consistent main results of this study, this may even reinforce the robustness of the derived cut-off and its ability to safely rule-out ACS over mid- and long-term in varying cohorts. Whereas 99.9% of patients completed follow-up in the derivation cohort, 9% of patients were lost to follow-up in the external validation cohort. Patients underwent coronary revascularization as clinically indicated, which may have affected the outcome. However, we have accounted for this by adjusting the models for early revascularization and performing a sensitivity analysis excluding patients who underwent early revascularization, showing consistent findings with the main analysis.

Conclusions

This study suggests that a PAV up to 2.6% quantified by AI is associated with low-ACS risk in two independent patient cohorts. This PAV cut-off may be helpful for clinical application of AI-guided CCTA analysis, by classifying 43% of patients at low risk for ACS, whereas AI detects any plaque in up to 96–97% of patients.

Supplementary data

Supplementary data are available at European Heart Journal - Cardiovascular Imaging online.

Funding

The study was funded by the Finnish Foundation for Cardiovascular Research, Finnish State Research Funding (VTR 13403), and the Research Council of Finland. S.B. was supported by the Swiss National Science Foundation (P500PM_210788) and the Doctoral Programme for Clinical Research of the University of Turku, Finland. Cleerly Inc. performed AI-QCT analysis without costs.

Data availability

Data will be made available upon reasonable request from the corresponding author.

References

Author notes