Artificial Intelligence Decision Support for Triple-Negative Breast Cancers on Ultrasound

Abstract

Objective

To assess performance of an artificial intelligence (AI) decision support software in assessing and recommending biopsy of triple-negative breast cancers (TNBCs) on US.

Methods

Retrospective institutional review board–approved review identified patients diagnosed with TNBC after US-guided biopsy between 2009 and 2019. Artificial intelligence output for TNBCs on diagnostic US included lesion features (shape, orientation) and likelihood of malignancy category (benign, probably benign, suspicious, and probably malignant). Artificial intelligence true positive was defined as suspicious or probably malignant and AI false negative (FN) as benign or probably benign. Artificial intelligence and radiologist lesion feature agreement, AI and radiologist sensitivity and FN rate (FNR), and features associated with AI FNs were determined using Wilcoxon rank-sum test, Fisher’s exact test, chi-square test of independence, and kappa statistics.

Results

The study included 332 patients with 345 TNBCs. Artificial intelligence and radiologists demonstrated moderate agreement for lesion shape and orientation (k = 0.48 and k = 0.47, each P <.001). On the set of examinations using 6 earlier diagnostic US, radiologists recommended biopsy of 339/345 lesions (sensitivity 98.3%, FNR 1.7%), and AI recommended biopsy of 333/345 lesions (sensitivity 96.5%, FNR 3.5%), including 6/6 radiologist FNs. On the set of examinations using immediate prebiopsy diagnostic US, AI recommended biopsy of 331/345 lesions (sensitivity 95.9%, FNR 4.1%). Artificial intelligence FNs were more frequently oval (q < 0.001), parallel (q < 0.001), circumscribed (q = 0.04), and complex cystic and solid (q = 0.006).

Conclusion

Artificial intelligence accurately recommended biopsies for 96% to 97% of TNBCs on US and may assist radiologists in classifying these lesions, which often demonstrate benign sonographic features.

Introduction

Artificial intelligence (AI) is a rapidly evolving field in radiology with increasing data demonstrating its ability to enhance image quality, increase interpretation accuracy, and improve time and cost efficiency in breast imaging. Nearly 3 decades after computer-aided diagnosis software first emerged to assist with interpretation of digital mammography, additional techniques such as deep learning and convolutional neural networks have further expanded AI applications to improve the detection and classification of breast lesions depicted on mammography, US, and breast MRI (1–4).

Koios Decision Support (DS) for Breast (Koios Medical Inc, New York, New York) is a proprietary, Food and Drug Administration–cleared AI software platform developed to assist radiologists in characterizing breast lesions on US. Using machine learning, this AI system generates a Breast Imaging Reporting and Data System (BI-RADS)–aligned probability of malignancy category for a breast lesion indicated by a region of interest (ROI) manually drawn on the static US image by the radiologist. Use of the system by radiologists has been shown to significantly improve interpretation accuracy, increase cancer detection rates, and reduce false-positive biopsies, both retrospectively and prospectively (5–7).

A potential application for this AI system is in recognizing the need to biopsy masses that have relatively benign features on US but may be triple-negative breast cancers (TNBCs). Triple-negative breast cancers represent a biologically aggressive subtype of breast cancer that lack hormone receptors, rendering them insensitive to hormonal therapy and resulting in poorer prognosis (8–10). TNBCs can be challenging to identify on US because of the frequent presence of relatively benign morphologic features with oval or round shape and circumscribed margins reported in up to 67% and 57% of cases, respectively, which can potentially delay diagnosis (11–21). In a meta-analysis of 620 patients with TNBC, suspicious sonographic features including irregular shape, noncircumscribed margins, nonparallel orientation, and posterior shadowing were often absent and demonstrated relatively low sensitivity (range 14%–68%) and specificity (range 19%–66%) (22). Few studies have highlighted the potential role for AI to assist radiologists in classifying these TNBC lesions on US using machine learning, deep convolutional neural networks, and radiomics (23–27).

The purpose of our study was to evaluate the diagnostic performance of the AI DS system in appropriately recommending biopsy of TNBCs on US. Additionally, we compared the AI-assessed lesion features and the AI diagnostic performance to that of the radiologist.

Methods

Institutional review board approval was obtained for this retrospective study, for which written informed consent was waived. The study was conducted in compliance with the Health Insurance Portability and Accountability Act.

Patient and lesion characteristics

Retrospective review of our radiology department database identified 375 consecutive patients who underwent US-guided core biopsy between July 2009 and July 2019 with pathology yielding TNBC at biopsy or surgical excision performed within 30 days of biopsy. Triple-negative breast cancer was defined as a tumor negative for estrogen (<1%), progesterone (<1%), and human epidermal growth factor 2 hormone receptors (0 or +1) as documented on pathology report. Human epidermal growth factor 2 status was determined by immunohistochemistry testing for all patients, and none required fluorescence in situ hybridization amplification.

For this study, 2 board-certified radiologists with fellowship training in breast imaging (2 years and 1 year of breast imaging experience, respectively) reviewed medical records and imaging studies to obtain patient and lesion characteristics and confirm radiology-pathology correlation. Patient characteristics included age, breast cancer risk factors, presence of clinical breast symptoms, presence of axillary adenopathy on clinical exam or imaging performed before US, and whether the patient was within 1 year postpartum or breastfeeding. Lesion characteristics including size, shape, and orientation were obtained directly from the US report or determined by the study radiologist if missing from the report. Any additional imaging features documented in the US report (margin, echotexture, posterior features, and vascularity) were also recorded. Concurrent mammography and/or other relevant imaging (MRI, PET/CT, and CT) were reviewed if available. Any available prior US examination was also reviewed to determine whether the TNBC lesion had been initially assessed by a radiologist on a prior US as BI-RADS 2 or 3. For these cases, the initial clinical history, BI-RADS assessment, and imaging features were recorded.

US examination and biopsy technique

All breast US examinations in this study were diagnostic and included grayscale and color Doppler images obtained with a handheld 11- to 15-MHz transducer. At our cancer center, US examinations were performed by certified breast imaging US technologists using a linear transducer (LOGIQ E9, GE Healthcare, Chicago, IL; ACUSON, Siemens Healthineers, Erlangen, Germany). For diagnostic US examinations performed at outside facilities and submitted to our cancer center for second opinions, various US vendor systems were used, and the operator experience was unknown. All US examinations in this study were interpreted by 1 of 17 breast imaging radiologists at our cancer center. For examinations performed at our facility, these were the standard clinical interpretations, and for examinations performed at outside facilities, these were second-opinion interpretations. Of the 17 breast imaging radiologists, 13 had received fellowship training in breast imaging, and the remaining 4 had over 25 years of experience interpreting breast imaging. All US-guided breast biopsies were performed with 12- to 14-gauge core needles to obtain 2 to 6 samples.

Artificial intelligence decision support system

For the purposes of this study, study author Brianna Aukland analyzed each TNBC lesion using the AI DS system (Koios DS for Breast, version 2.1.0, Koios Medical Inc, New York, NY) on the prebiopsy diagnostic US examination. Artificial intelligence analysis was performed by drawing an ROI around the lesion in 2 orthogonal views on the grayscale US images (Figure 1). Artificial intelligence system output was recorded, which included lesion features (round, oval, or irregular shape; parallel or not parallel orientation) and a BI-RADS–aligned categorical likelihood of malignancy (benign, probably benign, suspicious, or probably malignant). The AI likelihood of malignancy was generated from a machine-learning algorithm trained on over 450 000 breast images from 25 machines and 25 health care sites using pathology or 1-year imaging follow-up as ground truth. The 4 categories (benign, probably benign, suspicious, or probably malignant) are separated by 3 operating points that align with or exceed the sensitivity and specificity of radiologist-chosen BI-RADS categorization (2, 3, 4A–B, or 4C–5, respectively) (5). If the TNBC lesion was initially assigned BI-RADS 2 or 3 by a radiologist on a prior US, AI analysis was also performed on the initial US.

Outcome

For the AI system, a true positive (TP) result was defined as an output of suspicious or probably malignant, indicating a biopsy recommendation, and a false negative (FN) result was defined as an output of benign or probably benign, indicating no biopsy recommendation. For radiologists, TP results included BI-RADS 4 or 5 assessments, and FN results included BI-RADS 2 or 3 assessments. Sensitivity and FN rate (FNR) were calculated for both AI and radiologists using output from the immediate prebiopsy diagnostic US. If a lesion was initially assigned BI-RADS 2 or 3 by a radiologist on a prior US, sensitivities and FNRs were also separately calculated using output from the earlier US for these patients and immediate prebiopsy US examinations for the patients without a prior US. There were no true negative or false positive results in this study because all patients had cancer.

Exclusion criteria

After retrospective review of 375 cases, 43 patients were excluded (Figure 2). These included patients referred from another institution for whom digital images from the diagnostic prebiopsy US were unavailable (31/43), patients with a tumor that was too large for inclusion on a single US image to enable the ROI to be accurately drawn (8/43), patients who had a stereotactic or MRI-guided biopsy yielding TNBC performed the same day as the US-guided biopsy that did not yield TNBC (3/43), and 1 patient in whom the TNBC lesion was an axillary node (1/43). More than 1 TNBC was diagnosed in 13 patients. A total of 332 patients with 345 cancers constituted the study sample.

Statistical analysis

Statistical analysis was performed using R 4.2 (R Foundation for Statistical Computing, Vienna, Austria). The Wilcoxon rank-sum test, Fisher’s exact test, and Chi-square test of independence were used to compare descriptive characteristics between TP and FN. False discovery rate correction was performed for multiple testing comparison to yield adjusted P-values (q-values) with a result of ≤.05 indicating statistical significance. Test sensitivity was calculated with 95% confidence interval (95% CI). Interrater agreement between radiologist and AI assessments for lesion shape and orientation was determined using kappa statistics.

Results

Table 1 presents the characteristics of the 332 patients with 345 TNBCs (13 patients had 2 TNBC lesions). All patients were women, with a median age of 57 years (interquartile range [IQR] 47–67), and the characteristics were not significantly different between those with TP and FN AI lesions. The majority of patients (226/332, 68%) had no known breast cancer risk factor, and only 9/332 (2.7%) were known BRCA mutation carriers. Of the 106/332 (32%) women who had a risk factor, personal history of breast cancer was the most common (72/106, 68%). For 321/332 (97%) women in this study, the TNBC was identified on a diagnostic US performed to evaluate either a clinical breast symptom in 189/321 (59%) or a nonpalpable finding detected on another imaging modality in 132/321 (41%), including mammography, breast MRI, chest MRI, PET, and/or CT. Of the 189 symptomatic women, 89 (47%) also had axillary adenopathy that was either palpable on physical exam or evident on imaging performed before US. For the remaining 11/332 (3%) patients, the TNBC was initially detected on a screening US after a negative screening mammogram. Nearly all patients (305/332, 92%) had concurrent mammography, except for 4 young (under age 30) symptomatic patients and 23 patients with an MRI-detected lesion, all of whom proceeded directly to targeted US. Overall, 274/332 (83%) patients had their US performed and interpreted at our cancer center, while 58/332 (17%) had their US performed at another institution and submitted to our department for a second-opinion review. All 332 women had an immediate prebiopsy diagnostic US examination assigned BI-RADS 4 or 5, and 6/332 (1.8%) women had at least one prior earlier US examination assigned BI-RADS 2 or 3 depicting the index lesion.

Table 2 presents the lesion imaging features of the 345 TNBCs obtained from the final prebiopsy US reports. The US reports included lesion size and shape for 100% of lesions, whereas lesion orientation was missing for 111/345 (32%) lesions and was therefore retrospectively determined by the study radiologist. Median lesion size was 1.6 cm (IQR 1.1–2.6). The AI system–assessed lesion imaging features are also shown. Overall, moderate agreement was observed between AI and radiologist assessments for lesion shape (k = 0.48, P <.001) and orientation (k = 0.47, P <.001). Most TNBCs were irregular (263/345, 76% and 270/345, 78%) and not parallel (247/345, 72% and 195/345, 57%) by both radiologist and AI assessments, respectively. The majority of TNBCs were reported by the radiologist to have not circumscribed margins (321/345, 93%) and hypoechoic echotexture (284/345, 82%), features not included in the AI output. Of the 117 lesions that had posterior features described in the radiology report, 73 (62%) demonstrated posterior enhancement compared with 44 (38%) shadowing.

Six of 345 TNBCs were initially interpreted as BI-RADS 2 or 3 by a radiologist on an earlier diagnostic US, yielding a radiologist FNR of 1.7% and sensitivity of 98.3% (339/345, 95% CI: 96.3%–99.4%). The initial clinical and imaging features associated with these 6 radiologist FN lesions are presented in Table 3. Although the TNBC was palpable in 3/6 (50%) women, none were known to be at elevated risk for breast cancer. One palpable lesion was initially misclassified as BI-RADS 2 and subsequently biopsied for progressing clinical symptoms, including new nipple retraction. The other 5/6 lesions were initially misclassified as BI-RADS 3 and subsequently biopsied for interval growth (Figure 3). The median time between the earlier diagnostic US and immediate prebiopsy diagnostic US was 7.6 months (IQR 6.6–8.8).

Artificial intelligence analysis of the 345 TNBCs on the immediate prebiopsy diagnostic US accurately recommended biopsy for 331/345 (95.9%) lesions, with 123 (35.7%) categorized as probably malignant, 208 (60.3%) as suspicious, 10 (2.9%) as probably benign, and 4 as benign (1.2%). For the 6 lesions that were initially misclassified on the earlier diagnostic exams as BI-RADS 2 or 3 by the radiologist (radiologist FN), AI analysis of the earlier US accurately recommended biopsy for 6/6 (100%) lesions. Interestingly, 2/6 lesions initially categorized as suspicious by AI (AI TP) on the earlier diagnostic US examinations were later misclassified as benign or probably benign (AI FN) on the immediate prebiopsy diagnostic US examinations. For the set of US examinations using the 6 with earlier diagnostic US, the AI sensitivity was 96.5% (333/345, 95% CI: 94.0%–98.2%) with 12/345 FN lesions (FNR 3.5%). For the set of US examinations using the immediate prebiopsy diagnostic US, AI sensitivity was 95.9% (331/345, 95% CI: 93.3%–97.8%) with 14/345 FN lesions (FNR 4.1%).

Table 4 presents the clinical and imaging features associated with the 14 AI FN lesions on the immediate prebiopsy diagnostic exams (10/14 [71%] probably benign, 4/14 [29%] benign). Median lesion size was 1.45 cm (IQR 0.95–2.0), with 4/14 (29%) lesions measuring 2 cm or greater. Artificial intelligence false negativity was significantly associated with oval shape and parallel orientation by both AI and radiologist assessments (both q < 0.001) (Figure 4). All 14 (100%) FN lesions were considered parallel by both AI and radiologists. Radiologists described 4/14 (29%) of the AI FN lesions as circumscribed or complex cystic and solid, and both features were significantly associated with AI false negativity (q = 0.04 and q = 0.006, respectively). As shown in Table 1, AI false negativity was not significantly associated with any patient-level feature.

Five (1.5%) of 345 TNBCs initially yielded benign pathology on US-guided biopsy, of which 3/5 (60%) were subsequently surgically excised for high-risk pathology (atypia, papilloma, or squamous metaplasia) and 2/5 (40%) underwent repeat US-guided core biopsy. The AI system accurately categorized all 5 lesions as either probably malignant (3/5, 60%) or suspicious (2/5, 40%). Of these 5 lesions, one was a mammographically occult mass initially detected on screening US, whereas the other 4 lesions were identified on diagnostic US performed to evaluate a palpable abnormality (1 lesion) or a mammographic abnormality (3 lesions). Median size was 1.0 cm (IQR 1.0–1.7). Artificial intelligence and radiologists agreed on shape for 4/5 (80%) lesions (irregular) and orientation for 4/5 (80%) lesions (2 parallel, 2 not parallel).

Discussion

In this retrospective study of 345 TNBCs, an AI machine-learning DS platform demonstrated a 96% to 97% sensitivity in classifying lesions as suspicious or probably malignant on US, warranting a biopsy. At initial presentation on earlier diagnostic US examinations, radiologists demonstrated a 98% sensitivity, having initially misinterpreted 6/345 (1.7%) lesions as benign or probably benign. Although the AI and radiologist sensitivities are similar, the AI FNR using the earlier US and immediate prebiopsy US examinations were both ~4%, twice that of the 2% radiologist FNR. Artificial intelligence performance might be improved if it considered the presence of breast symptoms, axillary adenopathy, or suspicious findings on other imaging modalities, which accounted for 57%, 27%, and 40% of TNBCs in this study, respectively, and likely contributed to the radiologists’ decision to recommend biopsy. Triple-negative breast cancers have been reported to present symptomatically in the literature in up to 85% of cases (8,15,18,20). The majority (9/14, 64%) of AI FN lesions were palpable, including 2 masses that were also initially radiologist FN lesions and biopsied during follow-up for interval growth. Thus, our findings emphasize the importance of correlating US findings with clinical history and other modalities when relatively benign sonographic features are present. Other potential areas for system improvement include expanding lesion feature analysis (margins, echo pattern, and vascularity), enabling automated segmentation to reduce ROI variability, incorporating cine clips for 3D modeling, or comparing to prior US examinations to assess for interval change.

We found that AI was able to accurately identify all 6 radiologist FN lesions as suspicious on the earlier diagnostic US exams. These exams were performed a median of 7.6 months before the immediate prebiopsy US, and use of AI may have led to an earlier diagnosis in these 6 women, although further research would be needed to confirm this observation prospectively. An unexpected finding was that AI initially assessed 2/6 (33%) lesions as suspicious on the earlier diagnostic US and later assessed them as benign or probably benign on the immediate prebiopsy US. Given that AI-assessed lesion features (oval and parallel) did not change between the 2 US examinations, this class switching may reflect interoperator variability in US static image capture or ROI drawing, which are known limitations of this technology currently. However, one previous study using this AI software showed that ROI boundary variation had no significant impact on diagnostic performance, with minimal switching between suspicious and probably benign categories (5). Therefore, another possible explanation is that these TNBCs developed other, more benign features in the interval due to rapid cellular proliferation, which is believed to cause the regular shape and circumscribed margins associated with high-grade TNBCs, and that this subtle change was perceived by the AI on the prebiopsy US (10).

This sample of 345 TNBCs represents one of the largest in the US literature, with approximately one-quarter of lesions described by radiologists as having an oval or round shape (82/345, 24%) or parallel orientation (98/345, 28%), similar to frequencies reported in previous studies (11–17,19,21). The frequency of circumscribed margins, however, was only 7% in this study and lower than the 10% to 57% range reported in literature (11–15). This difference may reflect our larger cohort of TNBCs or perhaps, given the operator dependence of US, greater experience and/or expertise of the technologists and radiologists at our cancer center. Not surprisingly, AI false negativity was significantly associated with these benign sonographic features (parallel and oval q < 0.001, circumscribed q =.04), underscoring the diagnostic challenge for both AI and radiologists in recognizing these lesions as suspicious. Moreover, 14/14 (100%) of the AI FN lesions were parallel by both AI and radiologist assessments, which was significantly higher than the frequency in the TNBC cohort overall (28% by radiologist, 43% by AI), suggesting that parallel orientation may be a less reliable predictor of benignity, which warrants further research. Finally, complex cystic and solid echo pattern was the only suspicious sonographic feature that was significantly associated with AI false negativity (q = 0.006). Artificial intelligence underperformance within this subset of lesions highlights a potential area for system improvement that may be confirmed in a future study of benign cysts and benign and malignant complex cystic and solid masses.

Our findings support further prospective study of the AI system as a diagnostic aid for radiologists in clinical practice, specifically in the evaluation of masses with benign-appearing sonographic features (oval, parallel, and circumscribed). The main limitation of this study was the absence of benign lesions in the cohort, precluding assessment of AI false positive rate and specificity, which are important metrics for radiologists to know when using the AI tool. A previous study that prospectively evaluated this AI system on all breast lesions encountered in a large radiology practice demonstrated improved radiologist performance with AI DS, specifically, increased positive predictive value and decreased benign biopsy rate (6). A similar prospective study that includes both benign and malignant lesions with benign-appearing sonographic features could be performed to further investigate our results and better evaluate radiologist performance with and without the AI system in real time. Of note, the AI system may be helpful in low-resource clinical settings, for instance, in practices that do not have subspecialized breast US technologists or breast radiologists. Additionally, our finding that AI accurately categorized all 5 FN benign biopsies as probably malignant or suspicious before excision or repeat biopsy suggests another potential application for this AI tool in predicting surgical upgrades of high-risk lesions or determining concordance after benign biopsy, although further research is warranted.

Our study has limitations. The major limitations of this study are the retrospective design and inclusion of only malignant TNBC lesions, which limits assessment of the full spectrum of diagnostic performance of the AI system and the breast imaging radiologists. Secondly, the AI output is reliant on both the static US images that are selectively obtained by the individual scanning the patient as well as the ROIs drawn around the lesion by the radiologist; both tasks are operator-dependent and may introduce variation. This is especially true for the examinations obtained at other facilities, for which the US technique and operator experience was unknown and real-time scanning was not available to the interpreting radiologist. Thirdly, the AI system training determined benignity based on 1-year follow-up instead of 2 years. Finally, any cases that underwent surgical excisional biopsy or palpation-guided biopsy instead of initial US-guided biopsy were excluded from our study.

Conclusion

Artificial intelligence accurately identified 96% to 97% of TNBC lesions as suspicious on US and may assist radiologists in determining the need to biopsy such lesions when benign sonographic features are present. The sensitivity of the AI system was poorest among TNBCs that were oval, parallel, circumscribed, and complex cystic and solid, emphasizing the challenge for both AI and radiologists in evaluating tumors with relatively benign sonographic features. Further study into the prospective use of AI DS is warranted.

Acknowledgments

The authors thank Joanne Chin, MFA, ELS for her help in editing this manuscript.

Funding

None declared.

Conflict of interest statement

V.L.M. discloses research support from Pfizer unrelated to this study. The remaining authors report no funding or potential conflict of interest.

References