https://orcid.org/0000-0001-6385-604X
Abstract
There are limited data on the application of artificial intelligence (AI) on nonenriched, real-world screening mammograms. This work aims to evaluate the ability of AI to detect false negative cancers not detected at the time of screening when reviewed by the radiologist alone.
A commercially available AI algorithm was retrospectively applied to patients undergoing screening full-field digital mammography (FFDM) or digital breast tomosynthesis (DBT) at a single institution from 2010 to 2019. Ground truth was established based on 1-year follow-up data. Descriptive statistics were performed with attention focused on AI detection of false negative cancers within these subsets.
A total of 26 694 FFDM and 3183 DBT examinations were analyzed. Artificial intelligence was able to detect 7/13 false negative cancers (54%) in the FFDM cohort and 4/10 (40%) in the DBT cohort on the preceding screening mammogram that was interpreted as negative by the radiologist. Of these, 4 in the FFDM cohort and 4 in the DBT cohort were identified in breast densities of C or greater. False negative cancers detected by AI were predominantly luminal A invasive malignancies (9/11, 82%). Artificial intelligence was able to detect these false negative cancers a median time of 272 days sooner in the FFDM cohort and 248 days sooner in the DBT cohort compared to the radiologist.
Artificial intelligence was able to detect cancers at the time of screening that were missed by the radiologist. Prospective studies are needed to evaluate the synergy of AI and the radiologist in real-world settings, especially on DBT examinations.
Many commercial artificial intelligence (AI) applications are available for improving cancer detection on screening mammography, but few have been studied on nonenriched datasets. This study demonstrates that AI may improve cancer detection rate by decreasing false negative cancers when applied to a heterogeneous, real-world U.S. screening population.
In this retrospective review of 26 694 full-field digital mammography (FFDM) and 3183 digital breast tomosynthesis (DBT) examinations, this commercial AI algorithm correctly identified 54% (7/13) of false negative cancers in the FFDM cohort and 40% (4/10) of false negative cancers in the DBT cohort on the preceding screening mammogram that was interpreted as negative by the radiologist.
The false negative cancers detected by AI were all invasive and were predominantly (82%, 9/11) luminal A subtype. Large prospective studies are needed to determine the clinical impact of detecting these cancers earlier with the aid of AI.
Introduction
Population-level mammographic screening for breast cancer reproducibly decreases the rate of cancer-related deaths in the United States.1 As with any cancer screening program, false negative cancers, which represent malignancies detected between routine screening examinations, remain a challenge. Multiple definitions exist depending on the screening interval utilized, but false negative cancers are most commonly defined as cancers diagnosed within 1 year of a negative screening mammogram.2 False negative cancers can be further subdivided into symptomatic (ie, interval cancers) vs asymptomatic (ie, detected via another screening modality), with symptomatic interval cancers portending a worse prognosis.3-5 According to the Breast Cancer Screening Consortium, a representative false negative rate is 0.8 per 1000 examinations. However, rates have been reported in the literature ranging from 0.8 to 2.1 per 1000 examinations.4,6 The goal of screening is early detection of malignancies, ideally before they are clinically apparent, to derive the greatest mortality benefit. Thus, minimizing false negative cancer rate is of utmost importance.
Several breast artificial intelligence (AI) applications with regulatory approval are commercially available, all promising to increase cancer detection rates and therefore decrease false negative cancers.7-10 However, whether AI can consistently deliver these benefits when applied to diverse real-world populations is largely unknown. Previous studies have demonstrated the potential of AI to reduce false negative cancer rates by 19% to 37% in full-field digital mammography (FFDM) but were performed using enriched datasets with up to 1 in 3 cases representing malignancies.10,11 A large population-based study in Europe demonstrated increased false negative cancer detection with AI; however, this study was also limited to FFDM within a biannual screening program.12,13 There are limited peer-reviewed data on the application of these AI systems to digital breast tomosynthesis (DBT) and their effect on false negative cancer detection in annual screening programs.14
The purpose of this study was to evaluate the ability of AI to detect false negative cancers when applied to a representative, real-world patient population on both FFDM and DBT screening examinations in an annual screening program.
Methods
Cohort definition
Subsets of screening FFDM and DBT studies acquired between December 2010 and December 2019 at a single institution were retrospectively analyzed in an institutional review board–approved, HIPAA-compliant project using a Food and Drug Administration–cleared AI system (Transpara v1.7.1, ScreenPoint Medical). FFDM and DBT screening mammography examinations acquired at our institution were selected from women who participated in the Athena Breast Health Network, an observational study involving women undergoing breast cancer screening at 5 medical centers.15 These women completed a self-reported survey on clinical risk factors for breast cancer. Cancer outcome data were provided as International Classification of Diseases, 10th Revision codes via a regional registry maintained by the study institution, spanning December 2010 to December 2020 and providing at least 1-year follow-up for all examinations. For the FFDM screening cohort, this work utilized an existing study sample published as a part of a prior study.16 For the DBT screening cohort, we performed a similar selection process to the FFDM cohort by randomly selecting false positive and true negative DBT examinations that reflect an abnormal interpretation rate of 10%. The DBT cohort included tomosynthesis and synthetic 2D images; these women did not receive FFDM. False negative cancers were defined as malignancies diagnosed within 12 months of a benign or negative screening examination, including patients who were both symptomatic and asymptomatic (ie, detected on another imaging study).
Exclusion criteria and omitted data
Following guidance from the vendor regarding the use of the AI algorithm, patients with implants, >4 standard mammographic views, and self-reported prior history of breast cancer were excluded from the analysis. For the 3D cohort, the original tomosynthesis scans were encoded in a proprietary format that cannot be analyzed by the AI algorithm. We attempted to convert these studies from a secondary capture object into a breast tomosynthesis object using a conversion tool provided by the AI developer for research purposes. In all, 1909 examinations (34% of the DBT cohort) could not be successfully converted into the readable format and had to be omitted. The AI system was applied to all examinations that were not excluded based on the aforementioned criteria. An additional subset of 2D (n = 1598) and DBT (n = 76) examinations did not return an AI score for uncertain reasons and were omitted from the analysis.
Artificial intelligence algorithm
The commercially available AI system assigned screening examinations a malignancy risk score of 1 to 10. Per AI system vendor specifications, scores of 1 to 7 are considered negative examinations, with a reported negative predictive value of 99.97%. Scores of 10 are considered elevated risk because algorithm testing by the vendor demonstrated up to 87% of cases with malignancy were scored 10 by the AI system.17-19 Scores of 8 and 9 are considered intermediate risk, although the overall cancer rate in this category is low, approximately 6 in 1000 cases, similar to the general screening population. For examinations with a score of 8, 9, or 10, the AI system also flags the areas of concern on the mammogram, including specific slices for DBT examinations. For the purposes of our analysis of false negative cancers, malignancy risk scores of 8, 9, and 10 were considered a positive study identified by the AI algorithm given the fact that specific findings were flagged for examinations with these scores. For the DBT examinations, the AI system processed the tomosynthesis images only and not the synthetic 2D images.
False negative cancer classification
Examinations of patients with subsequent false negative cancers that were determined to be positive by the AI software were retrospectively reviewed by a group of breast fellowship–trained radiologists (8 practicing radiologists with 3 to 24 years of experience). The initial screening mammogram marked by the AI system was compared with each postbiopsy mammogram. If the area flagged by the AI algorithm correlated with the site of the subsequent biopsy-proven malignancy, as denoted by an appropriately positioned biopsy clip and/or mass, the case was defined as an accurately localized true positive. Alternatively, if the AI finding did not represent the true malignancy as defined above, the case was defined as coincidental. This group of radiologists also conducted a review to determine whether the false negative cancer was identifiable on the preceding screening mammogram. Classifications of no signs, minimal signs, obvious signs, or occult were assigned to the studies by consensus.20,21 A classification of no signs represented a cancer that was not seen on the preceding screening mammogram but was seen on the diagnostic examination that led to the false negative cancer diagnosis (for example, if the patient presented with an area of palpable concern after the screening mammogram). An occult classification represented a diagnosed malignancy that was never seen on the preceding screening mammogram or the subsequent diagnostic examination.
Statistical analysis
Descriptive statistics of false negative cancers were performed using Excel (version 2311). The electronic medical record was also reviewed to collect patient age, body mass index (BMI), race, and mode of cancer detection (eg, lump, screening MRI, etc). Mammographic densities for each of the false negative cancers were tabulated per radiographic reporting. Pathology reports were also reviewed to determine tumor biology and hormone receptor status. Finally, the time to detection of false negative cancers was calculated by comparing the date of screening mammogram with subsequently performed diagnostic breast imaging.
Results
Demographics
A total of 184 935 mammographic examinations were identified via the Athena cohort as above, which included 121 753 FFDM and 55 762 DBT studies. Following exclusion and case selection criteria, as described in the Methods section and Figure 1, 26 694 FFDM examinations (mean ± SD age 58 ± 11 years and BMI 29 ± 7 kg/m2) and 3183 DBT examinations (mean ± SD age 55 ± 11 years and BMI 28 ± 6 kg/m2) were analyzed (Figure 1). For the FFDM cohort, 55% (14 629/26 694) of examinations were in patients who identified as White, 8% (2002/26 694) Black, 10% (2597/26 694) Hispanic, 9% (2510/26 694) Asian, and 11% (3000/26 694) other. For the DBT cohort, 59% (1871/3183) of examinations were in patients who identified as White, 7% (219/3183) Black, 7% (236/3183) Hispanic, 9% (282/3183) Asian, and 9% (288/3183) other. For the FFDM and DBT groups, respectively, 7% (1956/26 694) and 9% (286/3183) did not identify their race (Table 1). The self-reported race/ethnicity of the patients with the 23 false negative cancers described below were as follows: 13 White (57%), 4 Asian (17%), 3 Hispanic (13%), 2 Black (1%), and 1 other (0.4%).
Cohort Demographics
| Cohort characteristics | FFDM | DBT |
|---|---|---|
| Total women, n (%) | 20 409 (87) | 2977 (13) |
| Total examinations, n (%) | 26 694 (89) | 3183 (11) |
| Age, mean (SD), years | 58 (11) | 55 (11) |
| BMI, mean (SD), kg/m2 | 29 (7) | 28 (6) |
| Race/ethnicity, n (%) | ||
| White | 14 629 (55) | 1871 (59) |
| Black | 2002 (8) | 219 (7) |
| Hispanic | 2597 (10) | 236 (7) |
| Asian | 2510 (9) | 283 (9) |
| Other | 3000 (11) | 288 (17) |
| Missing | 1956 (7) | 286 (9) |
| Breast density, n (%) | ||
| Almost entirely fatty (A) | 2961 (11) | 312 (10) |
| Scattered fibroglandular (B) | 11 035 (41) | 1172 (37) |
| Heterogeneously dense (C) | 5602 (21) | 605 (19) |
| Extremely dense (D) | 1290 (5) | 150 (5) |
| Density not specified | 5806 (22) | 944 (30) |
| Cohort characteristics | FFDM | DBT |
|---|---|---|
| Total women, n (%) | 20 409 (87) | 2977 (13) |
| Total examinations, n (%) | 26 694 (89) | 3183 (11) |
| Age, mean (SD), years | 58 (11) | 55 (11) |
| BMI, mean (SD), kg/m2 | 29 (7) | 28 (6) |
| Race/ethnicity, n (%) | ||
| White | 14 629 (55) | 1871 (59) |
| Black | 2002 (8) | 219 (7) |
| Hispanic | 2597 (10) | 236 (7) |
| Asian | 2510 (9) | 283 (9) |
| Other | 3000 (11) | 288 (17) |
| Missing | 1956 (7) | 286 (9) |
| Breast density, n (%) | ||
| Almost entirely fatty (A) | 2961 (11) | 312 (10) |
| Scattered fibroglandular (B) | 11 035 (41) | 1172 (37) |
| Heterogeneously dense (C) | 5602 (21) | 605 (19) |
| Extremely dense (D) | 1290 (5) | 150 (5) |
| Density not specified | 5806 (22) | 944 (30) |
Abbreviations: BMI, body mass index; DBT, digital breast tomosynthesis; FFDM, full-field digital mammography.
Cohort Demographics
| Cohort characteristics | FFDM | DBT |
|---|---|---|
| Total women, n (%) | 20 409 (87) | 2977 (13) |
| Total examinations, n (%) | 26 694 (89) | 3183 (11) |
| Age, mean (SD), years | 58 (11) | 55 (11) |
| BMI, mean (SD), kg/m2 | 29 (7) | 28 (6) |
| Race/ethnicity, n (%) | ||
| White | 14 629 (55) | 1871 (59) |
| Black | 2002 (8) | 219 (7) |
| Hispanic | 2597 (10) | 236 (7) |
| Asian | 2510 (9) | 283 (9) |
| Other | 3000 (11) | 288 (17) |
| Missing | 1956 (7) | 286 (9) |
| Breast density, n (%) | ||
| Almost entirely fatty (A) | 2961 (11) | 312 (10) |
| Scattered fibroglandular (B) | 11 035 (41) | 1172 (37) |
| Heterogeneously dense (C) | 5602 (21) | 605 (19) |
| Extremely dense (D) | 1290 (5) | 150 (5) |
| Density not specified | 5806 (22) | 944 (30) |
| Cohort characteristics | FFDM | DBT |
|---|---|---|
| Total women, n (%) | 20 409 (87) | 2977 (13) |
| Total examinations, n (%) | 26 694 (89) | 3183 (11) |
| Age, mean (SD), years | 58 (11) | 55 (11) |
| BMI, mean (SD), kg/m2 | 29 (7) | 28 (6) |
| Race/ethnicity, n (%) | ||
| White | 14 629 (55) | 1871 (59) |
| Black | 2002 (8) | 219 (7) |
| Hispanic | 2597 (10) | 236 (7) |
| Asian | 2510 (9) | 283 (9) |
| Other | 3000 (11) | 288 (17) |
| Missing | 1956 (7) | 286 (9) |
| Breast density, n (%) | ||
| Almost entirely fatty (A) | 2961 (11) | 312 (10) |
| Scattered fibroglandular (B) | 11 035 (41) | 1172 (37) |
| Heterogeneously dense (C) | 5602 (21) | 605 (19) |
| Extremely dense (D) | 1290 (5) | 150 (5) |
| Density not specified | 5806 (22) | 944 (30) |
Abbreviations: BMI, body mass index; DBT, digital breast tomosynthesis; FFDM, full-field digital mammography.
Flowchart of inclusions and exclusions for subjects and examinations used in the analysis. Abbreviations: AI, artificial intelligence; BTO, Breast Tomosynthesis Object; DICOM, Digital Imaging and Communications in Medicine; FN, false negative; FP, false positive; TN, true negative; TP, true positive.
Overall performance of AI algorithm
In the FFDM cohort of 26 694 examinations, the AI tool gave a positive score (defined for this study as a score of 8, 9, or 10) to 160 true positive results that contained cancer and 10 234 false positive results that did not contain a cancer. The AI tool gave a negative score (defined for this study as a score of 1–7) to 16 293 true negative results that did not contain a cancer and 7 false negative examination results that contained a cancer (Table 2).
Performance of AI Algorithm on Full-Field Digital Mammography Cohort
| Total N = 26 694 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 160 | False positive: 10 234 |
| AI score 1–7 | False negative: 7 | True negative: 16 293 | |
| Total N = 26 694 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 160 | False positive: 10 234 |
| AI score 1–7 | False negative: 7 | True negative: 16 293 | |
Overall performance of the AI tool based on examinations deemed positive vs negative by the AI tool. These numbers do not take into account “coincidental” cases for the examinations containing a cancer (ie, when the examination is marked as “positive” by the AI tool, but the AI tool marks an entirely different area separate from where the breast cancer was located).
Abbreviation: AI, artificial intelligence.
aAI score of 10 is considered “high risk,” an AI score of 8 or 9 is considered “intermediate risk,” and a score of 1 to 7 is considered “low risk.” In this study evaluating interval cancer detection, a score of 8 to 10 was considered a “positive” flag by the AI system.
Performance of AI Algorithm on Full-Field Digital Mammography Cohort
| Total N = 26 694 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 160 | False positive: 10 234 |
| AI score 1–7 | False negative: 7 | True negative: 16 293 | |
| Total N = 26 694 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 160 | False positive: 10 234 |
| AI score 1–7 | False negative: 7 | True negative: 16 293 | |
Overall performance of the AI tool based on examinations deemed positive vs negative by the AI tool. These numbers do not take into account “coincidental” cases for the examinations containing a cancer (ie, when the examination is marked as “positive” by the AI tool, but the AI tool marks an entirely different area separate from where the breast cancer was located).
Abbreviation: AI, artificial intelligence.
aAI score of 10 is considered “high risk,” an AI score of 8 or 9 is considered “intermediate risk,” and a score of 1 to 7 is considered “low risk.” In this study evaluating interval cancer detection, a score of 8 to 10 was considered a “positive” flag by the AI system.
In the DBT cohort of 3183 examinations, the AI tool gave a positive score to 50 true positive results that contained a cancer and 882 false positive results that did not contain a cancer. The AI tool gave a negative score to 2244 true negative examination results that did not contain a cancer and 7 false negative examination results that contained a cancer (Table 3).
Performance of AI Algorithm on Digital Breast Tomosynthesis Cohort
| Total N = 3183 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 50 | False positive: 882 |
| AI score 1–7 | False negative: 7 | True negative: 2244 | |
| Total N = 3183 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 50 | False positive: 882 |
| AI score 1–7 | False negative: 7 | True negative: 2244 | |
Overall performance of the AI tool based on examinations deemed positive vs negative by the AI tool. These numbers do not take into account “coincidental” cases for the examinations containing a cancer (ie, when the examination is marked as “positive” by the AI tool, but the AI tool marks an entirely different area separate from where the breast cancer was located).
Abbreviation: AI, artificial intelligence.
aAI score of 10 is considered “high risk,” an AI score of 8 or 9 is considered “intermediate risk,” and a score of 1-7 is considered “low risk.” In this study evaluating interval cancer detection, a score of 8 to 10 was considered a “positive” flag by the AI system.
Performance of AI Algorithm on Digital Breast Tomosynthesis Cohort
| Total N = 3183 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 50 | False positive: 882 |
| AI score 1–7 | False negative: 7 | True negative: 2244 | |
| Total N = 3183 | Breast cancer | ||
|---|---|---|---|
| Yes | No | ||
| AI malignancy risk score | AI score 8–10a | True positive: 50 | False positive: 882 |
| AI score 1–7 | False negative: 7 | True negative: 2244 | |
Overall performance of the AI tool based on examinations deemed positive vs negative by the AI tool. These numbers do not take into account “coincidental” cases for the examinations containing a cancer (ie, when the examination is marked as “positive” by the AI tool, but the AI tool marks an entirely different area separate from where the breast cancer was located).
Abbreviation: AI, artificial intelligence.
aAI score of 10 is considered “high risk,” an AI score of 8 or 9 is considered “intermediate risk,” and a score of 1-7 is considered “low risk.” In this study evaluating interval cancer detection, a score of 8 to 10 was considered a “positive” flag by the AI system.
The overall performance of the AI algorithm, as outlined in Tables 2 and 3, does not take into account lesion-specific analysis. Therefore, it is likely that some of these “true positive” cases were coincidental, meaning the AI tool scored the examination as positive but flagged an area in the breast that did not correlate to the area of the cancer.
Full-field digital mammography false negative and screen-detected cancers
The FFDM cohort contained 167 cancers: 13 false negative cancers and 154 screen-detected cancers (125 invasive and 42 in situ). Of these, 157 total malignancies were in examinations marked as positive (score of 8, 9, or 10) by the AI system, including 13 false negative cancers (100%, 13/13) and 144 screen-detected cancers (94%, 144/154). Although all 13 false negative cancers were given a positive AI score of 8 to 10, 6 of these AI-flagged FFDM interval cancer cases were coincidental, and 7 were accurately localized true positive results (Figures 2–4). Thus, the AI system accurately localized 54% (7/13) of the false negative cancers in this cohort. Of these 7 accurately localized false negative cancers, 5 demonstrated minimal signs on the preceding screening mammogram, 1 demonstrated obvious signs, and 1 was occult. Of the 6 coincidental cases, 2 were flagged by AI in the contralateral breast, 2 were flagged on the correct side but an incorrect quadrant, and 2 cases had malignancies that were outside the field of view on the screening mammogram. Of the 4 coincidental cases with malignancies within the mammographic field of view, 2 cases demonstrated obvious signs, 1 demonstrated minimal signs, and 1 was occult.
Example of artificial intelligence (AI) accurately localized true positive on a full-field digital mammography screening examination. The AI system gave this examination a high-risk score of 10 and correctly identified the location of the malignancy on the screening mammogram that was read as BI-RADS category 2 (benign) by the radiologist. A: Initial screening examination with superimposed AI markings. B: Postbiopsy mammogram confirming malignancy in the same location marked by the AI system. Red markings were reproduced by the authors based on the AI output and do not represent actual AI markings from the vendor.
Example of artificial intelligence (AI) accurately localized true positive on a full-field digital mammography screening examination. The AI system gave this examination a high-risk score of 10 and correctly identified the location of the malignancy on the screening examination that was read as BI-RADS category 2 (benign) by the radiologist. A: Initial screening examination with superimposed AI markings. B: Postbiopsy mammogram confirming malignancy in the same location marked by the AI system. Red markings were reproduced by the authors based on the AI output and do not represent actual AI markings from the vendor.
Example of an artificial intelligence (AI) “coincidental” positive on a full-field digital mammography screening examination. The AI system gave this examination an intermediate score of 8 but did not mark the correct location of the subsequently diagnosed interval cancer. The study was read as negative (BI-RADS category 2) by the radiologist. A: Initial screening examination left craniocaudal (CC) view with superimposed AI marking in the lateral breast. B: Subsequent mammogram demonstrating that the false negative cancer was outside of field of view on initial screening and located in the medial breast. Orange marking was reproduced by the authors based on the AI output and does not represent actual AI markings from the vendor.
Digital breast tomosynthesis false negative and screen-detected cancers
The DBT cohort contained 57 cancers: 10 false negative cancers and 47 screen-detected cancers (50 invasive and 7 in situ). Of these, 50 total malignancies were in examinations marked as positive by the AI system, including 5 false negative cancers (50%, 5/10) and 45 screen-detected cancers (96%, 45/47). Of the 5 AI-flagged DBT false negative cancer cases, 4 cases were accurately localized, and 1 was coincidental; thus, the AI system correctly flagged 40% (4/10) of false negative cancers in the DBT cohort. Of the 4 accurately localized false negative cancers, 3 demonstrated minimal signs on the preceding screening mammogram, and 1 demonstrated obvious signs (Figures 5 and 6). The single DBT AI coincidental case flagged the incorrect site in the ipsilateral breast to a mammographically occult malignancy. Of the 5 false negative cancers not identified by the AI system, 1 was mammographically occult (not visible on mammography at any time), 1 had no mammographic signs (ie, not visible at screening but visible on subsequent diagnostic mammogram), 2 had minimal signs, and 1 demonstrated obvious signs.
Example of artificial intelligence (AI) accurately localized true positive on a digital breast tomosynthesis screening examination. The AI system gave this examination a high-risk score of 10 and correctly identified the location of the malignancy on the screening examination that was read as BI-RADS category 2 (benign) by the radiologist. A: Initial screening examination with superimposed AI markings. B: Postbiopsy mammogram confirming malignancy in the same location marked by AI. Orange and red markings were reproduced by the authors based on the AI output and do not represent actual AI markings from the vendor.
Example of artificial intelligence (AI) accurately localized true positive on a digital breast tomosynthesis screening examination. The AI system gave this examination a high-risk score of 10 and correctly identified the location of the malignancy on the screening examination that was read as BI-RADS category 2 (benign) by the radiologist. A: Initial screening examination with superimposed AI markings. B: Postbiopsy mammogram confirming malignancy in the same location marked by AI. Red markings were reproduced by the authors based on the AI output and do not represent actual AI markings from the vendor.
Overall, including the total of 23 false negative cancers in both cohorts (13 in the FFDM cohort and 10 in the DBT cohort), the AI algorithm correctly identified the location of the false negative cancer on the preceding screening mammogram 48% (11/23) of the time. However, the AI algorithm flagged (ie, AI examination score of 8, 9, or 10) 78% (18/23) of these screening mammograms.
False negative cancer characteristics
The overall interval cancer rate (FFDM and DBT) in our cohort was 0.08%, or 0.8 per 1000 women screened. Twenty-two of the 23 false negative cancers (96%) were symptomatic, and 1 was asymptomatic and diagnosed on a screening MRI. This single asymptomatic false negative cancer was in the FFDM cohort; it was of the luminal A subtype and accurately localized by AI.
All of the false negative cancers accurately localized by AI were invasive, and the majority were luminal A subtypes (82%, 9/11) identified in scattered and heterogeneously dense breasts (Table 4). In the FFDM cohort, 100% (7/7) of the false negative cancers accurately localized by AI were ER+, 4 were PR+, none were HER2+, and 2 demonstrated a Ki-67 ≥20%. In the DBT cohort, 100% (4/4) of the false negative cancers accurately localized by AI were in heterogeneously dense breasts, 100% (4/4) were ER+/PR+, and none were HER2+ or with a Ki-67 ≥20%.
Demographics and Tumor Biology of False Negative Cancers
| FN cancer cohort | FFDM | DBT | ||
|---|---|---|---|---|
| Total FN cancers (n = 13) | FN cancers detected by AIa (n = 7) | Total FN cancers (n = 10) | FN cancers detected by AIa (n = 4) | |
| Invasive pathology, n (%) | 12 (92) | 7 (100) | 10 (100) | 4 (100) |
| Receptors present, n (%) | ||||
| ER+ | 11 (85) | 7 (100) | 7 (70) | 4 (100) |
| PR+ | 8 (62) | 4 (57) | 6 (60) | 4 (100) |
| HER2+ | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| KI-67 (≥20%), n (%) | 5 (38) | 2 (29) | 4 (40) | 0 (0) |
| Cohort characteristics | ||||
| Age, mean (SD), years | 57 (14) | 55 (12) | 64 (14) | 70 (7) |
| BMI, mean (SD), kg/m2 | 27 (8) | 29 (10) | 22 (7) | 23 (6) |
| Breast density, n (%) | ||||
| Almost entirely fatty (A) | 0 (0) | 0 (0) | 2 (20) | 0 (0) |
| Scattered fibroglandular (B) | 4 (31) | 3 (43) | 2 (20) | 0 (0) |
| Heterogeneously dense (C) | 7 (54) | 4 (57) | 5 (50) | 4 (100) |
| Extremely dense (D) | 2 (15) | 0 (0) | 1 (10) | 0 (0) |
| Days to FN detection, median (range) | 281 (11–350) | 272 (11–350) | 275 (41–351) | 248 (35–326) |
| FN cancer cohort | FFDM | DBT | ||
|---|---|---|---|---|
| Total FN cancers (n = 13) | FN cancers detected by AIa (n = 7) | Total FN cancers (n = 10) | FN cancers detected by AIa (n = 4) | |
| Invasive pathology, n (%) | 12 (92) | 7 (100) | 10 (100) | 4 (100) |
| Receptors present, n (%) | ||||
| ER+ | 11 (85) | 7 (100) | 7 (70) | 4 (100) |
| PR+ | 8 (62) | 4 (57) | 6 (60) | 4 (100) |
| HER2+ | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| KI-67 (≥20%), n (%) | 5 (38) | 2 (29) | 4 (40) | 0 (0) |
| Cohort characteristics | ||||
| Age, mean (SD), years | 57 (14) | 55 (12) | 64 (14) | 70 (7) |
| BMI, mean (SD), kg/m2 | 27 (8) | 29 (10) | 22 (7) | 23 (6) |
| Breast density, n (%) | ||||
| Almost entirely fatty (A) | 0 (0) | 0 (0) | 2 (20) | 0 (0) |
| Scattered fibroglandular (B) | 4 (31) | 3 (43) | 2 (20) | 0 (0) |
| Heterogeneously dense (C) | 7 (54) | 4 (57) | 5 (50) | 4 (100) |
| Extremely dense (D) | 2 (15) | 0 (0) | 1 (10) | 0 (0) |
| Days to FN detection, median (range) | 281 (11–350) | 272 (11–350) | 275 (41–351) | 248 (35–326) |
Abbreviations: AI, artificial intelligence; BMI, body mass index; DBT, digital breast tomosynthesis; FFDM, full-field digital mammography; FN, false negative.
aFalse negative cancers marked as positive and accurately localized by AI—that is, AI gave the examination a positive score (defined for this study as score of 8–10 on a 1–10 point scale) and correctly marked the location of the FN cancer on the preceding screening mammogram that was interpreted as negative by the radiologist.
Demographics and Tumor Biology of False Negative Cancers
| FN cancer cohort | FFDM | DBT | ||
|---|---|---|---|---|
| Total FN cancers (n = 13) | FN cancers detected by AIa (n = 7) | Total FN cancers (n = 10) | FN cancers detected by AIa (n = 4) | |
| Invasive pathology, n (%) | 12 (92) | 7 (100) | 10 (100) | 4 (100) |
| Receptors present, n (%) | ||||
| ER+ | 11 (85) | 7 (100) | 7 (70) | 4 (100) |
| PR+ | 8 (62) | 4 (57) | 6 (60) | 4 (100) |
| HER2+ | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| KI-67 (≥20%), n (%) | 5 (38) | 2 (29) | 4 (40) | 0 (0) |
| Cohort characteristics | ||||
| Age, mean (SD), years | 57 (14) | 55 (12) | 64 (14) | 70 (7) |
| BMI, mean (SD), kg/m2 | 27 (8) | 29 (10) | 22 (7) | 23 (6) |
| Breast density, n (%) | ||||
| Almost entirely fatty (A) | 0 (0) | 0 (0) | 2 (20) | 0 (0) |
| Scattered fibroglandular (B) | 4 (31) | 3 (43) | 2 (20) | 0 (0) |
| Heterogeneously dense (C) | 7 (54) | 4 (57) | 5 (50) | 4 (100) |
| Extremely dense (D) | 2 (15) | 0 (0) | 1 (10) | 0 (0) |
| Days to FN detection, median (range) | 281 (11–350) | 272 (11–350) | 275 (41–351) | 248 (35–326) |
| FN cancer cohort | FFDM | DBT | ||
|---|---|---|---|---|
| Total FN cancers (n = 13) | FN cancers detected by AIa (n = 7) | Total FN cancers (n = 10) | FN cancers detected by AIa (n = 4) | |
| Invasive pathology, n (%) | 12 (92) | 7 (100) | 10 (100) | 4 (100) |
| Receptors present, n (%) | ||||
| ER+ | 11 (85) | 7 (100) | 7 (70) | 4 (100) |
| PR+ | 8 (62) | 4 (57) | 6 (60) | 4 (100) |
| HER2+ | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| KI-67 (≥20%), n (%) | 5 (38) | 2 (29) | 4 (40) | 0 (0) |
| Cohort characteristics | ||||
| Age, mean (SD), years | 57 (14) | 55 (12) | 64 (14) | 70 (7) |
| BMI, mean (SD), kg/m2 | 27 (8) | 29 (10) | 22 (7) | 23 (6) |
| Breast density, n (%) | ||||
| Almost entirely fatty (A) | 0 (0) | 0 (0) | 2 (20) | 0 (0) |
| Scattered fibroglandular (B) | 4 (31) | 3 (43) | 2 (20) | 0 (0) |
| Heterogeneously dense (C) | 7 (54) | 4 (57) | 5 (50) | 4 (100) |
| Extremely dense (D) | 2 (15) | 0 (0) | 1 (10) | 0 (0) |
| Days to FN detection, median (range) | 281 (11–350) | 272 (11–350) | 275 (41–351) | 248 (35–326) |
Abbreviations: AI, artificial intelligence; BMI, body mass index; DBT, digital breast tomosynthesis; FFDM, full-field digital mammography; FN, false negative.
aFalse negative cancers marked as positive and accurately localized by AI—that is, AI gave the examination a positive score (defined for this study as score of 8–10 on a 1–10 point scale) and correctly marked the location of the FN cancer on the preceding screening mammogram that was interpreted as negative by the radiologist.
The false negative cancers that the AI system detected retrospectively at the time of screening were ultimately diagnosed by the radiologist after a median of 272 days (range 11–350 days) for FFDM examinations and 248 days (range 35–326 days) for DBT examinations.
Discussion
Application of the studied AI algorithm to a real-world dataset demonstrates promise in earlier detection of false negative invasive cancers. In the FFDM cohort, AI flagged 94% of all examinations with malignancies and correctly identified the location of 54% of cancers not initially detected by radiologists at the time of screening (ie, false negative cancers). In the DBT cohort, AI flagged 88% of all examinations with malignancies and correctly identified the location of 40% of cancers not initially detected by radiologists at the time of screening. Specifically, AI flagged all 13 false negative cancer cases in the FFDM cohort, of which 7 were accurately localized where the AI system identified the correct location of the cancer, and the remaining 6 represented coincidental findings. In the DBT cohort, AI flagged 5/10 false negative cancer cases, including 4 that were accurately localized and 1 that was a coincidental finding. Of the AI accurately localized false negative cancers, 6 of the 7 FFDM cases and all 4 of the DBT cases had only minimal signs on mammography by consensus review.
Artificial intelligence demonstrates potential to improve detection of false negative cancers; however, this has primarily been studied using cohorts enriched with more breast cancers compared with the rate seen in the general population. Analyses of European cohorts found that AI can improve false negative cancer detection anywhere from 19% to 58%; however, these studies were performed on populations without case control or up to a 1 in 3 prevalence of cancer.7-9 In addition, these European studies did not include a lesion-specific analysis (that is, they considered all flagged cases to be detected cancers even if the AI tool marked the incorrect location), thereby overestimating the sensitivity of the AI algorithm. Our study represents a minimally enriched subset of patients with a roughly 1 in 200 cancer prevalence in the FFDM cohort and a 1 in 50 cancer prevalence in the DBT cohort. We conducted a lesion-specific analysis to exclude “coincidental” positive examinations and found that this AI tool accurately localized 48% (11/23) of false negative cancers overall, all invasive disease, with the majority in dense breasts.
Double reading has been shown to improve screening accuracy but may not always be feasible economically or with regard to practice workflow.22,23 Artificial intelligence has been proposed as a form of double reading that offers to improve accuracy. The effect on overall clinical burden and economic cost is not well established. Many works have demonstrated the ability of AI to improve cancer detection yet have similarly shown a negative effect on recall rate.8,9,24 Our study demonstrates the ability to improve cancer detection rates at the time of screening but does not entirely remove the role of the interpreting radiologist. Artificial intelligence coincidental cases—meaning that the examination was given a high AI score, but the AI system marked the wrong area (ie, not the area of the false negative cancer)—were not uncommon, found in approximately 20% of DBT and 50% of FFDM flagged studies. Two of the 7 total AI coincidental cases were occult on mammography based on retrospective radiologist review. However, the majority of the false negative cancer cases overall had minimal signs on the preceding screening mammography, which were identified by the AI system and not by the radiologist. It is also important to point out that the AI tool missed 6% (10/154) of the screen-detected cancers in the FFDM cohort and 4% (2/47) in the DBT cohort. The AI tool marked these examinations as negative, and it is unknown whether this negative endorsement by the AI system could have dissuaded the radiologist from calling back these examinations that contained cancers. Prospective studies are needed to determine whether the radiologist will listen to the AI tool when it is correct but disregard it when it is incorrect, thus maximizing cancer detection.
The false negative cancers detected by AI were all invasive disease and predominantly (82%) luminal A subtypes. Typically, false negative cancers—in particular those that are symptomatic—are associated with clinically worse outcomes and more aggressive tumor biologies when compared with those detected at screening.4,25-27 Our false negative cohort was predominantly (96%, 22/23) symptomatic, and yet the tumor biologies were predominantly hormone receptor–postive (ie, less aggressive subtypes: none demonstrating HER2+ status, only 1 triple-negative phenotype, and only 2 cases with increased Ki-67 >20%; Table 4). This contrast is likely due to the overall small sample size of false negative cancers to review because the AI tool did correctly identify the single triple-negative false negative cancer in our combined FFDM and DBT cohort. Interestingly, work by Lång et al demonstrated that AI reduced false negative cancers in patients with grave outcomes (defined as death or stage IV disease) by 23%.8 On the other hand, Vachon et al found that an AI tool could predict invasive cancers 2.5 to 5 years before screening examination but did not significantly predict false negative cancers in an FFDM cohort.28 Although our study demonstrated that at least 48% of the false negative cancers in our cohort could have been detected earlier with the application of the AI system, the mortality benefit is still unknown and may be attenuated by their overall favorable tumor biologies. More study is needed to evaluate the clinical value of earlier detection of false negative cancers by AI.
While this study focused on false negative cancer detection, it is important to consider the effect of AI tools on false positive results. In our study, the AI tool gave an intermediate (score 8 or 9) or high (score 10) malignancy risk score to 10 396 examinations in the FFDM cohort, which translated to 160 true positive results and 10 227 false positive results (38% of FFDM cohort). In the DBT cohort, there were 50 true positive results and 882 false positive results (28% of DBT cohort). These AI “false positive” results mean that the AI algorithm marked these examinations as increased risk, but they do not mean that all of these examinations would ultimately be called back by the radiologist. In fact, a prospective study in Europe using the same AI tool as our study demonstrated no significant change in recall rate with the addition of AI for breast cancer screening.29 However, different screening practices in Europe (all FFDM and double reading) affect the applicability of these results to the United States. Nonetheless, our results suggest a significant potential increased workload for the radiologist to review these flagged AI examinations. In addition, there is no guarantee that the radiologist would ultimately call back the examinations that did in fact contain a false negative cancer picked up by the AI tool. In particular, the “intermediate risk” category (AI score of 8 or 9) has a relatively low cancer rate (6 in 1000) such that the radiologist is expected to disregard many AI markings in this category in order to maintain an appropriate recall rate and thus could miss a false negative cancer in this category. Prospective studies need to examine the effect of AI on recall rate and radiologist workload.
Although this work expands upon the use of AI in a more real-world population, the study is limited by several factors. The study design is retrospective from a single institution. There is likely some element of selection bias given initial voluntary enrollment in the Athena Breast Health Network. Exclusion criteria of breast implants, women with >4 views, examinations that could not be successfully converted to the format that could be processed by the AI algorithm, and postsurgical changes related to a history of breast cancer remove a significant portion of screening patients, including those with increased risk factors. For example, women with a history of breast cancer will develop a contralateral breast cancer at a rate of 2% to 11%, which represents a 2- to 6-fold increase relative to the general population.30
The study is additionally limited by the small number of false negative cancers. However, the rate of false negative cancers within our combined FFDM and DBT cohorts (8 per 10 000) is similar to that of the general screening population and thus reflective of a real-world population. Nonetheless, larger population studies are needed to better understand the impact of AI on false negative cancer detection. Additionally, the tumor biology subtype detected by the AI algorithm (HER2−, low KI-67, luminal A) may have lesser clinical impact given excellent prognosis with these malignancies. Finally, while both our FFDM and DBT cohorts are >40% non-White, the majority are White, which limits generalizability to all races and ethnicities. It is critical that larger prospective studies are performed in diverse and underserved populations to ensure health equity with regard to AI algorithm performance in breast cancer screening.
Conclusion
This work demonstrates the potential for an AI tool to improve rates of false negative cancer detection, earlier than by radiologist review alone, in a U.S.-based screening population. Large prospective studies are needed to evaluate AI fidelity in combination with a radiologist in a real-world clinical setting and to determine the clinical impact of earlier detection of false negative cancers flagged by an AI system.
Acknowledgments
The authors thank Elizabeth Catalan for her support with data organization.
Funding
W.H. and A.H. are co–Principal Investigators on a National Institutes of Health (NIH) R21 grant that contributed to this study (the Agency for Healthcare Research and Quality, R21 HS029257).
Conflict of interest statement
A.H. has medical industry–related stock holdings unrelated to the current manuscript that include Gilead, United Healthcare, Medtronic, and Abvie. A.H. is a prior stockholder in Hologic. W.H. receives consulting fees from the Radiological Society of North America. S.R.P, H.M., C.S., J.C., C.F., and M.J. have no conflicts of interest to disclose.
References
Author notes
Co-first author
