A systematic review of automatic text summarization for biomedical literature and EHRs

Abstract

Objective

Biomedical text summarization helps biomedical information seekers avoid information overload by reducing the length of a document while preserving the contents’ essence. Our systematic review investigates the most recent biomedical text summarization researches on biomedical literature and electronic health records by analyzing their techniques, areas of application, and evaluation methods. We identify gaps and propose potential directions for future research.

Materials and Methods

This review followed the PRISMA methodology and replicated the approaches adopted by the previous systematic review published on the same topic. We searched 4 databases (PubMed, ACM Digital Library, Scopus, and Web of Science) from January 1, 2013 to April 8, 2021. Two reviewers independently screened title, abstract, and full-text for all retrieved articles. The conflicts were resolved by the third reviewer. The data extraction of the included articles was in 5 dimensions: input, purpose, output, method, and evaluation.

Results

Fifty-eight out of 7235 retrieved articles met the inclusion criteria. Thirty-nine systems used single-document biomedical research literature as their input, 17 systems were explicitly designed for clinical support, 47 systems generated extractive summaries, and 53 systems adopted hybrid methods combining computational linguistics, machine learning, and statistical approaches. As for the assessment, 51 studies conducted an intrinsic evaluation using predefined metrics.

Discussion and Conclusion

This study found that current biomedical text summarization systems have achieved good performance using hybrid methods. Studies on electronic health records summarization have been increasing compared to a previous survey. However, the majority of the works still focus on summarizing literature.

INTRODUCTION

Information overload in the biomedical and health sciences

Biomedical and healthcare information has been increasing exponentially. The unprecedented amount of diverse textual data leads to information overload.¹ Information overload interferes with information seekers’ information processing, diminishes their productivity, and prevents them from acquiring knowledge at the right time. For example, the records of chronically ill patients are difficult to present coherently.² A physician cannot read hundreds of clinical notes during a regular medical visit without any technological assistance. To achieve a comprehensive understanding of a patient’s situation, a physician sometimes retrieves information from other sources, which worsens the scenario.

Studies have shown that the consequences of information overload can be fatal.^3–5 Information overload increases task demand and mental effort, which potentially impairs healthcare providers’ understanding of patients’ medical conditions and hinders the providers from making optimal medical decisions.

Reducing information overload with automatic text summarization

Interacting with computer-based systems and accessing textual data has become an integral part of healthcare providers’ workflow.⁶ As an application of Automatic Text Summarization (ATS) in the biomedical domain, biomedical text summarization (BTS) generates condensed and relevant representations of the source documents⁷ computationally. It helps healthcare providers focus on the most valuable information, which benefits medical decision-making and enhances healthcare quality. Usability studies conducted with physicians for EHR summarization indicated the effectiveness of reading an automatically generated summary instead of the raw records.⁸

Background and history

New ideas and techniques are emerging in ATS.⁹ ATS systems usually need modifications before becoming applicable in the biomedical domain. There has been a growing interest in BTS.¹⁰ An informal literature survey conducted by Afantenos et al¹⁰ identified 10 BTS studies published between 1999 and 2003. This survey focused on issues such as scaling to large collections of documents, personalization, portability to new subdomains, and the integration of summarization technology with practical applications. In 2014, Mishra et al published a systematic review on BTS, which covered articles collected from MEDLINE, IEEE Digital Library, and ACM Digital Library from 2000 to 2013. They identified the need to apply and evaluate text summarization in research and patient care settings.¹¹ Advances have been achieved since Mishra’s review, and a more recent review is needed. Moradi & Ghadiri⁸ provided a brief overview of the recent advances of BTS, but they did not take a systematic approach to examining the cited studies.

Study scope

This survey focused on EHR and biomedical literature summarizations because they are the major types of content that BTS research typically concentrates on. It should be noted, however, the major goals of summarizations differ significantly for the user groups that utilize EHRs and biomedical literature. The EHR summarization systems are designed to help healthcare providers make optimized clinical decisions by extracting and organizing patient-specific information and evidence. Whereas, the primary goal of the biomedical literature summarization is to provide researchers and other end users an effective and efficient way to read and interpret biomedical research evidence .

Therefore, this review aims at (1) systematically investigating the most recent research (ie, published between 2013 to 2021) of the BTS application on biomedical literature and EHR documents; (2) identifying techniques, areas of application, and evaluation methods in these studies; (3) identifying gaps and proposing potential directions for future research.

MATERIALS AND METHODS

This study followed Mishra et al’s approach¹¹ based on IOM Standards for Systematic Reviews¹² and refined the review protocol for the research aims. We expanded the searching scope by adding 2 more databases (Web of Science and Scopus). The search formulation was modified with an expert review committee and a medical librarian. The search strategies and results can be found in Supplementary Appendix A. The following subsections describe the steps we followed to identify, screen, and extract relevant data from the included studies.

Data sources and searches

We searched PubMed, ACM Digital Library, Scopus, and Web of Science (containing IEEE). The literature search was from January 1st, 2013 to April 8th, 2021, aiming to update previous findings by Mishra et al¹¹ Important search terms included “biomedical text summarization,” “medical summarization,” “biomedical automatic summarization,” etc. (see Supplementary Appendix A).

Article selection

We included an original research study if it developed and evaluated ATS methods in biomedical and health sciences and biomedical and health informatics. We excluded an article if (1) the summarized text was outside of the biomedical domain; (2) the article was not a research article, such as editorials and opinion papers; (3) the emphasis was on text summarization tools, but without an evaluation component; (4) the related techniques (eg, text mining) in the article were used to support text summarization but did not produce a summary at the end; (5) the article was not written in English; (6) the article was about image and multimedia summarization without a text summarization component; or (7) the study was included in Mishra et al.¹¹

Title and abstract screening

Two reviewers (the first and second author) independently screened each article’s title and abstract in Covidence.¹³ Two reviewers had substantial agreement¹⁴ on title-abstract screening (Cohen’s kappa = 0.63). A third reviewer (the fourth author) helped resolve the conflicts. Also, both reviewers met to discuss any disagreement until they resolved discrepancies and achieved a consensus before moving to full-text screening.

Full-text screening

The same 2 reviewers independently screened the full text of 304 articles which passed the title-abstract screening. The agreement achieved in the full-text round of screening was much higher (Cohen’s kappa = 0.86).¹⁴ The third reviewer helped resolve the conflicts. The logic and the flow of the selection process are in Figure 1.

Data extraction and analysis

We developed a data extraction spreadsheet using Mani’s standards.¹⁶ Two reviewers (the first and second author) independently extracted the information from each included article according to the standards and resolved the conflicts with the third reviewer. Table 1 delineates the dimensions of our data extraction.

BTS input

Input characterizes the attributes of texts to be summarized, including (1) single vs multiple documents; (2) monolingual vs multilingual; (3) abstract vs full-text; (4) biomedical research literature vs EHR vs other document types (eg, medical news, clinical trial description, etc.).

BTS purpose

Purpose characterizes if the summarization system is (1) generic or user-oriented and if the system is (2) to facilitate clinical decision-making, to facilitate biomedical research, or to facilitate patient health information-seeking. A generic summarization system takes the predefined document(s) and generates a summary. In a user-oriented summarization system, the user provides a query or a list of parameters to customize the summaries.

“Healthcare providers” is the target user group for the clinical decision-making subcategory. The purpose of the summarization in the latter context was to digest clinical-related documents and deliver evidence-based knowledge. Some of the systems found were designed to aid patients in seeking health information. The subcategory of facilitating biomedical research is relatively broad. Studies summarizing biomedical literature without specifying a clinical purpose were counted as part of this group. It was found that some of the systems could belong to multiple subcategories. For example, the system discussed in Shree & Kiran¹⁷ took EHRs as input and assisted with clinical decision support. At the same time, the latter system had features for protecting sensitive information, which contributes to biomedical research.¹⁷

BTS output

Output in this study focused on the following attributes of summarization.

(1) extractive vs abstractive

An extractive summarization system extracts sentences from the original text according to their importance. In contrast, an abstractive system extracts knowledge from the text and reconstructs them in a new piece of text.

(2) informative vs indicative

Indicative summaries provide users with a general idea of the input source content, but users need to refer back to the original text to understand the content. Informative summaries offer enough details so that users do not need to check the original content.

BTS method

The 3 categories of method are described below:

(1) Statistical

Using a rule-based statistical method, a researcher manually selects features and cues and makes calculations using a predefined formula. The features could be the position of the sentence, the important keywords that it contains, etc.

(2) Machine Learning (ML)

For a model defined up to some parameters, ML is the execution of a computer program to optimize the model’s parameters using the training data or past experience.¹⁸ ML uses statistics in building mathematical models because the core task is making inferences from a sample.¹⁸ However, unlike statistical methods, the features of an ML method are selected automatically by an algorithm, and the parameters of a formula are not predefined. In our review, systems adopting ML usually combined other approaches. For example, Rouane et al¹⁹ extracted key concepts from the sentences using MetaMap²⁰ and sorted concepts of each sentence into separate itemsets. K-means clustering, an unsupervised learning approach, was applied to group these itemsets automatically. Frequent itemsets were mined to weigh the sentences, and the higher-weighted sentences were extracted for the final summaries. In recent years, systems adapting pretrained word embeddings and trained using seq2seq models are thriving.^21–23 These systems are categorized as pure ML, as they do not need human-defined features or language analysis before training.

(3) Computational linguistics (CL)

CL investigates computational modeling of natural language. It includes simple applications, like word counting, and complicated ones, such as language generation. Our study categorizes a system as using CL techniques if it adopts text processing functions, such as extraction of lexical knowledge, lexical and structural disambiguation, grammatical inference, or robust parsing. For example, the system developed by Scott et al²⁴ took the Chronicle,²⁵ a knowledge-based semantic graph generated from raw clinical health records, as input. Chronicle retrieved subgraphs according to the type and extent of the summary requested by users. Their system determined the order of utterances using the knowledge retrieved²⁴ and generated sentences using a template-based grammar.²⁴

(4) Hybrid

The hybrid method refers to using 2 or more methods from (1) to (3). For example, Gayathri et al²⁶ combined CL and statistical approaches. They extracted cue words using Medical Subject Headings (MeSH) and scored sentences based on the cue words’ frequency as well as other features, such as sentence position and sentence length.

BTS evaluation

Evaluation could be Intrinsic or Extrinsic,²⁷ quantitative or qualitative.

(1) Intrinsic vs Extrinsic

An intrinsic evaluation method assesses the summaries internally according to specific criteria, such as scoring metrics (eg, ROUGE) or attributes, like readability, comprehensiveness, accuracy, and relevancy.

An extrinsic evaluation method assesses the summaries by applying them in a downstream task. It may measure users’ efficiency, time, or accuracy when they complete a quiz using the automatically generated summaries.

(2) Quantitative vs Qualitative

In a quantitative evaluation, the metrics of measurement are clearly defined. The performance of the systems is evaluated based on their scores. On the contrary, qualitative evaluations do not have clear standards. The goal of a qualitative evaluation is usually exploration. In our review, most of the studies evaluated their systems quantitatively using predefined metrics, and some of the authors gave short qualitative analyses based on the quantitative results. We consider that a study adopts qualitative evaluation only if it has a separate section on qualitative analysis. Taking Moradi et al²⁸ as an example, besides the ROUGE metrics, it did a deep analysis on an example summary to bring out insights into their system.

RESULTS

A total of 58 publications were included. The characteristics and statistics of the included 58 studies (BTS systems) are summarized in Tables 2 and 3 at the end of section 3 and Table B.1 in Supplementary Appendix B. The percentages in the following results are approximated.

BTS input

There were 39 (67%) studies/systems designed for single-document summarization, 18 (31%) studies designed for multiple-document summarization, and 1 (2%) study designed for both single- and multiple-document summarization. All included systems were designed for monolingual documents. Forty-seven (81%) studies took full text as input, 10 (17%) took abstract as input, and 1 (2%) took both. Forty (69%) systems summarized biomedical research literature, 11 (19%) systems summarized EHRs, and 7 (12%) systems summarized other biomedical documents. As indicated by the results above, it is popular to summarize biomedical scientific articles while using abstracts as reference summaries. The primary reason could be that scientific papers are easily accessible and have clear themes.

BTS purpose

Sixteen (28%) systems considered users’ input (user-oriented) while 42 (72%) were generic. Seventeen (29%) systems were specifically designed to facilitate clinical decision-making, 5 (9%) systems were designed for helping with patient health information-seeking, and 34 (59%) focused on facilitating biomedical research. We classified 2 (^17,47) as having multiple purposes.

BTS output

Forty-seven (81%) systems adopted extractive approaches, 10 (17%) used abstractive approaches, and 1 (2%) system applied both—then compared the results. Most included summarization systems output informative summaries, and 2 (3%) of the systems generated a summary containing informative and indicative content. Generating informative summaries by selecting salient sentences is 1 of the widely used strategies in BTS. As extractive summarization avoids the challenging language-generating task, it brings redundant information and incoherent sentences. We will further address this issue in section “Abstractive Approaches are under Development.”

BTS method

Fifty-four (91%) systems applied hybrid methods, including the combination of Statistical and CL (22, 38%), the combination of ML and CL (4, 7%), the combination of Statistical and ML (1, 2%), and the combination of Statistical, ML, and CL (26, 45%). Four studies (7%) utilized ML only, and 1 (2%) applied CL only. The hybrid methods adopting CL were prevalent and efficient. We will further discuss these ideas in sections “Hybrid Methods,” “CL Techniques,” and “Syntactic Structures are Worth Further Investigation.”

BTS evaluation

Fifty-one (88%) studies conducted an intrinsic evaluation, 3 (5%) studies conducted an extrinsic evaluation, and the remaining 4 (7%) studies conducted both types of evaluations. Thirty-eight (66%) studies conducted quantitative analyses only, and 2 (3%) studies used qualitative methods only. Eighteen (31%) studies included both metrics for quantitative evaluation and case analyses as qualitative evaluation. Some frequently used baselines include LexRank,⁷⁹ TextRank,⁸⁰ MEAD,⁸¹ and the first/last several sentences.

Public availability

Fifty (86%) studies evaluated their systems using publicly available data, 2 (3%) of which included tests on private data as well. Eight (14%) studies linked to their source code or applications. More details can be found in Supplementary Table B.1 in Appendix B.

DISCUSSION

Our study is the first systematic review of BTS since 2014. Compared to the previous review,¹¹ we searched more literature databases and identified more relevant studies while keeping a similar scope for literature searching, study selection, and data extraction. Our results confirmed some previous findings¹¹ and showed unique BTS research trends in the most recent years.

State-of-the-art and improvement

Hybrid methods

The use of hybrid methods (91% in our review) has become the norm since 2013 (44% in Mishra et al¹¹) confirming their observation that hybrid methods had great potential. The system proposed by Sarker et al⁶⁸ generated all possible 3-sentence combinations and then selected the combination having the highest ROUGE-L F-score as the ideal summary. They took advantage of supervised learning by using these ideal summaries to train their system and derived the final statistics of their summarization model. Besides the auto-generated statistics, they had relative sentence position and semantic types identified using UMLS. The system features also included manually composed formulas. Therefore, Sarker’s system adopted a hybrid mode by combining CL, statistical, and ML. With the increasing availability of the pretrained word embeddings, the application of seq2seq ML models is growing. In reviewing the literature, it was found that, despite the increasing use of the latter ML models, the hybrid models remain in demand and they are frequently used in BTS.

CL techniques

Most hybrid methods (52 out of 53 articles) in this study included CL techniques. Knowledge-rich approaches that combine predefined domain knowledge are common in these methods. Mishra et al¹¹ believed that many publicly available knowledge resources, such as UMLS and MeSH and tools, such as MetaMap²⁰ and SemRep,⁸² contributed to the high interest in CL. Including an expert-maintained domain knowledge database significantly improved the performance of various models. Gayathri et al²⁶ showed an example of a hybrid method combining CL and statistical approaches. They extracted cue words using MeSH terms and scored sentences based on the cue words’ frequency and other features, such as sentence position. Gigioli et al⁴⁰ applied neural abstractive techniques⁸³ to the biomedical domain. Their abstractive summarization system was capable of generating novel summaries while considering domain knowledge. In addition, Gigioli et al explored maximum likelihood learning, reinforcement learning, and a mixed learning policy in their pointer–generator model. Both Gayathri’s and Gigioli’s studies compared their proposed methods with and without integrating biomedical knowledge. Their results indicated that integrating domain-related knowledge improved the performance of their models.

Public corpora

Increasingly, researchers have contributed to developing public corpora for BTS. For example, some studies (eg, Bavani et al³⁰) used a specialized evidence-based medicine corpus—which was gathered and annotated by Mollá et al⁸⁴ for the sole purpose of BTS. This corpus was sourced from the Clinical Inquiries section in the Journal of Family Practice, consisting of 456 clinical queries with 1396 bottom-line evidence-based answers. For each bottom-line answer, there existed a set of detailed justifications. Each detailed justification was in turn associated with at least 1 source document. Thus, this dataset can be reused for either multi-document or single-document summarization tasks.

Experiments in real-world settings and usability tests with physicians and patients

Among the studies designed for facilitating clinical decision-making by summarizing EHRs, 7 of them assessed their systems in real-world settings by conducting usability tests. Goldstein et al⁴³ evaluated their system through intrinsic and extrinsic usability tests. There were 3 components in their evaluation session: (1) relative completeness by requesting physicians to tag if the missing items in a generated summary were essential; (2) quality analysis by checking readability, comprehensiveness, clinical course, and continuity of care; (3) functional analysis based on the correctness and time of physicians answering 5 basic clinical decision questions. On average, physicians answered the questions 40% faster (P < .001) when using a system-generated letter than when using a physician-composed letter. Considering the correctness of the answers, they found that for 4 out of 5 questions, physicians did equally well or significantly better (P < .005) when using the system-generated letter. Moen et al⁵³ tried to validate the reliability of automatic evaluations on EHR summarization. They tested Spearman’s rank correlation coefficient between the scores assessed by domain experts and by ROUGE metrics.

For those studies designed for facilitating patients’ information-seeking, 2 of them conducted usability tests. In Liu et al,⁴⁹ the participants rated their satisfaction regarding the usefulness and representativeness of the summaries generated by the proposed and the baseline system. And in Yin et al,⁷⁸ 3 users compared the generated summaries and the results by a search engine. Both of them found the proposed systems satisfied users.

All usability tests with users (either physicians or patients) were conducted using surveys or interviews with predefined metrics. It needs to be noted that, in some of the studies, the authors manually analyzed examples. These were referred to as “Additional qualitative evaluation” and “Preliminary evaluation.” More details can be found in Supplementary Appendix B Table B.1.

Automatic text summarization of EHRs

As shown in the studies by Goldstein et al⁴³ and Scott et al,²⁴ a summary of health records helped physicians save a significant amount of time processing patients’ information and improved clinical decision accuracy. Nineteen percent of the included studies provided solutions to summarizing EHRs. This is a significant increase from the 9% identified by Mishra et al’s review.¹¹

Worldwide development

Text summarization in the biomedical domain has become a research focus worldwide. The included studies were conducted by researchers from 17 different countries (Table 3), 6 more countries than identified previously.¹¹ The USA and Australia remain among the top producers of BTS research, but researchers from several other countries (Iran, China, Israel, and India) produced 4 or more studies included in this review. The trend of multinational collaboration in BTS research has been persistent. Besides, systems have been developed for text summarization in different languages other than English.³³

Gaps and challenges

Despite research progress made in recent years, this study identified a few gaps and challenges.

Syntactic structures are worth further investigation

When using CL techniques, most approaches emphasized semantic knowledge. Syntactic features that compose an essential part of CL are frequently ignored. However, combining syntactic features can increase concept and relation extraction accuracy.⁸⁵ As for information extraction, understanding the role of the extracted piece in the sentence is crucial. Integrating syntactic features in BTS systems is worth further investigation. In those seq2seq models where no features need to be specified, we believe that the order information embedded in the hidden layers is potentially beneficial.

Abstractive approaches are under development

The majority of the systems are extractive. Abstractive summarization has extra challenges of natural language generation. Parveen et al⁶² found that the machine-generated abstractive summaries might have readability issues even if they cover all essential information. However, extractive summaries might contain redundant information that impacts the summary quality. As the space is limited, redundancy may result in a core information deficiency. Therefore, investigations are needed for developing intuitive, efficient, and context-sensitive abstractive summarization systems.

Challenges of EHR summarization

BTS for EHRs in clinical settings is still under development. This study observed several challenges in the EHR summarizations. First, free notes containing inconsistent abbreviations, incomplete sentences, and unclear implications increased the difficulty of text summarization. CL tools built on semantic knowledge databases are widely used to deal with these problems. However, the databases need intensive maintenance and are not always up-to-date. Second, we still lack universal datasets for EHR summarization. Concerning patients’ privacy and security, researchers have difficulty accessing real patient records or acquiring the corresponding gold standard summaries generated by human professionals. Goldstein et al⁴³ conducted their experiment on MIMIC, an openly available dataset comprising deidentified health data.⁸⁶ Due to the lack of gold standard summaries, they used the discharge summary as an alternative, potentially impacting their system’s overall performance and the user experience. Third, there has been no widespread adoption of BTS in clinical settings. Deployment is often hindered by a wide variety of established commercial EHR systems. The lack of rigorous evaluation is another large barrier for translating research into clinical practices.⁸⁷ Therefore, developing universal and high-quality EHR summarization datasets is vital for research and actual deployment.

LIMITATIONS

First of all, due to the scope of this study, some commercial BTS systems may have been inadvertently overlooked. Second, a meta-analysis comparing the performance of different approaches was not possible due to the heterogeneity of the evaluation methods in the studies reviewed. ROUGE metrics, although prevalent, are not the only option. The lack of a widely used and standardized dataset brought challenges for comparing different systems. Third, as our data screening, extraction, and analysis were guided by Mani’s framework,¹⁶ there might be additional dimensions and trends outside of Mani’s framework not included in this study. For example, since most of the systems utilized CL techniques and extractive approaches, these 2 dimensions could be further divided into more granular classes in future reviews. Finally, relevant studies published in languages other than English were excluded.

CONCLUSION

This study systematically reviewed the latest research publications of text summarizations of biomedical literature and EHRs. The review covered articles published from 2013 to April 8, 2021, immediately following the last published systematic review on the same topic. Our findings demonstrate that the current BTS systems had achieved good performance using hybrid methods. It was found that CL techniques, especially knowledge-rich approaches, deliver positive outcomes. However, as essential components of CL techniques, the power of syntactic parsing and features have not been fully leveraged in BTS systems. Last but not least, most BTS systems were still designed for summarizing biomedical research literature rather than EHRs.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors

AUTHOR CONTRIBUTIONS

MeW, MaW, and YY are the reviewers of the searched studies. MeW, FY, and JW contributed to the searching and deduplicating of the studies. MeW, FY, and JM contributed to the writing and editing of the paper.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

DATA AVAILABILITY STATEMENT

The data underlying this article are available in the article and in its online supplementary material .

CONFLICT OF INTEREST STATEMENT

None declared.

REFERENCES