Development and validation of the provider documentation summarization quality instrument for large language models

Abstract

Objectives

As large language models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation and as models and documentation practices evolve. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed to evaluate LLM-generated clinical summaries. This study aimed to validate the PDSQI-9 across key aspects of construct validity.

Materials and Methods

Multi-document summaries were generated from real-world EHR data across multiple specialties using several LLMs (GPT-4o, Mixtral 8x7b, and Llama 3-8b). Validation included Pearson correlation analyses for substantive validity, factor analysis and Cronbach’s $α$ for structural validity, inter-rater reliability (ICC and Krippendorff’s $α$⁠) for generalizability, a semi-Delphi process for content validity, and comparisons of high- versus low-quality summaries for discriminant validity. Raters underwent standardized training to ensure consistent application of the instrument.

Results

Seven physician raters evaluated 779 summaries and answered 8329 questions, achieving over 80% power for inter-rater reliability. The PDSQI-9 demonstrated strong internal consistency (Cronbach’s $α$ = 0.879; 95% CI, 0.867-0.891) and high inter-rater reliability (ICC = 0.867; 95% CI, 0.867-0.868), supporting structural validity and generalizability. Factor analysis identified a 4-factor model explaining 58% of the variance, representing organization, clarity, accuracy, and utility. Substantive validity was supported by correlations between note length and scores for Succinct (⁠$ρ$ = −0.200, P = .029) and Organized (⁠$ρ$ = −0.190, P = .037). The semi-Delphi process ensured clinically relevant attributes, and discriminant validity distinguished high- from low-quality summaries (⁠$P<.001$⁠).

Discussion

The PDSQI-9 showed high inter-rater reliability, internal consistency, and a meaningful factor structure that reliably captured key dimensions of documentation quality. It distinguished between high- and low-quality summaries, supporting its practical utility for health systems needing an evaluation instrument for LLMs.

Conclusions

The PDSQI-9 demonstrates robust construct validity, supporting its use in clinical practice to evaluate LLM-generated summaries and facilitate safer, more effective integration of LLMs into healthcare workflows.

Introduction

The volume of notes in the electronic health record (EHR) has increased in the past decade, exacerbating the burden on providers and highlighting the growing difficulty of the unassisted “chart biopsy” task. One in 5 patients arrives at the hospital with a chart comparable to the size of “Moby Dick” (206 000 words).¹ While the EHR’s centralized storage of medical notes is beneficial, and studies have shown that access to prior records improves diagnostic accuracy, the growing volume of data presents a significant challenge.² The tension between the EHR’s role as a documentation repository and its function as a tool for retrieving actionable information has become increasingly unmanageable without additional filtering and summarization tools.³

In the field of clinical Natural Language Generation (NLG), multi-document summarization has emerged as an important task to address the challenge of note bloat and reduce the cognitive burden on healthcare providers. As large language models (LLMs) continue to advance NLG capabilities, they offer a promising alternative for summarizing clinical documentation and alleviating the time-intensive nature of human-authored summaries. Recent improvements in the introduction of larger context windows in LLMs^4,5 allow for the input of a patient’s entire hospital course. However, these new capabilities also introduce challenges; Liu et al⁶ and Li et al⁷ highlighted that LLMs may not fully process all the text present in larger context windows. Performance degradation as a result of chronological errors or missed details becomes especially important in medical domain tasks where patient safety is a priority. Despite their potential, the rapid advancements of LLMs have outpaced the development of robust evaluation instruments to assess the quality of their outputs.

Current human evaluation and automated evaluation workflows were not developed to account for both the clinical aspects of evaluation and the technical complexities associated with LLM-generated text.⁸ Automated evaluation metrics developed to assess the output in summarization tasks, like n-gram overlap or semantic scores, have not been shown to perform adequately on such tasks in the medical domain.⁹ They fail to account for the nuanced nature of clinically relevant summarizations by relying on surface-level heuristics to produce evaluative scores. In particular, most automated evaluation metrics emphasize textual overlap and similarity, resulting in scores that may not fully reflect the factual accuracy or relevance of the generated content. Instead, these scores primarily indicate how closely the output aligns structurally and lexically with a given reference. While such similarities are important, they represent only a portion of a thorough assessment. The reliability of these evaluations is also heavily influenced by the quality of the reference text. Moramarco et al¹⁰ highlighted this concern, demonstrating that many automated metrics exhibit a strong bias toward the structure of the reference text in their comparative analysis of various evaluation tools. This bias highlights the need for evaluation methods that go beyond semantic and lexical resemblance to better assess the actual relevance of the generated text. The complex reasoning and in-depth medical knowledge required in a clinical setting lend themselves more to abstractive summarization. While human evaluation workflows are able to account for these aspects, there remains a paucity of evidence on human evaluation instruments designed for LLM summarizations developed from real-world, multi-document EHR data and supported by psychometric validation.¹¹ Tam et al.’s systematic review of 142 studies on human evaluation methodologies for LLMs in healthcare highlighted significant gaps, including the lack of sample size calculations, insufficient details on the number of evaluators, and inadequate evaluator training.¹² While their paper provided areas for improvement, the suggested strategies were primarily based on patterns observed in the literature rather than being grounded in statistical frameworks. There is a significant gap in human evaluation methodologies designed to address the complexities of healthcare applications and the unique challenges posed by LLMs.

Existing evaluative instruments for provider notes are designed primarily for provider-authored documentation. One widely adopted instrument, which has also been applied to AI-generated notes,^13–15 is the Physician Documentation Quality Instrument (PDQI). The instrument has demonstrated strong reliability, achieving high inter-rater reliability and internal consistency.¹⁶ Although the PDQI-9 tool is validated and reliable for evaluating provider-authored notes, it was not designed to address the unique challenges posed by LLM summarization of notes. Summaries generated by LLMs must be assessed for additional factors such as relevancy, hallucinations, omissions, and factual accuracy, areas where LLMs have demonstrated limitations.¹⁷ To address these gaps, this study introduces the Provider Documentation Summarization Quality Instrument (PDSQI-9), an LLM-centric adaptation of the PDQI-9. The PDSQI-9 is specifically designed to evaluate LLM-generated summaries, developed and validated on real-world EHR data, and rigorously tested for psychometric properties with adequate statistical power.

Methods

Study design, setting, and data

The corpus of notes was designed for multi-document summarization and evaluation using inpatient and outpatient encounters from the University of Wisconsin Hospitals and Clinics (UW Health) in Wisconsin and Illinois between March 22, 2023 and December 13, 2023. The initial study corpus from which we sampled records consisted of 1 811 763 encounters across 471 569 unique patients. The evaluation was conducted from the perspective of the provider during their initial office visit with the patient (“index encounter”), representing a real-world clinic appointment where the provider benefits from a summary of the patient’s prior encounters with outside providers. Other inclusion and exclusion criteria were the following: (1) patient was alive at time of index encounter with provider; (2) patient had at least one encounter in 2023; and (3) excluded psychiatry notes. Psychotherapy notes were excluded due to their highly sensitive nature and additional regulatory protections under HIPAA and 42 CFR Part 2, which require approvals beyond the minimal necessary standard for research. The corpus was further filtered to patients with 3 or more encounters before the index encounter, to provide a multi-document occurrence and fit the context window of many large language models. The resultant corpus consisted of 2123 patients, with 554 having 3 encounters, 389 having 4 encounters, and 1180 having 5 encounters. The derived dataset, consisting of encounters with concatenated provider notes leading to the index encounter of interest, was built as a random sample from the corpus with stratification across 11 specialties (25% from gynecology, urgent care, neurosurgery, neurology, or urology; 40% from dermatology, surgery, orthopedics, or ophthalmology; and 35% from family medicine or internal medicine). The sample size needed for evaluation was determined a priori (see sample size estimation) and included additional examples for pilot testing by the instrument developers and training by the raters. The final dataset was 200 unique patients and their encounters and 22.5%, 22%, and 55.5% had 3, 4, and 5 notes per patient summarized prior to the index encounter. This study was approved by the University Wisconsin-Madison Institutional Review Board and qualified as exempt human subjects research.

Development of PDSQI-9

The instrument development process employed a semi-Delphi methodology, an iterative consensus-driven approach commonly used for gathering expert opinions and refining complex frameworks.¹⁸ The semi-Delphi process consisted of 3 iterative rounds, each involving 9 stakeholders with diverse expertise: 3 physicians who were also clinical informaticists with specialized knowledge in human factors design and natural language processing (M.A., B.P., and K.W.); 2 software developers with experience in generative AI (N.P. and E.F.); 1 quality improvement specialist (M.S.); 2 data scientists (E.C. and G.W.); and 1 computer scientist with expertise in computational linguistics (Y.G.).

Round 1—literature review and domain identification

The panel reviewed existing literature and methodologies for evaluating clinical text summarizations. The PDQI-9 was selected as the benchmark due to its demonstrated validity, interpretability, and applicability to evaluating physician clinical documentation. The panel then identified key domains essential for high-quality multi-document summarization, as well as dimensions where LLMs are known to underperform, such as hallucinations, omissions, and relevancy.^8,17,19

The identified dimensions were mapped to existing PDQI-9 attributes where feasible, with modifications to improve applicability of clarity and relevance attributes. Two PDQI-9 attributes, Up-to-Date and Consistent, were removed as their conceptual scope was adequately captured by modifications to other attributes. Two new attributes were added to address concerns in LLM-generated summaries: use of stigmatizing language and inclusion of citations linking facts in the summary to the original documentation. Stigmatizing language was defined using the Center for Health Care Strategies tool “Words Matter: Strategies to Reduce Bias in Electronic Health Records.”²⁰ This tool was created to guide clinicians towards actively avoiding biased language in their notes. This document was included in the final rubric that was provided to all human evaluators and referenced during their training for consistency in the application of the attribute.

Round 2—attribute refinement and mapping to Likert scales

The instrument definitions for each attribute were refined, and detailed instructions were developed for scoring on a 5-point Likert scale. The panel iteratively revised attribute definitions to ensure clarity and usability. Special emphasis was placed on designing attribute definitions that captured the nuances of clinical text summarization, including factors such as relevancy, factual accuracy, and faithfulness to the source documentation. The panel placed a particular focus on the vulnerability of the attribute Accurate to hallucinations and the attribute Thorough to omissions. In this context, hallucinations are defined as falsifications or fabrications present in the summary. A fabrication is defined as made-up information or data which could be plausible, but based on non-existent facts. A falsification is defined as distorted information or data which contains information distorted from the original note. In this context, omissions are only considered when classified as pertinent or potentially pertinent. A pertinent omission refers to information that is essential for the use case or target provider and would have adverse effects on patient care decisions. A potentially pertinent omission refers to details relevant to a patient’s expected clinical course, but would not directly influence their care. An example of a hallucination and an omission seen in our dataset our present in Figures 1 and 2.

Round 3—pilot testing and consensus adjudication

Pilot testing was performed with 3 senior physicians (M.A., K.W., and B.P.) from different specialties to evaluate the usability and clarity of the attributes and scoring instructions. Feedback from these testers was incorporated iteratively, with the adjudication of disagreements conducted by the expert panel to achieve consensus. Pilot testing continued until all testers agreed on the final instrument definitions and scoring instructions, ensuring content validity of the instrument through the semi-Delphi process. The final instrument is in Appendix S3.

The final instrument includes 9 attributes: Cited, Accurate, Thorough, Useful, Organized, Comprehensible, Succinct, Synthesized, and Stigmatizing (Table 1). While the original PDQI-9 tool used the same 5-point Likert scale for every attribute, our adapted version incorporates a combination of 5-point Likert scales and binary scales tailored to the specific requirements of each attribute. The PDSQI-9 was developed and managed using REDCap.^32,33 The REDCap instrument and data dictionary are available at the following GitLab repository: https://git.doit.wisc.edu/smph-public/dom/uw-icu-data-science-lab-public/pdsqi-9.

LLM summarizations for validating the PDSQI-9

To generate summaries of varying quality, we employed different prompts across different LLMs to summarize notes for each patient encounter leading up to the index encounter. The LLMs utilized in this study included OpenAI’s GPT-4o,³⁴ Mixtral 8x7B,³⁵ and Meta’s Llama 3-8B³⁶ (Table 2). GPT-4 operates within the secure environment of the health system’s HIPAA-compliant Azure cloud. No PHI was transmitted, stored, or used by OpenAI for model training or human review. All interactions with proprietary closed-source LLMs were fully compliant with HIPAA regulations, maintaining the confidentiality of patient data. The open-source LLMs, Mixtral 8x7B and Llama-3-8B, were downloaded from HuggingFace³⁷ to HIPAA-compliant, on-premise servers.

We used 4 strategies for engineering the evaluation prompts: minimizing perplexity, in-context examples, chain-of-thought reasoning, and self-consistency. The prompt for each LLM included a persona with the following instruction: “You are an expert doctor. Your task is to write a summary for a specialty of [target specialty], after reviewing a set of notes about a patient.” To generate lower-quality summaries, additional variations of the prompt removed instructions or encouraged the inclusion of false information. The persona and instruction were followed by 2 chains of thought: Rules and Anti-Rules, delineating positive and negative summarization steps. The Rules targeted specific attributes in the PDQSI-9 to generate high-quality summaries. Each Rule has a corresponding Anti-Rule which is its equivalent on a word-for-word basis, but with inverted meaning (ie, Rule: Keep the summary succinct and Anti-Rule: Keep the summary meandering and long). Anti-Rules introduced intentional errors (eg, hallucinations, omissions) to include mistakes. Both prompting strategies were developed in-house, like the rest of our summarization prompt, to create a useful and heterogeneous dataset. For each patient, the LLM was provided a randomized subset of Rules and Anti-Rules, ensuring heterogeneity in the generated summaries. The open-source models, Mixtral 8x7b and Llama 3-8b, were tasked with producing the lowest-quality summaries and were exclusively provided Anti-Rules as instructions. This approach aimed to provide a wide distribution of PDSQI-9 scores and to allow for discriminant validity testing. The prompts are available at https://git.doit.wisc.edu/smph-public/dom/uw-icu-data-science-lab-public/pdsqi-9. The final corpus had 100 summaries generated by GPT-4o, 50 by Mixtral 8x7b, and 50 by Llama 3-8b. Examples for high- and low-quality summaries are presented in Figure 3.

Sample size estimation and rater training

Sample size calculations were performed assuming a minimum of 5 raters and an even score distribution across a 5-point Likert scale. To achieve a desired statistical power of 80% with a precision of 0.1, each rater was required to complete at least 84 evaluations.³⁸

Five junior physician raters with 1 to 5 years of post-graduate experience were recruited (M.P., K.B., C.E., S.K., and T.R.). Additionally, 2 senior physician raters (J.G. and M.K.), each with at least 10 years of post-graduate experience, were recruited to complete a subset of evaluations for further validation. To standardize evaluation criteria and scoring, a group of 3 senior physician trainers (M.A., K.W., and B.P.) conducted evaluations on 3 exemplar cases. These cases were subsequently used as reference materials for all raters during a live training session and have been fully deidentified and made publicly available in our GitLab repository: https://git.doit.wisc.edu/smph-public/dom/uw-icu-data-science-lab-public/pdsqi-9. Given the varying levels of expertise among the raters, all were provided with the Center for Health Care Strategies’ documentation on identifying bias and stigmatizing language in EHRs.²⁰ Following the training session, raters were encouraged to pose questions or highlight disagreements during subsequent practice days. After the training period, all raters independently completed 3 additional example cases. Agreement among the raters was established before proceeding with independent evaluations of the full dataset.

Analysis plan and validation

Baseline characteristics of the corpus notes and evaluators were analyzed. Distributions of evaluative scores for each attribute were visualized using density ridge plots. Token counts were derived using the Llama 3-8b tokenizer.

The PDSQI-9 was evaluated through multiple metrics to assess its validity and reliability, informed by Messick’s Framework of validity.³⁹ To examine the theoretical alignment of the PDSQI-9, Pearson correlation coefficients were calculated to evaluate relationships between input characteristics (eg, note length) and attribute scores (eg, Succinct and Organized). This tested whether the instrument captured expected relationships consistent with the theoretical underpinnings of summarization challenges and provided substantive validity. For generalizability, inter-rater reliability was assessed using the Intraclass Correlation Coefficient (ICC) and Krippendorf’s⁴⁰  $α$⁠, ensuring the instrument produced consistent results across evaluators with varying levels of expertise. ICC is derived from an analysis of variance (ANOVA) and has several forms tailored to specific use cases.⁴¹ In this study, a 2-way mixed-effects model was used for consistency among multiple raters, specifically ICC(3, k).⁴² Unlike ICC, which is a variance-based measure, Krippendorff’s $α$ was calculated based on observed disagreement among raters and adjusted for chance agreement. The performance of junior physician evaluators was tested against senior physician evaluators (J.G. and M.K.) using the Wilcoxon signed-rank test, to assess differences in median scores between the 2 groups. To assess the PDSQI-9’s discriminative validity, a Mann-Whitney U test was performed between the lowest and highest quality summaries. The summaries generated by GPT-4o with error-free prompts were considered the highest quality, while those generated by Llama 3-8b and Mixtral 8x7b with error-prone prompts were considered the lowest quality.

For structural validity, internal consistency was measured using Cronbach’s $α$⁠,⁴³ which evaluates whether the instrument items reliably measure the same underlying construct. Confirmatory factor analysis was conducted to identify latent structures underlying the survey attributes and to evaluate alignment with theoretical constructs. Factor loadings were used to assess variance within the instrument. A 4-factor model was selected based on eigenvalues, the scree plot, and model fit indices (Appendix S2).

Cronbach’s $α$⁠, ICC, and Krippendorff’s $α$ produced coefficients ranging between 0 and 1, where higher values indicate greater reliability or agreement. 95% Confidence intervals (CI) were provided for all coefficients and calculated using the Feldt procedure,⁴⁴ Shrout and Fleiss procedure,⁴⁵ and bootstrap procedure, respectively. Analyses were performed using Python (version 3.11)^46,47 and R Studio (version 4.3).^48–52

Results

Seven physician raters evaluated 779 summaries and scored 8329 PDSQI-9 items to achieve greater than 80% power for examining inter-rater reliability. No difference was observed in the scores by rater expertise when comparing junior and senior physicians (n = 48 summarizations; P-value = .187). The median time required for the junior physicians to complete a single evaluation, including reading the provider notes and the LLM-generated summary, was 10.93 minutes (IQR: 7.92-14.98). Senior physician raters completed evaluations with a median time of 9.82 minutes (IQR: 6.28-13.75) (Appendix S1).

The provider notes, concatenated into a single input for each patient, had a median word count of 2971 (IQR: 2179-3371) and a median token count of 5101 (IQR: 3614-7464). The provider note types included notes from 20 specialties (medicine, family medicine, orthopedics, ophthalmology, emergency medicine, surgery, dermatology, urgent care, urology, neurology, gynecology, psychiatry, anesthesiology, neurosurgery, somnology, pediatrics, audiology, and radiology). The LLM-generated summaries of the provider notes had a median length of 328.5 words (IQR: 205.8-519.8) and 452.5 tokens (IQR: 313.5-749.5). A modest positive correlation was observed between the input text’s length and the generated summaries’ length (⁠$ρ$ = 0.221 with P-value = .002).

Figure 4 illustrates the average scores for each attribute of a summary, as evaluated by our raters, in relation to the length of the notes being summarized. As the length of the notes increased, the quality of the generated summaries was rated lower in the attributes of Organized (⁠$ρ$ = -0.190 with P-value = .037), Succinct (⁠$ρ$ = -0.200 with P-value = .029), and Thorough (⁠$ρ$ = -0.31 with P-value < .001). Additionally, the variance in scores among the raters increased with longer note lengths for the attributes of Thorough (⁠$ρ$ = 0.26 with P-value = .004) and Useful (⁠$ρ$ = -0.28 with P-value = .003) (Figure 5).

Figure 6 illustrates the distribution of evaluative scores for each attribute of the PDQSI-9. As intended, the scores spanned the entire range of the PDQSI-9 for nearly all attributes. The only exception was the Comprehensible attribute, where no summary received a score of “1” on the Likert scale. This was attributed to the inherent quality of the LLMs used and the challenges encountered in jailbreaking them to generate incomprehensible outputs. The attributes Succinct and Thorough exhibited the smoothest distributions in scoring. The median score and IQR for each attribute was 3.0 (3.0-4.0) for Useful, 4.0 (2.0-5.0) for Thorough, 3.0 (3.0-4.0) for Synthesized, 4.0 (3.0-5.0) for Succinct, 3.0 (2.0-4.0) for Organized, 5.0 (4.0-5.0) for Comprehensible, 4.0 (1.0-5.0) for Cited, and 5.0 (4.0-5.0) for Accurate.

The PDSQI-9 demonstrated discriminant validity when comparing the lowest quality summaries and the highest quality summaries (P-value < .001). Reliability metrics for each instrument attribute are detailed in Table 3. Overall intraclass correlation coefficient (ICC) was 0.867 (95% CI, 0.867-0.868) and Krippendorff’s $α$ was 0.575 (95% CI, 0.539-0.609). Cronbach’s $α$ was 0.879 (95% CI, 0.867-0.891), indicating good internal consistency. The goodness-of-fit metrics for the 4-factor model indicated strong validity (Root Mean Squared Error of Approximation: 0.05, Bayesian Information Criterion: −6.94). The 4 factors cumulatively explained 58% of the total variance in the dataset. The first factor, accounting for 23% of the explained variance, included the attributes Cited, Useful, Organized, and Succinct. The second, third, and fourth factors, representing Comprehensible, Accurate, and Thorough, accounted for 13%, 12%, and 10% of the variance, respectively. The attribute Synthesized did not exhibit significant positive associations, likely reflecting its inherent complexity and the challenges evaluators faced in abstraction summarization. Responses were unanimous for only 8.5% of the 117 summaries when asked if the notes presented an opportunity for abstraction in the summary. Further details on the factor analysis and loadings are provided in Appendix S2.

The attribute for stigmatizing language was excluded from Table 3 due to the binary nature of its evaluative responses. Raters were in complete agreement on the presence of stigmatizing language in the notes 61% of the time and in the summaries 87% of the time.

Discussion

This study introduces the PDSQI-9 as a novel and rigorously validated instrument designed to assess the quality of LLM-generated summaries of clinical documentation. Using Messick’s Framework, multiple aspects of construct validity were demonstrated, ensuring that the PDSQI-9 provides a well-developed and reliable tool for evaluating summarization quality in complex, real-world EHR data. Strong inter-rater reliability (ICC: 0.867) and moderate agreement (Krippendorff’s $α$⁠: 0.575), combined with consistent performance across evaluative attributes were key findings. No differences in scoring were observed between junior and senior physician raters, underscoring the instrument’s reliability across varying levels of clinical experience. Strong discriminant validity was shown between high- and low-quality summaries. To our knowledge, the PDSQI-9 is the first evaluation instrument developed using a semi-Delphi consensus process, applied to real-world EHR data, and supported by a well-powered study design with nearly 800 patient summaries.

The strong inter-rater reliability (ICC = 0.867) was comparable to the results reported in the original PDQI-9 study, which highlighted the reliability of the instrument in evaluating clinician-authored notes.¹⁶ The moderate Krippendorff’s $α$ (0.575) reflects robust agreement despite the complexity of the evaluation tasks. The strong internal consistency (Cronbach’s $α$ = 0.879) supports the structural validity of the instrument, demonstrating that its attributes cohesively measure the construct of summarization quality. The 4-factor model further demonstrated strong construct validity, aligning attributes with theoretical constructs relevant to evaluating LLM-generated clinical summaries. The identified factors capture key dimensions of clinical summarization quality, including organization, clarity, accuracy, and utility, validating the instrument’s use for this purpose.^8,11,12,53

The semi-Delphi process facilitated the inclusion of clinically relevant attributes, grounded in expert consensus, to ensure the instrument’s applicability in real-world settings. This iterative process refined the PDSQI-9 to address critical issues unique to LLM-generated text, such as hallucinations, omissions, and stigmatizing language. By incorporating attributes specifically designed to evaluate LLM outputs, such as hallucinations and omissions, the PDSQI-9 effectively identifies risks associated with LLM-generated summaries, reinforcing safer applications of LLMs in clinical practice. The inclusion of a stigmatizing language attribute further enhances the instrument by identifying potentially harmful language in notes or summaries. Given the importance of equitable care, LLMs tasked with summarization must avoid introducing language that could perpetuate provider bias or negatively influence clinical decision-making.⁵⁴

The evaluation process revealed notable differences in efficiency between junior and senior physician raters, with senior physicians completing evaluations more quickly compared to the overall median time; however, this did not affect the scores between the groups and shows the instrument is reliable across different levels of experience. Notable differences were the observed correlations between input note length and declining quality scores highlighting the need for careful consideration of input complexity when deploying LLMs in clinical workflows.⁵⁵

The selected LLMs included state-of-the-art models such as GPT-4o, alongside smaller, open-source models that are more prone to errors, allowing for a comparative evaluation across diverse capabilities. The smallest context window among the models was 8K tokens, and the median input length of approximately 5K tokens provided a long yet manageable input size with available compute resources. With 3-5 provider notes from the EHR per case, the design allowed for realistic testing of LLM performance in a clinical context, highlighting their strengths and limitations in processing multi-document inputs and generating specialty-relevant summaries.

Although the generated summaries were designed to represent varying levels of quality, achieving an even distribution of scores across attributes such as Comprehensible, Synthesized, and Accurate proved challenging. The skewed distributions impacted reliability metrics, with the degree of impact varying based on each metric’s robustness to unbalanced data. The Comprehensible attribute was likely influenced by advancements in LLMs, which can produce coherent text regardless of relevancy. In contrast, Accurate and Synthesized attributes highlight the challenges of evaluating extractive versus abstractive summarization. Extractive summarization reflects content directly from the notes, while abstractive summarization requires synthesizing and expanding on information, both of which are critical but more subjective. To address this, an additional step was added to the instrument, asking raters to determine whether abstraction opportunities existed in each note/summary pair. Factor analysis results emphasized the importance of these attributes, with Synthesized showing weak factor association, reflecting the difficulty of evaluating this skill even for humans. Nevertheless, abstraction in synthesis is important in clinical contexts and remains a challenge for both humans and LLMs.

In this study, we developed a 1-5 Likert scale to evaluate the quality of LLM-generated summaries, but an important next step is to establish cut-off points that determine when a summary is of sufficient quality for clinical use. Defining such thresholds would allow clinicians to classify summaries as “usable” or “unusable” for patient care, addressing the instrument’s practical utility. Future research could explore methods to define these cut-off points, such as convening expert panels to reach a consensus on minimum acceptable scores or conducting studies to associate Likert scores with clinical outcomes, such as decision accuracy or patient impact. These approaches represent a promising direction for extending the applicability of our evaluation instrument in clinical settings.

As with other human evaluation frameworks, the PDSQI-9 relies on significant human labor costs which may prevent its implementation on larger scales. Our future work will focus on the potential for automation of this workflow to reduce the upfront human labor costs associated with robust and reliable evaluation. The introduction of Reinforcement Learning with Human Feedback and Human Aware Loss functions have highlighted the potential for a well-designed LLM evaluator to align its output with human preferences.⁸ These “LLMs-as-a-Judge” workflows may be able to straddle the gap between the high reliability of human evaluations and the efficiency of automated methods.

In conclusion, the PDSQI-9 is introduced as a comprehensive tool for evaluating clinical text generated through multi-document summarization. This human evaluation framework was developed with a strong emphasis on aspects of construct validity. The PDSQI-9 offers an evaluative schema tailored to the complexities of the clinical domain, prioritizing patient safety while addressing LLM-specific challenges that could adversely affect clinical outcomes.

Limitations

The empirical results reported herein should be considered in the light of some limitations. This study focuses on a single task, multi-document summarization, and utilizes a dataset from a single health system. The applicability of our results to other Natural Language Generation tasks (ie, Question/Answering) and health systems would require additional testing for external validation.

Author contributions

Emma Croxford (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing—original draft, Writing—review & editing), Yanjun Gao (Conceptualization, Funding acquisition, Investigation, Methodology, Supervision, Writing—review & editing), Nicholas Pellegrino (Conceptualization, Investigation, Methodology, Supervision, Writing—review & editing), Karen Wong (Conceptualization, Data curation, Investigation, Methodology, Supervision, Validation, Writing—review & editing), Graham Wills (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Supervision, Writing—review & editing), Elliot First (Conceptualization, Investigation, Methodology, Project administration, Supervision, Writing—review & editing), Miranda Schnier (Project administration, Writing—review & editing), Kyle Burton (Data curation, Writing—review & editing), Cris Ebby (Data curation, Writing—review & editing), Jillian Gorski (Data curation, Writing—review & editing), Matthew Kalscheur (Data curation, Writing—review & editing), Samy Khalil (Data curation, Writing—review & editing), Marie Pisani (Data curation, Writing—review & editing), Tyler Rubeor (Data curation, Writing—review & editing), Peter D. Stetson (Supervision, Writing—review & editing), Frank Liao (Project administration, Resources, Supervision, Writing—review & editing), Cherodeep Goswami (Project administration, Resources, Supervision, Writing—review & editing), and Brian W. Patterson (Conceptualization, Data curation, Investigation, Methodology, Project administration, Supervision, Validation, Writing—review & editing), Majid Afshar (Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing—review & editing)

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was supported by the National Institute of Health/National Library of Medicine grant numbers 5T15LM007359, R00 LM014308-02, and R01LM012973.

Conflicts of interest

The authors have no competing interests to declare.

Data availability

The data underlying this article are available in our GitLab repository, at https://git.doit.wisc.edu/smph-public/dom/uw-icu-data-science-lab-public/pdsqi-9/.

References