The relationship between electronic health records user interface features and data quality of patient clinical information: an integrative review

Abstract

Objectives

Electronic health records (EHRs) user interfaces (UI) designed for data entry can potentially impact the quality of patient information captured in the EHRs. This review identified and synthesized the literature evidence about the relationship of UI features in EHRs on data quality (DQ).

Materials and methods

We performed an integrative review of research studies by conducting a structured search in 5 databases completed on October 10, 2022. We applied Whittemore & Knafl’s methodology to identify literature, extract, and synthesize information, iteratively. We adapted Kmet et al appraisal tool for the quality assessment of the evidence. The research protocol was registered with PROSPERO (CRD42020203998).

Results

Eleven studies met the inclusion criteria. The relationship between 1 or more UI features and 1 or more DQ indicators was examined. UI features were classified into 4 categories: 3 types of data capture aids, and other methods of DQ assessment at the UI. The Weiskopf et al measures were used to assess DQ: completeness (n = 10), correctness (n = 10), and currency (n = 3). UI features such as mandatory fields, templates, and contextual autocomplete improved completeness or correctness or both. Measures of currency were scarce.

Discussion

The paucity of studies on UI features and DQ underscored the limited knowledge in this important area. The UI features examined had both positive and negative effects on DQ. Standardization of data entry and further development of automated algorithmic aids, including adaptive UIs, have great promise for improving DQ. Further research is essential to ensure data captured in our electronic systems are high quality and valid for use in clinical decision-making and other secondary analyses.

Background and significance

Electronic health records (EHRs) include information about patients’ health status, care decisions, care received, and care underway. Such information is often used by clinicians (nurses, doctors, and other health professionals) to support decision making in day-to-day care. When aggregated through secondary analyses, it can serve as the foundation for delivering robust clinical decision support (CDS) in the form of alerts and recommendations to clinicians at the point of care.¹ High-quality CDS has been shown to increase the effectiveness of clinical decision-making and reduce documentation time.² For EHR data to be genuinely useful to clinicians in day to day practice and as meaningful CDS, however, the data collected (eg, assessments, medication administration,³ progress notes,⁴ and nursing care plans⁵) must be of high quality. Though there are no standard definitions, many experts agree that the main dimensions of data quality (DQ) are completeness, correctness, and currency.^6–10 Data are: (1) complete if the expected patient data are present;^7,8 (2) accurate or correct if the data are true, truly reflective of real observations, in agreement with other relevant elements or coherent with what the element is measuring;^7,8 and (3) current if the data were recorded in the time frame of interest.^6,7,9,10

The design and features of the EHR user interface (UI), the computer screens with which the user interacts,^11–13 have a potential impact on the quality of data gathered in the EHRs.^12,14,15 A well-designed UI should facilitate the generation of high-quality EHR data by providing the content, features, and functionality that support clinicians to accurately document desired data.^9,14,16 The requisite components of the UI needed to produce high-quality data vary based on the purpose and type of data being documented (eg, care planning information, vital signs, discharge status) and the structure of the data (eg, structured, unstructured data, or both).¹⁴

Recently, concerns have been raised about the unintended consequences of poorly designed EHR UIs.^11,17–19 Though the application of usability metrics has become popular to limit interface design flaws, improve the user experience, and reduce errors, these metrics do not routinely include explicit measures assessing the impact on DQ.^20–22 Poorly designed UIs can cause data entry errors, failure to capture needed information, and dissemination of inaccurate information potentially resulting in dire consequences for patients.^12,23 Further, use of these inaccurate data in CDS algorithms will result in generating invalid CDS^3,15,24,25 that may go undetected. Metrics continue to evolve for evaluating the DQ at the backend (database side) but these do not adequately address the errors that can occur at data entry or UI side.⁷ The importance of understanding the relationship between the EHR UI data collection features and the quality of data captured in the EHR database is just now gaining attention.¹¹ Since the data collected in EHRs are used in care decisions of all types that directly influence patient outcomes, it is crucial to determine what is known about the relationship between UI features and the quality of data collected. Although a positive usability assessment is evidence of user acceptance of a UI, it does not fully address the relationship between the UI and DQ. Below we report the results of our comprehensive integrated literature review to address this gap.

Objective

The specific aim of this integrative review is to synthesize findings from primary research studies to better understand what is known about the relationship between EHR UI features and the quality of data.

Materials and methods

This was an integrative review following the updated methodology proposed by Whittemore and Knafl.²⁶ The identification of the literature; extraction of the concepts, patterns, and relationships between UI features and DQ; and synthesis involved multiple iterations (PROSPERO identification number: CRD42020203998).²⁷

Literature sources and search strategy

Our reproducible search strategy was developed with the assistance of a health sciences librarian (M.A.) with no set begin date and an end date of October 10, 2022. We searched 5 databases (EMBASE, CINAHL, Web of Science, PubMed, and APA PsycInfo) using a common strategy for all but adapting it to the nuances of each database. The search strings thus varied by database, but all included the key concepts of “user interface,” “electronic health records,” and “data quality” connected with logical operators “and” and their equivalent terms connected with “or.” We downloaded the records from the search results into an electronic reference manager (EndNote version X9) and systematic review production tool (Covidence). Appendix S1 displays the details of the database search strategies.

Inclusion and exclusion criteria

Our inclusion criteria for selecting eligible studies were: (1) peer-reviewed, (2) written in English, (3) focused on the UI data entry features and functionality of systems (eg, EHRs) that captured health care information, (4) included clinicians (eg, nurses, physicians, and other health providers), and (5) included at least 1 measure of DQ. Exclusion criteria were: (1) opinion articles, case studies, conference abstracts, letters, newspapers, and editorials; (2) studies without clinicians; (3) studies that focused on data entry with no DQ component, and (4) studies that examined DQ with no data entry component. We did not exclude articles based on dates or study design.

Screening and selection

Following the literature search, all identified articles were downloaded and duplicates removed. The title, abstract screening, and full-text reviews were performed independently by individual reviewers in 2 different dyads; (O.O.M.) and (F.D.S.) reviewed nearly 95% of the articles with (O.O.M.) and (G.M.K.) reviewing those remaining. The cumulative interrater reliability was 95.2% with a Cohen Kappa of 0.45 indicating moderate agreement. Disagreements were settled through discussion during the abstract and title review phase. When consensus could not be reached, the article was moved to full-text review. The full-text review process was conducted similarly by both dyads with disagreements being settled through consensus and when needed by a third reviewer (R.I.B.).

Extraction of review article information

For each of the included articles, we extracted: (1) reference information, (2) purpose of study, (3) description of the UI, (4) study design, sample, data source, and study location, (5) measures of DQ and usability (if assessed), (6) results, and (7) the study quality assessment rating. The extraction was iterative and validated by 4 co-authors (O.O.M., R.I.B., G.M.K., and Y.Y.) to ensure the desired content was captured clearly and accurately. Based on the content extracted, we assessed the quality of each study article.

The DQ measures identified in each review article were classified into categories identified by Weiskopf et al,⁸ 3 of which are identified as core by most experts: (1) completeness: if the expected patient data are present;⁸ (2) correctness: if the data are accurate, truly reflective of real observations, in agreement with other relevant elements or coherent with what the element is measuring;⁸ and (3) currency: if the data were recorded in the time frame of interest.^7,8 All studies included a UI description (an inclusion criteria).

Appraisal of the quality of studies

We adapted Kmet, Lee and Cook quality assessment tool²⁸ to evaluate the quality of the studies (see Table 1). The updated version of the risk bias quality assessment tool helped us focus on rating the strength of the elements of the studies of interest to us. The 4 areas examined included: (1) system description (1 item), (2) usability assessment (3 items), (3) DQ assessment (3 items), and (4) study design and findings (3 items). Two raters (O.O.M. and R.I.B.) used the quality assessment tool to independently evaluate the studies. Raters reached a consensus after discussing their differences in scores, and the third rater (G.M.K.) reconciled disagreements. Each of the 10 items could be scored on a 0-2 scale with a possible total score range of 0-20, with higher scores indicating higher quality of studies for the purposes of this review.

Results

Our search yielded 1,323 studies, with 30 studies retained to full-text review. Of those, 11 met the inclusion criteria and were included as final studies in the review. The reviewing process and results are depicted in Figure 1.

Study characteristics

The 11 included studies from 9 different countries across 3 continents. Five studies^29–33 came from North America (Canada [n = 1] and United States [n = 4]), 4 from Europe^34–37 (Denmark, Sweden, Turkey, United Kingdom, n = 1 in each), and 2 from Asia^38,39 (China, Israel, n = 1 in each). Seven of the studies were comparative studies of UIs^{29,30,33,36,37,39} or data entry processes,³¹ comparing DQ before and after an update implementation. The remaining 4 were cross-sectional studies, examining DQ without a pre-post comparison.^32,34,35,38 Three studies^32,34,35 were reported to have been underpinned by a theoretical model or conceptual framework. One of them followed the Weiskopf & Wang DQ assessment framework,³⁴ the second applied a usability engineering perspective in the analysis phase,³² and the third was rooted in narrative theory and distributed cognition.³⁵

Eight studies were quantitative,^29,31,33,34,^36–39 2 were qualitative,^32,35 and 1 used mixed methods.³⁰ The study designs included randomized control trial,³⁹ experimental design,³³ pre-post comparison,^29,37 retrospective chart review,^31,38 descriptive,³⁴ within-participant study,³⁶ mixed methods,³⁰ qualitative analysis,³² and ethnographic study.³⁵ Samples of clinicians were small ranging from 6 to 85 with one of the studies having a sample drawn from more than 1 system or location (sites = 5).³⁵ Patient record data were included in 10 studies.^{29–34,36–39} All but one³⁴ of the studies that utilized subsets of actual patient record data (cases) extracted these from a single system or utilized hypothetical scenarios.^36,39 In 5 studies,^29,30,^37–39 1 or more aspects of usability (eg, task completion, time required for task, user satisfaction, keystrokes) were also assessed. Usability measures included those locally developed or previously validated instruments such as the System Usability Scale.^22,40Table 2 displays details of study characteristics and extracted evidence in the Supplementary Material.

Data quality measures

Per our criteria, all 11 studies^29–39 included at least 1 measure of DQ and provided rich descriptions of the UIs and/or screen layouts. Nine studies^{29–32,34–36,38,39} assessed 2 or more DQ measures and 5 studies reported usability assessments.^29,30,^37–39

Completeness was examined in 10 studies.^{29–36,38,39} Across the 10 studies, completeness was analyzed as the percentages or proportions of missing^30,33,38 or complete fields;^29,31,34 counts of complete fields;³⁹ and/or the mean/average number missing,³³ complete³⁶ patient information, or perceptions of completeness by nurses³² or physicians.³⁵

Correctness was examined in all but one.³³ Within the construct of correctness, 3 studies explored concordance^34,36,38 (matching information across anesthesia records, concordance across 3 systems, amount of unrelated diagnostic information documented), 4 examined accuracy by comparing expert designed sources (predictive modeling,³⁹ data modeling,³⁷ templates,^29,31), and 3 reported user perceptions of correctness or accuracy.^30,32,35

Lastly, currency was addressed in 3 studies. One study evaluated currency by assessing the time between entry to emergency room to documentation of vital signs.³⁴ The 2 qualitative studies provided user perceptions; one described currency contrasting documentation done at the point of care visit versus elsewhere after the visit³² and the second in terms of chronological coherence.³⁵

UI features and DQ

In all 11 studies, at least 1 UI feature was identified and expected to impact 1 or more of the 3 core quality measures. As no a priori categorization of UI features linked to DQ was found, we inductively derived categories from our findings in this review and classified these UI features into 1 of 4 categories based on the mode of data capture aid: (1) mandatory data capture aids;^33,38 (2) nonmandatory data capture aids;^29,31 (3) automated algorithmic data capture aids,^30,36,37,39 and (4) other ways to relate UI features and DQ.^32,34,35 Below we report the findings by group category.

Mandatory data capture aids

This feature requires users to complete data fields before being allowed to move to the next task. Two studies reported the use of mandatory fields. In 1,³⁸ the feature was used to capture important anesthesia details. Completeness after 12 months was at 99.6% with correctness measured through concordance with significant associations (P<.0001) between age and tracheal tube, laryngoscope, and catheter sizes. In the second study, as an alternative to paper entry, clinicians were provided an electronic tablet to enter key information on patients with psoriatic arthritis that disallowed submission until all highlighted fields were completed. The authors reported improvement in completeness while increasing documentation time.

Nonmandatory data capture aids (checklists and templates)

This category included 2 studies comparing free text data capture to data entered using a checklist²⁹ or template.³¹ Checklists and templates of data items are typically evidence based and set-up as memory aids to remind users to adequately characterize, deliver, and document appropriate care. In 1 study,²⁹ researchers sought solutions to reduce missing vaccine doses in a multistep immunization data entry process.²⁹ By switching to a single-point data entry system with selections from predefined standardized vaccine information lists, they improved the completeness and correctness by reducing missing doses from 37.3% to 0%.²⁹

A second study addressed completeness of pediatric orthopedics data by replacing nontemplate (unstructured) forms with template forms for structured documentation of clinical examination.³¹ After implementing template forms, the completeness of patient history and physical documentation rose from 10% to 67% (P < .001). No significant difference was found between the proportions of accurate documentation of nerve palsies (assessed through comparison with attending’s notes) in the template form (84%) versus the nontemplate forms (93%) (P = .16).³¹

Automated algorithmic data capture aids

There were 4 studies in this category representing different purposes and varying degrees of complexity in the type of model or algorithms used to support DQ. Three studies focused on capturing structured data^30,36,39 with one³⁹ also including unstructured data capture. One used a nationally recognized standardized terminology by mapping data to a domain-specific ontology.³⁰ In these studies, predictive algorithms that generated auto-suggestions were used to enhance the completeness and accuracy of patient data entered: for falls,³⁹ presenting problems,³⁰ and symptoms and diagnoses.³⁶ Completeness was significantly higher in the Hua and Gong³⁹ study on unstructured falls data (34%, P = .000) in the treatment (auto-suggestions) versus control group (no auto-suggestions). Significant differences in completeness of structured data capture were found for the treatment groups who received auto-suggestions for problems in the Greenbaum et al³⁰ study (P <.0001) and for problems and symptoms in the Kostopoulou et al³⁶ study (P < .001) versus the groups receiving none. Correctness was also examined in all 3 studies.^30,36,39 In the falls study, the treatment group (cueing lists) had significantly higher correct structured data capture (3.8%, P = .000)³⁹ compared to those receiving no cueing lists. In the study on symptoms and diagnosis, the treatment group had significantly higher correctness (P < .001)³⁶ than the control group measured as amount of unrelated diagnoses data entry. In the presenting problems study no significant differences, as measured by user perceptions, were found between the treatment and control groups.³⁰

In the fourth study, Kuru et al³⁷ examined the reporting of esophagus related data by radiologist. For the treatment group, in-line editing based on the Structured, Interactive, Standardized, and Decision Supporting Method (SISDS) developed model and algorithm, was provided to enhance the correctness of data captured.³⁷ A success rate of 81.25% in capture of correct diagnosis was achieved with the SISDS group compared to 43.75% for the transcriptionist control group.³⁷

Other ways to relate UI features and DQ

There were 3 studies in this category, all examining completeness, correctness, and currency.^32,34,35 In 1 study, 3 vital sign data entry formats were compared across 5 institutions.³⁴ The other 2 are qualitative studies; one assessed DQ of first home visit documentation in 1 EHR;³² and the second examined physician perceptions of DQ of content in an EHR overlay screen.³⁵

In the first study, Skyttberg et al³⁴ compared DQ of 3 different formats; paper-based, mixed (paper transferred to electronic), and electronic of vital signs of data entry from 5 emergency rooms. The percentage of vital signs captured (completeness) was highest in the mixed (M) documentation format (95%), followed by electronic (E) data entry (62%-71%), and finally paper-based (P) entry (1%-2%).³⁴ Correctness was measured as concordance with vital sign norms and found to be high across all 3 formats. Currency was measured as the time between admission to EHR and entry of vital signs into the record (with “15” was the desired benchmark).³⁴ Paper-based and mixed had low currency and the electronic format had medium currency.³⁴

The 2 qualitative studies in the review focused on nurses³² and physicians³⁵ perceptions about UI features that influence DQ. In the Yang et al³² study, home care nurses from 1 agency were queried about their reasons for delaying full documentation of a patient’s first home visit until a later time and place away from the patient. The nurses indicated that documentation requirements and system design prevented complete data entry at the point of care (compromised currency). Work arounds were created and used as memory aids (eg, entering some key data into EHR and creating written notes) for the purpose of ensuring completeness and correctness at the time of delayed data entry.³²

In the second qualitative study Jensen et al,³⁵ conducted ethnographic research with a small sample of physicians from 5 hospitals to examine reasons for use and nonuse of the UI overlay screen. The overlay was originally designed to provide a comprehensive view of patient’s key data points extracted from various parts of the EHR. The study was motivated by learning that there were a surprisingly low percentage of users. The findings indicated the need to consider the complexity of the patient population and physician specialty when designing UI features intended to deliver high-quality data. Some found the overlay sufficiently complete, correct, and current. Other physicians found the data incomplete, inaccurate, and lacking in chronological coherence resulting in the lack of trust and nonuse of the overlay.³⁵

Quality appraisal of studies

Quality Scores ranged from 4 to 16 (total possible = 20), with one of the studies ranking at high quality,³⁹ 7 at moderate^{29–31,33,36–38} quality and 3 studies at low^32,34,35 quality (Table 3).

Discussion

Given the growing emphasis on secondary uses of EHR data (eg, primary data compiled into multidimensional reports, CDS, research etc.) and availability of powerful analytical tools and methods (eg, artificial intelligence), it is crucial to ensure high quality of the data entered into the EHR. As noted, this integrative review of 11 studies represented a wide range of study designs, rigor, UI features, data types involved, and measures of DQ. The designs of the 11 studies varied from descriptive qualitative studies to 1 RCT with the majority involving comparative designs. Clinician sample sizes across the studies were small with the data source being unique to each study. Measures used to assess DQ were diverse, making clear comparisons across studies difficult. Limited sample sizes and varying quality of evidence hinder our ability to make inferences about the relationship between particular UI features and quality of data. To ensure robust findings that are generalizable, future research should emphasize randomized controlled trials with powered samples to investigate the impact of EHR UI features on DQ. There are nonetheless a number of important findings about DQ and the relationship between UI features and DQ that can be drawn from this review.

As noted, the reviewed studies examined DQ measures consistent with the core set of correctness, completeness, and currency identified in the literature.^6,7 However, authors in our review used a variety of measures to assess these categories in order to accommodate the diverse data sources. Several researchers also used indirect proxies as validation when direct observation was not possible. For example, tube size could be validated as correct by a second observer or by use of concordant measures in which the documented tube size is considered valid if it matches the range of sizes recommended for the patient’s age.³⁸ While it makes sense to tailor the measures to the differing data sources, some standardization of measures is necessary to enable comparisons across similar datasets. This requires development or validation of standardized measures with high precision while ensuring that the context of data is adequately captured.

Currency, though recognized as a pivotal measure as indicated in the literature,^6,7 was assessed in only 3 of the 11 studies in this review.^32,34,35 This suggests the potential lack of robust interface (UI) features to ensure data currency. Knowledge of the time or time period covered for the events entered into the record is essential to sequencing events and evaluating impact on outcomes over time. Thus, systems should support its capture. As was reported, there is often a delay in reporting information due to context factors (eg, need to give comprehensive hands on care to multiple patients before charting on them). Factors such as patient workload and acuity also delay timely entry.⁴¹ One potential solution could be to expand use of equipment at the point of care that automatically collects and transfers data and the time of capture to the EHR (eg, vital signs, cardiac rhythms, and intravenous drips).⁴¹ Another is to use voice activated charting which could allow the clinician to document key information into the computer while caring for the patient.⁴² While automation might not suit all data types, we should strive to develop approaches that enable instant quality data capture at the point of care. These innovative UI features would also ensure that data-driven clinical decision-making is based on the current condition of the patient at the time of intervention.

Consistent with findings from a recent review on the practice of DQ evaluation, completeness and correctness were commonly assessed.⁷ Of the 11 included studies in this review, 10 studies assessed completeness and 10 assessed correctness with 9 evaluating both. As such, completeness and correctness stood out as pivotal DQ measures. UI features such as mandatory fields (forced data entry)^33,38 and predefined templates³¹ and picklists²⁹ (limited set of responses to check) were reported to be helpful in capturing complete and correct data. However, these features frustrated some clinicians; the former increased documentation time,³³ and the latter often did not contain the desired responses.^29,31 The interface overlay, a feature designed to present a valid general summary (completeness, correctness, and currency) of the patient’s circumstance to the clinician, was found to be very helpful to some specialties and unacceptable to others.³⁵ The finding reinforces the need to tailor DQ interface features to the dataset and user characteristics.

The potential benefits of using structured and standardized terminologies at the UI to support data correctness were highlighted in the included studies in this review. For example, consistent use of the same measures, such as vital sign measurements that are recorded in the exact same way, was found to support easy interpretation and comparison of these data within and across organizations.³⁴ This avoids the need for others who did not enter the data to standardize them for multiple uses through mapping, a process that will naturally compromise correctness. Use of nationally accepted standardized terminologies at the UI also has the potential to enhance correctness of data entered since responses sets for an element are limited to terminologies that have been systematically developed for the purpose. Such use further supports the ability to easily compare data with all others who utilize the same standardized terminologies, a national goal for making health data interoperable.⁴³

Auto-suggestion and autocomplete were UI functionalities that emerged as promising for enhancing data completeness and correctness, especially when powered by predictive models and standardized terminologies. Greenbaum et al’s study serves as evidence of the power of combining predictive algorithms with nationally recognized terminologies. Thus, the development and adoption of standardized data elements, especially when combined with intuitive UI features, promise significant enhancements in DQ measures and set the stage for more rigorous data-driven methodologies in future research.

It is important to underscore again, however, that UI features and measurements of correctness and completeness will depend on the data sources and the complexity of them. For instance, in Kostopoulou et al³⁶ study, the researchers added fields for capturing details of the patient’s situation to eliminate errors that occurred with an earlier version of an automated diagnoses selection aid. As a result, the team now uses the amount of other details as a proxy for assessing correctness of diagnosis. Another example is correctness being assessed by user perceptions. An applicable example is clinician perceptions about the correctness of their fellow clinician’s data entry captured in the EHR.

In general, we believe that automated algorithmic data capture features that utilize standardized terminologies show the most potential for improving data correctness^30,36,37,39 and completeness.^30,36,39 This is because these features reduce the need for recall while using data entered to predict and deliver most likely responses for the data fields. Automated algorithmic aids including contextual autocomplete and auto-suggestions are typically driven by predictive models that enhance capture of clinical information within a single context of use (eg, entry of a standardized patient chief complaint, enhancing data entry of fall patients). These features have commonalities with the robust adaptive UIs (AUIs) that are gradually gaining traction in healthcare systems.^44–47 AUIs are interactive systems designed to preserve usability of UIs across various contexts of use (user, platforms, and environment) based on adaptation rules.^11,48 Similar to the automated algorithmic aids they also utilize algorithms and datasets to enhance UI features for data entry. AUI features are more sophisticated than those found in our studies because they carefully consider variability in context of use. These interfaces are expensive to develop because they include multiple parameters such as cognitive complexity, behavioral predictability, predictive accuracy, users’ autonomy to opt out of adaptation rule, users environment, and interface elements that accommodate both novice and experts.^11,48,49 Their UI features focus on ensuring a high degree of data correctness, speed of data capture.^48,49 Thus, AUI could enhance DQ in healthcare, especially when augmented with generative artificial intelligence (AI) models such as Large Language Models to analyze complex heterogeneous data, generate auto-suggestions that could influence complete, correct, and current data. Further research is needed to explore the relationship between AUIs and DQ.

In summary, the studies collectively provide preliminary evidence of the crucial impact UI features can have on DQ. There are features, such as automated algorithmic capture aids, that have the potential to enhance completeness, and correctness. However, little is known about their impact on currency. This underdeveloped area is critical for clinical decision making, quality improvement, and research. Ultimately, further research is required to understand impact of UI features on DQ.

Limitations of the study

We identified several limitations of this review. First, we might have missed identifying important research studies in the 5 databases searched due to the unique features involved in completing advanced searches (eg, variations in registered terminologies such as Medical Subject Headings [MESH] and keywords). The permutation of keywords to create search strings may have also introduced some bias and resulted in the omission of relevant publications. Finally, we may have failed to identify all of the appropriate terms that characterize this area of science, also causing us to potentially miss applicable published studies.

A second limitation was the use of an untested tool for our risk assessment. When no appropriate tool was found, the Kmet et al²⁸ tool was modified to help rate study facets pertaining to this review. A third limitation was the difficulty we encountered in synthesizing the study findings due to the wide variations in quality and content of the 11 studies. As a result, we may have inadvertently missed important points. To address this issue, we worked hard to make all tables clean and concise and included Table 4 to simply illustrate important study points.

Conclusion

In summary, we have identified key features and elements ranging from simple to complex automated data capture aids, that support the capture of high-quality EHR data relevant for purposes such as making care decisions, using predictive modeling to generate care suggestions, health care research. However, this area of research is underdeveloped, and gaps exist in the relationship between UI features and DQ. Therefore, we encourage researchers to identify ways the tools and research strategies uncovered in this review can be used to ensure one’s EHR dataset of interest (eg, care planning data, diversity inclusion and equity data, emergency room visits, epidemiological data) is capturing complete, correct, and current data. We believe a clear priority for future work is to carefully examine and validate ways UI features can be adapted to meet user’s needs while ensuring the efficient capture of complete, correct, and current data.

Author contributions

O.O.M. played a significant role in the creation and development of the research project. Along with R.I.B. and G.M.K., they collaborated in all stages of the study, including the design of methodology and refinement of data extraction. Y.Y. further improved the data analysis and extraction of evidence. O.O.M., R.I.B., and G.M.K. interpreted the results. F.D.S. and G.M.K. assisted with abstract and full-text review. O.O.M., G.M.K., and M.A. analyzed the search strategies and ensured reproducibility of the methodology. O.O.M. drafted the initial manuscript, with further revisions and completion by all authors. All authors were involved in critically analyzing the validity of the intellectual content in the final manuscript.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This research was supported by the National Institute of Health (NIH), the National Institute of Nursing Research (NINR) grant award (R01 NR018416/NR/NINR NIH HHS/United States). The author is responsible for the content of this research and does not necessarily represent the official view of NIH or NINR.

Conflicts of interest

None declared.

Data availability

The data extracted from the 5 databases used in this review are available on reasonable request.

References