Abstract
The objective of the study was to determine, after medication review, the patient risk score threshold that would distinguish between stays with prescriptions triggering pharmacist intervention (PI) and stays with prescriptions not triggering PI.
The study was retrospective and observational, conducted in the clinical pharmacy team. The patient risk score was adapted from a Canadian score and was integrated in the clinical decision support system (CDSS). For each hospital stay, the score was calculated at the beginning of hospitalization and we retrospectively showed if a medication review and a PI were conducted. Then, the optimal patient risk score threshold was determined to help pharmacist in optimizing medication review.
During the study, 973 (56.7%) medication reviews were performed and 248 (25.5%) led to a PI. After analyzing sensitivity, specificity, and positive predictive value of different thresholds, the threshold of 4 was deemed discriminating to identify hospital stays likely to lead to a PI following a medication review. At this threshold, 600 hospital stays would have been detected (33.3% of the latter led to a PI), and 5.0% of stays with a medication review would not have been detected even though they were hospital stays that had triggered a PI.
Integration of a patient risk score in a CDSS can help clinical pharmacist to target hospital stays likely to trigger a PI. However, an optimal threshold is difficult to determine. Constructing and using a score in practice should be organized with the local clinical pharmacy team, in order to understand the tool’s limitations and maximize its use in detecting at-risk drug prescriptions.
Lay Summary
Our study investigates the use of a patient risk score integrated in a clinical decision support system (CDSS) for pharmacists. Clinical decision support systems are tools that detect in real time at-risk situations. Clinical pharmacist daily reviewed drug prescriptions, and we aimed at developing a patient risk score to alert on prescriptions which is more likely to trigger a pharmacist intervention to medical ward. The use of a score could help clinical pharmacist to prioritize themselves by maximizing their capacity to detect at-risk drug prescriptions, but finding the optimal threshold is complex and need to be improved to be used in practice. This research provides a first work to develop a score in a CDSS to help clinical pharmacist in medication review. Collaboration with the CDSS’s developers is required to evaluate the routine use of the patient risk score by clinical pharmacy teams.
Introduction
Medications generally increase a patient’s life expectancy and improved his/her quality of life. However, according to a study performed in 2008, the estimated annual cost linked to adverse drug reactions (ADRs) in Europe was 79 billion euros.1 Moreover, ADRs are thought to be responsible for around 197 000 deaths per year and occur in about 10% of hospital inpatients.1,2 The impact and management of ADRs are complex, and the costs associated with the resulting hospital admissions or prolongations of ongoing hospital stays are nonnegligible. Clinical pharmacists help to make health-care products safer, more relevant, and more effective. The growth of clinical pharmacy activities means that staff need access to the patients’ health records and the clinical information therein. The clinical pharmacist’s missions include medication reviews and, if required, pharmacist interventions (PIs) following a drug prescription by a physician. A medication review is a structured analysis of a patient’s drug prescription, aiming to optimize the prescription by focusing on the achievement of therapeutic objectives, the detection of adverse effects and the patient’s adherence to medication. Following this medication review, a PI can be provided to the medical team to modify the treatment of one or more heath products. A PI is defined as any action initiated by a clinical pharmacist in order to change the patient’s therapeutic management (eg, substitution of a drug in case of drug-drug interaction or discontinuation of potassium supplementation in the event of hyperkalemia).3 Type and impact of PIs are structured by the French Society of Clinical Pharmacy.4 It has been shown that PIs are associated with lower ADR and mortality rates.5 With the growing number of hospital inpatients and the development of electronic health records (EHRs), the medication review process has to be optimized. Clinical decision support systems can help to optimize drug prescriptions. Indeed, the use of a CDSS is associated with higher ADR detection and prevention rates.6–10 One approach to optimization is the calculation of a score that identifies the hospital stays that are most likely to trigger a PI: this score would then enable pharmacists to optimize medication reviews. Although many such tools have been developed, few can be integrated into a CDSS.11–14 Among the patient risk scores found in the literature, one developed in Canada is usefully based on variables that can be integrated into a CDSS.15,16 The patient risk score is used to identify patients with prescriptions likely to result in an ADR and for whom a medication reconciliation on admission is necessary. We looked at whether incorporating and adapting this patient risk score into a CDSS would help us to target prescriptions requiring attention from the clinical pharmacist. Our primary endpoint was the issuance (or not) of a PI.
Hence, the objective of the present study of clinical pharmacy activities at Lille University Hospital (Lille, France) was to determine the patient risk score threshold that would distinguish between stays with prescriptions triggering a PI and stays with prescriptions not triggering a PI.
Materials and methods
Ethical approval
In line with the French legislation on retrospective studies of clinical practice, the study protocol was approved by a hospital committee (Lille University Hospital, Lille, France; reference: 1345) with competency for research not requiring approval by an independent ethics committee.
Study design
The present retrospective, observational study was conducted between March 2 and April 16, 2022, at Lille University Hospital. The study covered 905 hospital beds on nonsurgical wards for which EHRs were available and in which a clinical pharmacist was present (the general medicine, postacute care, rehabilitation, and psychiatry departments). The clinical pharmacy team comprised 18 clinical pharmacists and 6 junior residents. Each clinical pharmacist performed various tasks in several wards during office hours from Monday to Friday, including patient interviews about specific drugs and medication reviews on admission or on discharge (via the patient’s EHRs). These activities sometimes triggered PIs, which were sent to the medical team. The PIs were logged in the hospital’s computerized physician order entry system (CPOE, Sillage, SIB, Rennes, France). The medical staff’s responses to the PIs were also logged in the CPOE system.
In July 2019, the clinical pharmacy team decided to use a CDSS (PharmaClass from Keenturtle, Paris, France) to help them with medication reviews in their routine clinical practice. This CDSS is not intended for use by prescribers but assists pharmacists with medication review. Thanks to previously coded alert rules, the CDSS can detect certain at-risk prescriptions in real time. The CDSS is separate from the CPOE system and is being developed as part of an ongoing collaboration between the clinical pharmacy team and the software house. We therefore decided to jointly develop a patient risk score module as part of the CDSS.
Development of the patient risk score module
Design of the score
The new patient risk score (PharmaScore) was an adaptation of a score that had already been developed at the Centre Hospitalier Affilié Universitaire de Québec (Québec, Canada).15 The PharmaScore module can only be consulted via PharmaClass software (Keenturtle). Only the project team’s pharmacists could consult the patient risk score; the other clinical pharmacists were blinded to the module’s output.
The score depends on the patient’s age, the number of drugs prescribed by the clinician on admission of the patient, and the number and types of certain drug classes (antiepileptics, cancer drugs, anticoagulants, antidiabetics, and cardiovascular drugs) prescribed upon admission (ie, the same as the drugs being taken at home immediately prior to admission) (Table 1). Each criterion is weighted to give a final score ranging from 0 (the lowest risk) to 21 (the highest risk).
Calculation of the patient risk score included in the CDSS.
| Criteria | Points | |
|---|---|---|
| Age | ≤74 years of age | 0 |
| 75-84 | 1 | |
| ≥85 | 2 | |
| Number of drugs prescribed on admission | ≤3 | 0 |
| ≥4 and ≤6 | 2 | |
| ≥7 | 4 | |
| ≥1 drug from the following ATC classes anticoagulants | ATC B01AA—vitamin K antagonists | 3 |
| ATC B01AB—heparin group | ||
| ATC B01AE—direct thrombin inhibitors | ||
| ATC B01AF—direct factor Xa inhibitors | ||
| ATC B01AX—other antithrombotic agents | ||
| ≥3 drugs from the following ATC classes cardiovascular drugs | ATC B01AC—platelet aggregation inhibitors, excluding heparin | 5 |
| ATC C—cardiovascular system | ||
| ≥1 drug from the following ATC class diabetes drugs | ATC A10—drugs used in diabetes | 2 |
| ≥1 drug from the following ATC classes cancer drugs | ATC L01—antineoplastic agents | 3 |
| ATC L02—endocrine therapy | ||
| ATC L03—immunostimulants | ||
| ≥1 drug from the following ATC class antiepileptics | ATC N03—antiepileptics | 2 |
| Criteria | Points | |
|---|---|---|
| Age | ≤74 years of age | 0 |
| 75-84 | 1 | |
| ≥85 | 2 | |
| Number of drugs prescribed on admission | ≤3 | 0 |
| ≥4 and ≤6 | 2 | |
| ≥7 | 4 | |
| ≥1 drug from the following ATC classes anticoagulants | ATC B01AA—vitamin K antagonists | 3 |
| ATC B01AB—heparin group | ||
| ATC B01AE—direct thrombin inhibitors | ||
| ATC B01AF—direct factor Xa inhibitors | ||
| ATC B01AX—other antithrombotic agents | ||
| ≥3 drugs from the following ATC classes cardiovascular drugs | ATC B01AC—platelet aggregation inhibitors, excluding heparin | 5 |
| ATC C—cardiovascular system | ||
| ≥1 drug from the following ATC class diabetes drugs | ATC A10—drugs used in diabetes | 2 |
| ≥1 drug from the following ATC classes cancer drugs | ATC L01—antineoplastic agents | 3 |
| ATC L02—endocrine therapy | ||
| ATC L03—immunostimulants | ||
| ≥1 drug from the following ATC class antiepileptics | ATC N03—antiepileptics | 2 |
Abbreviations: ATC, anatomical, therapeutic, and chemical; CDSS, clinical decision support system.
Calculation of the patient risk score included in the CDSS.
| Criteria | Points | |
|---|---|---|
| Age | ≤74 years of age | 0 |
| 75-84 | 1 | |
| ≥85 | 2 | |
| Number of drugs prescribed on admission | ≤3 | 0 |
| ≥4 and ≤6 | 2 | |
| ≥7 | 4 | |
| ≥1 drug from the following ATC classes anticoagulants | ATC B01AA—vitamin K antagonists | 3 |
| ATC B01AB—heparin group | ||
| ATC B01AE—direct thrombin inhibitors | ||
| ATC B01AF—direct factor Xa inhibitors | ||
| ATC B01AX—other antithrombotic agents | ||
| ≥3 drugs from the following ATC classes cardiovascular drugs | ATC B01AC—platelet aggregation inhibitors, excluding heparin | 5 |
| ATC C—cardiovascular system | ||
| ≥1 drug from the following ATC class diabetes drugs | ATC A10—drugs used in diabetes | 2 |
| ≥1 drug from the following ATC classes cancer drugs | ATC L01—antineoplastic agents | 3 |
| ATC L02—endocrine therapy | ||
| ATC L03—immunostimulants | ||
| ≥1 drug from the following ATC class antiepileptics | ATC N03—antiepileptics | 2 |
| Criteria | Points | |
|---|---|---|
| Age | ≤74 years of age | 0 |
| 75-84 | 1 | |
| ≥85 | 2 | |
| Number of drugs prescribed on admission | ≤3 | 0 |
| ≥4 and ≤6 | 2 | |
| ≥7 | 4 | |
| ≥1 drug from the following ATC classes anticoagulants | ATC B01AA—vitamin K antagonists | 3 |
| ATC B01AB—heparin group | ||
| ATC B01AE—direct thrombin inhibitors | ||
| ATC B01AF—direct factor Xa inhibitors | ||
| ATC B01AX—other antithrombotic agents | ||
| ≥3 drugs from the following ATC classes cardiovascular drugs | ATC B01AC—platelet aggregation inhibitors, excluding heparin | 5 |
| ATC C—cardiovascular system | ||
| ≥1 drug from the following ATC class diabetes drugs | ATC A10—drugs used in diabetes | 2 |
| ≥1 drug from the following ATC classes cancer drugs | ATC L01—antineoplastic agents | 3 |
| ATC L02—endocrine therapy | ||
| ATC L03—immunostimulants | ||
| ≥1 drug from the following ATC class antiepileptics | ATC N03—antiepileptics | 2 |
Abbreviations: ATC, anatomical, therapeutic, and chemical; CDSS, clinical decision support system.
The score was modified so that it reflected the drugs taken at home by the patient as accurately as possible, with the notable exclusion of intravenously administered drugs. For example, the number of drugs prescribed on admission tends to correspond to the number of drugs prescribed at home before admission. Hence, all drugs with an anatomical, therapeutic, and chemical (ATC) code were included (ie, international classification of medicines according to the organ or system on which they act, and to their therapeutic and chemical properties). Drugs administered intravenously (whether via a peripheral catheter or a central catheter) were excluded from the number of drugs prescribed on admission. Likewise, enoxaparin (2000 IU/0.2 mL and 4000 IU/0.4 mL) prescribed to reduce the thromboembolic risk in hospitalized patients was excluded from the number of drugs prescribed on admission and from the list of anticoagulants prescribed in hospital.
Calculation of the score
The score was calculated for each hospital admission. The score was calculated automatically between 12 and 36 h after the hospital admission, so that the physician had enough time to prescribe drugs at the start of the patient’s hospital stay. The eligibility criteria were age 18 or over and admission to a ward attended by a clinical pharmacist.
Data extracted for the study
The analysis was performed by hospital stay (not by unique patient, nor by drug prescription). The study data were extracted from the above-mentioned software tools. The following data were extracted from the PharmaScore module: the patient’s pseudonym identifier, the ward, the date on which the score was calculated, the score’s value, the patient’s age, and the details of each criterion for score calculation. The data were extracted for the period running from March 2 to April 2, 2022.
The following data on medication reviews and PIs were extracted from the CPOE: the patient’s pseudonym identifier, the ward, the date of the medication review, the date of the PI, the description of the problem encountered, the action taken (according to the ActIP rating developed by the French Society of Clinical Pharmacy [Société Française de Pharmacie Clinique]), and whether or not the physician had accepted the PI. The data were extracted for the period running from March 2 to April 16, 2022. Since the study’s objective was to identify a risk threshold for at least 1 PI, we decided to give the pharmacists 2 weeks after the PharmaScore calculation to log a PI in the CPOE.
Analysis of the scores
Firstly, the score was calculated for each patient aged 18 or over and having been admitted to a ward attended by a clinical pharmacist. Secondly, the patient’s EHRs were examined to determine whether or not the patient’s medications had been reviewed during the hospital stay. A patient’s medications might not have been reviewed if he/she had been admitted to hospital outside office hours or if the clinical pharmacist was pressed for time. Lastly, prescriptions having triggered at least 1 PI (regardless of whether the latter was accepted or not) were extracted from the CPOE.
A patient risk score threshold for prescriptions having triggered a PI was determined by comparing hospital stays comprising a medication review and at least 1 PI with hospital stays comprising a medication review but no PIs.
Statistical analyses
Statistical analyses were performed using R software (version 4.1.0, Posit Software PBC).17 Quantitative variables were described as the mean (SD) if normally distributed (as determined in the Shapiro-Wilk test) or the median (interquartile range) if not. Qualitative variables were described as the frequency (percentage).
Student’s t-test was applied to comparisons of quantitative variables. Chi-square test was applied to comparisons of qualitative variables. All tests were 2-sided, and the threshold for statistical significance was set to P < .05.
The relationship between the calculated patient risk score and issuance (or not) of a PI was evaluated using a receiver operating characteristic (ROC) curve of sensitivity vs 1-specificity. The area under the ROC curve (AUC) was determined, together with its 95% CI: the closer the curve is to the diagonal (ie, an AUC of 0.5), the worse the score’s ability to distinguish between 2 groups. By convention, an AUC’s discriminant power is rated as follows: 0.9-1: outstanding; 0.8-0.9: excellent; 0.7-0.8: acceptable, 0.6-0.7: poor, 0.5-0.6: very poor. We calculated the sensitivity (the probability of having a positive test result and a PI), specificity (the probability of having a negative test result and no PI), and positive predictive value (PPV, the probability of a PI when the test is positive) for each score threshold. The Youden index was calculated as a guide to the optimal threshold value, that is, the score above which an inpatient stay was likely to trigger a PI.
Results
Overall analysis
During the study period, 1717 scores were calculated for stays of eligible inpatients (Figure 1). A medication review was performed for 973 (56.7%) stays. Of the latter, 248 (25.5%) gave rise to a PI. Almost all the PIs (94.0%) were accepted by the medical staff.
Study flow chart.
For stays with a medication review and a patient risk score (n = 973), the mean patient age was 62.7 (21.3) and the mean score was 5.6 (4.6). For stays with a patient risk score but no medication review (n = 519), the mean patient age was 61.8 (15.9) and the mean score was 5.9 (4.9).
For stays with vs without a PI, there were significant differences between the PI and no PI groups with regard to the patient age, sex ratio, and overall score (Table 2). The proportion of patients aged 75 or over was 56.9% in the PI group and 28.3% in the no PI group. Over 60% of the stays in the PI group featured 7 or more drugs, and the most frequently observed drugs were cardiovascular drugs and anticoagulants.
Characteristics of the stays with a medication review (n = 973).
| Stays with a PI, n = 248 | Stays without a PI, n = 725 | P | |
|---|---|---|---|
| Characteristics of the groups | |||
| Age, mean (SD) | 72.3 (19.0) | 59.4 (21.1) | <.001 |
| Females, n (%) | 143 (57.7) | 358 (49.4) | .02933 |
| Overall score, mean (SD) | 7.4 (4.4) | 5.0 (4.5) | <.001 |
| Variable of the patient risk score | |||
| Age | <.001 | ||
| 18-74 years of age, n (%) | 107 (43.1%) | 520 (71.7%) | |
| 75-84, n (%) | 60 (24.2%) | 124 (17.1%) | |
| ≥85 years, n (%) | 81 (32.7%) | 81 (11.2%) | |
| Number of drugs prescribed on admission | <.001 | ||
| 0-3, n (%) | 29 (11.7%) | 247 (34.1%) | |
| 4-6, n (%) | 59 (23.8%) | 203 (28.0%) | |
| ≥7, n (%) | 160 (64.5%) | 275 (37.9%) | |
| High-risk drugs | |||
| Cardiovascular drugs, n (%) | 78 (31.5%) | 181 (25.0%) | .05591 |
| Anticoagulants, n (%) | 75 (30.2%) | 138 (19.0%) | <.001 |
| Diabetes drugs, n (%) | 60 (24.2%) | 141 (19.4%) | .133 |
| Antiepileptics, n (%) | 51 (20.6%) | 85 (11.7%) | <.001 |
| Cancer drugs, n (%) | 6 (2.4%) | 26 (3.6%) | .4945 |
| Stays with a PI, n = 248 | Stays without a PI, n = 725 | P | |
|---|---|---|---|
| Characteristics of the groups | |||
| Age, mean (SD) | 72.3 (19.0) | 59.4 (21.1) | <.001 |
| Females, n (%) | 143 (57.7) | 358 (49.4) | .02933 |
| Overall score, mean (SD) | 7.4 (4.4) | 5.0 (4.5) | <.001 |
| Variable of the patient risk score | |||
| Age | <.001 | ||
| 18-74 years of age, n (%) | 107 (43.1%) | 520 (71.7%) | |
| 75-84, n (%) | 60 (24.2%) | 124 (17.1%) | |
| ≥85 years, n (%) | 81 (32.7%) | 81 (11.2%) | |
| Number of drugs prescribed on admission | <.001 | ||
| 0-3, n (%) | 29 (11.7%) | 247 (34.1%) | |
| 4-6, n (%) | 59 (23.8%) | 203 (28.0%) | |
| ≥7, n (%) | 160 (64.5%) | 275 (37.9%) | |
| High-risk drugs | |||
| Cardiovascular drugs, n (%) | 78 (31.5%) | 181 (25.0%) | .05591 |
| Anticoagulants, n (%) | 75 (30.2%) | 138 (19.0%) | <.001 |
| Diabetes drugs, n (%) | 60 (24.2%) | 141 (19.4%) | .133 |
| Antiepileptics, n (%) | 51 (20.6%) | 85 (11.7%) | <.001 |
| Cancer drugs, n (%) | 6 (2.4%) | 26 (3.6%) | .4945 |
Abbreviation: PI: pharmacist intervention.
Characteristics of the stays with a medication review (n = 973).
| Stays with a PI, n = 248 | Stays without a PI, n = 725 | P | |
|---|---|---|---|
| Characteristics of the groups | |||
| Age, mean (SD) | 72.3 (19.0) | 59.4 (21.1) | <.001 |
| Females, n (%) | 143 (57.7) | 358 (49.4) | .02933 |
| Overall score, mean (SD) | 7.4 (4.4) | 5.0 (4.5) | <.001 |
| Variable of the patient risk score | |||
| Age | <.001 | ||
| 18-74 years of age, n (%) | 107 (43.1%) | 520 (71.7%) | |
| 75-84, n (%) | 60 (24.2%) | 124 (17.1%) | |
| ≥85 years, n (%) | 81 (32.7%) | 81 (11.2%) | |
| Number of drugs prescribed on admission | <.001 | ||
| 0-3, n (%) | 29 (11.7%) | 247 (34.1%) | |
| 4-6, n (%) | 59 (23.8%) | 203 (28.0%) | |
| ≥7, n (%) | 160 (64.5%) | 275 (37.9%) | |
| High-risk drugs | |||
| Cardiovascular drugs, n (%) | 78 (31.5%) | 181 (25.0%) | .05591 |
| Anticoagulants, n (%) | 75 (30.2%) | 138 (19.0%) | <.001 |
| Diabetes drugs, n (%) | 60 (24.2%) | 141 (19.4%) | .133 |
| Antiepileptics, n (%) | 51 (20.6%) | 85 (11.7%) | <.001 |
| Cancer drugs, n (%) | 6 (2.4%) | 26 (3.6%) | .4945 |
| Stays with a PI, n = 248 | Stays without a PI, n = 725 | P | |
|---|---|---|---|
| Characteristics of the groups | |||
| Age, mean (SD) | 72.3 (19.0) | 59.4 (21.1) | <.001 |
| Females, n (%) | 143 (57.7) | 358 (49.4) | .02933 |
| Overall score, mean (SD) | 7.4 (4.4) | 5.0 (4.5) | <.001 |
| Variable of the patient risk score | |||
| Age | <.001 | ||
| 18-74 years of age, n (%) | 107 (43.1%) | 520 (71.7%) | |
| 75-84, n (%) | 60 (24.2%) | 124 (17.1%) | |
| ≥85 years, n (%) | 81 (32.7%) | 81 (11.2%) | |
| Number of drugs prescribed on admission | <.001 | ||
| 0-3, n (%) | 29 (11.7%) | 247 (34.1%) | |
| 4-6, n (%) | 59 (23.8%) | 203 (28.0%) | |
| ≥7, n (%) | 160 (64.5%) | 275 (37.9%) | |
| High-risk drugs | |||
| Cardiovascular drugs, n (%) | 78 (31.5%) | 181 (25.0%) | .05591 |
| Anticoagulants, n (%) | 75 (30.2%) | 138 (19.0%) | <.001 |
| Diabetes drugs, n (%) | 60 (24.2%) | 141 (19.4%) | .133 |
| Antiepileptics, n (%) | 51 (20.6%) | 85 (11.7%) | <.001 |
| Cancer drugs, n (%) | 6 (2.4%) | 26 (3.6%) | .4945 |
Abbreviation: PI: pharmacist intervention.
Calculation of the threshold, depending on whether or not a PI was issued.
For each possible score threshold (ie, from 0 to 21), we evaluated the sensitivity, specificity, and PPV (Table 3). The AUC (95% CI) (calculated as a measure of the ability to distinguish between hospital stays with a PI and those without a PI) was 0.66 (0.62-0.70) (Figure 2). Values between 0.6 and 0.7 are generally considered to be poor.
The ROC curve for the score’s ability to distinguish between hospital stays with a PI and those without a PI. Abbreviations: PI, pharmacist intervention; ROC, receiver operating characteristic.
Sensitivity, specificity, and PPV for each score threshold value.
| Score threshold | Occurrence, n (%) | Sensitivity (%) | Specificity (%) | PPV (%) |
|---|---|---|---|---|
| 0 | 174 (17.9%) | 0 | 100 | 25.5 |
| 1 | 18 (1.8%) | 95.6 | 22.5 | 29.7 |
| 2 | 138 (14.2%) | 94.4 | 24.6 | 30.0 |
| 3 | 43 (4.4%) | 83.9 | 40.0 | 32.3 |
| 4 | 120 (12.3%) | 80.6 | 44.8 | 33.3 |
| 5 | 40 (4.2%) | 67.7 | 57.0 | 35.0 |
| 6 | 72 (7.4%) | 62.9 | 60.8 | 35.5 |
| 7 | 50 (5.1%) | 50.8 | 66.6 | 34.2 |
| 8 | 37 (3.8%) | 45.2 | 71.6 | 35.2 |
| 9 | 55 (5.7%) | 39.5 | 74.8 | 34.9 |
| 10 | 36 (3.7%) | 31.9 | 79.7 | 35.0 |
| 11 | 58 (6.0%) | 28.2 | 83.4 | 36.8 |
| 12 | 34 (3.5%) | 20.2 | 88.7 | 37.9 |
| 13 | 32 (3.3%) | 16.9 | 92.3 | 42.9 |
| 14 | 37 (3.8%) | 11.7 | 94.9 | 43.9 |
| 15 | 16 (1.6%) | 5.6 | 97.9 | 48.3 |
| 16 | 7 (0.7%) | 2.0 | 98.9 | 38.5 |
| 17 | 3 (0.3%) | 0.8 | 99.4 | 33.3 |
| 18 | 2 (0.2%) | 0.8 | 99.9 | 66.7 |
| 19 | 0 | 0.4 | 100 | 0 |
| 20 | 0 | 0.4 | 100 | 0 |
| 21 | 1 (0.1%) | 0.4 | 100 | 0 |
| Score threshold | Occurrence, n (%) | Sensitivity (%) | Specificity (%) | PPV (%) |
|---|---|---|---|---|
| 0 | 174 (17.9%) | 0 | 100 | 25.5 |
| 1 | 18 (1.8%) | 95.6 | 22.5 | 29.7 |
| 2 | 138 (14.2%) | 94.4 | 24.6 | 30.0 |
| 3 | 43 (4.4%) | 83.9 | 40.0 | 32.3 |
| 4 | 120 (12.3%) | 80.6 | 44.8 | 33.3 |
| 5 | 40 (4.2%) | 67.7 | 57.0 | 35.0 |
| 6 | 72 (7.4%) | 62.9 | 60.8 | 35.5 |
| 7 | 50 (5.1%) | 50.8 | 66.6 | 34.2 |
| 8 | 37 (3.8%) | 45.2 | 71.6 | 35.2 |
| 9 | 55 (5.7%) | 39.5 | 74.8 | 34.9 |
| 10 | 36 (3.7%) | 31.9 | 79.7 | 35.0 |
| 11 | 58 (6.0%) | 28.2 | 83.4 | 36.8 |
| 12 | 34 (3.5%) | 20.2 | 88.7 | 37.9 |
| 13 | 32 (3.3%) | 16.9 | 92.3 | 42.9 |
| 14 | 37 (3.8%) | 11.7 | 94.9 | 43.9 |
| 15 | 16 (1.6%) | 5.6 | 97.9 | 48.3 |
| 16 | 7 (0.7%) | 2.0 | 98.9 | 38.5 |
| 17 | 3 (0.3%) | 0.8 | 99.4 | 33.3 |
| 18 | 2 (0.2%) | 0.8 | 99.9 | 66.7 |
| 19 | 0 | 0.4 | 100 | 0 |
| 20 | 0 | 0.4 | 100 | 0 |
| 21 | 1 (0.1%) | 0.4 | 100 | 0 |
Abbreviation: PPV, positive predictive value.
Sensitivity, specificity, and PPV for each score threshold value.
| Score threshold | Occurrence, n (%) | Sensitivity (%) | Specificity (%) | PPV (%) |
|---|---|---|---|---|
| 0 | 174 (17.9%) | 0 | 100 | 25.5 |
| 1 | 18 (1.8%) | 95.6 | 22.5 | 29.7 |
| 2 | 138 (14.2%) | 94.4 | 24.6 | 30.0 |
| 3 | 43 (4.4%) | 83.9 | 40.0 | 32.3 |
| 4 | 120 (12.3%) | 80.6 | 44.8 | 33.3 |
| 5 | 40 (4.2%) | 67.7 | 57.0 | 35.0 |
| 6 | 72 (7.4%) | 62.9 | 60.8 | 35.5 |
| 7 | 50 (5.1%) | 50.8 | 66.6 | 34.2 |
| 8 | 37 (3.8%) | 45.2 | 71.6 | 35.2 |
| 9 | 55 (5.7%) | 39.5 | 74.8 | 34.9 |
| 10 | 36 (3.7%) | 31.9 | 79.7 | 35.0 |
| 11 | 58 (6.0%) | 28.2 | 83.4 | 36.8 |
| 12 | 34 (3.5%) | 20.2 | 88.7 | 37.9 |
| 13 | 32 (3.3%) | 16.9 | 92.3 | 42.9 |
| 14 | 37 (3.8%) | 11.7 | 94.9 | 43.9 |
| 15 | 16 (1.6%) | 5.6 | 97.9 | 48.3 |
| 16 | 7 (0.7%) | 2.0 | 98.9 | 38.5 |
| 17 | 3 (0.3%) | 0.8 | 99.4 | 33.3 |
| 18 | 2 (0.2%) | 0.8 | 99.9 | 66.7 |
| 19 | 0 | 0.4 | 100 | 0 |
| 20 | 0 | 0.4 | 100 | 0 |
| 21 | 1 (0.1%) | 0.4 | 100 | 0 |
| Score threshold | Occurrence, n (%) | Sensitivity (%) | Specificity (%) | PPV (%) |
|---|---|---|---|---|
| 0 | 174 (17.9%) | 0 | 100 | 25.5 |
| 1 | 18 (1.8%) | 95.6 | 22.5 | 29.7 |
| 2 | 138 (14.2%) | 94.4 | 24.6 | 30.0 |
| 3 | 43 (4.4%) | 83.9 | 40.0 | 32.3 |
| 4 | 120 (12.3%) | 80.6 | 44.8 | 33.3 |
| 5 | 40 (4.2%) | 67.7 | 57.0 | 35.0 |
| 6 | 72 (7.4%) | 62.9 | 60.8 | 35.5 |
| 7 | 50 (5.1%) | 50.8 | 66.6 | 34.2 |
| 8 | 37 (3.8%) | 45.2 | 71.6 | 35.2 |
| 9 | 55 (5.7%) | 39.5 | 74.8 | 34.9 |
| 10 | 36 (3.7%) | 31.9 | 79.7 | 35.0 |
| 11 | 58 (6.0%) | 28.2 | 83.4 | 36.8 |
| 12 | 34 (3.5%) | 20.2 | 88.7 | 37.9 |
| 13 | 32 (3.3%) | 16.9 | 92.3 | 42.9 |
| 14 | 37 (3.8%) | 11.7 | 94.9 | 43.9 |
| 15 | 16 (1.6%) | 5.6 | 97.9 | 48.3 |
| 16 | 7 (0.7%) | 2.0 | 98.9 | 38.5 |
| 17 | 3 (0.3%) | 0.8 | 99.4 | 33.3 |
| 18 | 2 (0.2%) | 0.8 | 99.9 | 66.7 |
| 19 | 0 | 0.4 | 100 | 0 |
| 20 | 0 | 0.4 | 100 | 0 |
| 21 | 1 (0.1%) | 0.4 | 100 | 0 |
Abbreviation: PPV, positive predictive value.
The optimal threshold (according to the Youden index) was 4 (Figure 3); this score gave a sensitivity of 80.6%, a specificity of 44.8%, and a PPV of 33.3%.
The Youden index for each score threshold.
Clinical perspectives
In our study, a score threshold of 4 flagged up 600 hospital stays (61.7% of those with a medication review); 200 (33.3%) of the latter led to a PI. At this threshold, however, 48 stays (5.0% of those with a medication review) that triggered a PI were not detected.
A score threshold of 3 flagged up 643 hospital stays (66.1% of those with a medication review); 208 (32.3%) of the latter led to one or more PIs. At this threshold, 40 stays (4.1% of those with a medication review) that triggered a PI would not have been detected.
Lastly, a score threshold of 5 flagged up 480 hospital stays (49.3% of the those with a medication review); 168 of the latter (35.0%) led to a PI. At this threshold, 80 stays (8.2% of those with a medication review) that triggered a PI would not have been detected.
Discussion
The objective of the present study was to identify a patient risk score threshold that could identify hospital stays which were likely to result in a PI after a medication review had been performed by a clinical pharmacist. The patient risk score was generated by the CDSS used by our hospital’s clinical pharmacists, with a view to optimize patient management. Our results showed that a threshold of 4 had the greatest discriminant ability for identifying hospital stays likely to lead to the issuance of a PI. At this threshold, 33.3% of the detected hospital stays resulted in a PI. In contrast, 48 stays (5.0% of the stays with a medication review) that triggered a PI were not detected at this threshold. During the study period, 600 (61.7%) of the stays with a medication review had a patient risk score of 4 or more (ie, about 30 admissions per 5-day working week). However, given the number of stays concerned and how easy it is to exceed the threshold of 4, a higher score appeared to be more relevant and might be more discriminant. Moreover, our results confirmed that older people are more likely to experience an ADR; a score of 4 is easily reached if the patient is aged 85 or over and is taking 4 or more drugs. Even polymedication alone is enough to achieve a score of 4, since 4 points are attributed to a patient taking 7 or more drugs. A score threshold of 5 flagged up 480 stays (ie, 20.0% less than a threshold of 4). This increase in the optimal threshold was associated with a decrease in the score’s sensitivity. The difficulty in determining an optimal score is to avoid over-alerting and to detect only relevant hospital stays that could lead to a PI. This score is complementary to the pharmacist’s review of prescriptions, as it helps him/her to prioritize prescriptions to be reviewed as part of his/her daily work. The score could be used without a threshold, not to select a list of hospital stays to be reviewed, but rather to reorder the list of hospital stays to be reviewed, leaving it up to the pharmacist to prioritize according to the available time.
The detection of at-risk situations is a complex and often subjective process. Thus, data from routine clinical practice or from prescriptions with a PI constitute useful sources of information. A French study of a PI database showed that combining machine learning with a CDSS increased the ability to detect potentially inappropriate prescriptions.18 In a study in the United States, the use of data from routine clinical practice helped to identify prescriptions requiring PI.19 Furthermore, a modified score based on drug prescriptions helped to predict 1-year mortality in an Italian study.20
In the present study, the PI acceptance rate (94.0%) was high—emphasizing the importance of PIs and their impact on the medical staff. This rate is in line with the literature data.21,22
Perspectives
The PharmaScore module was at the prototype stage when the present study was performed. For routine use, research on the score module’s usability and optimization must be considered, so that the system can be better integrated into the hospital’s clinical pharmacy activities. In our study, the score was calculated 12-36 h after the patient’s admission, and we excluded hospital stays for which a medication review was performed before the patient risk score had been calculated. In actual clinical practice, however, the score must be calculated rapidly so that it fits with the clinical pharmacists’ usual procedures. The score’s objective is to optimize medication reviews on admission of a patient. To be relevant, the score must be calculated before the pharmacist decides whether or not to review a patient’s medications.
Other research might involve the inclusion of new variables, in order to increase the tool’s sensitivity and specificity. However, any such variables must be codable and easily integrated in the CDSS and thus the patient risk score.
Study limitations
Firstly, our results might have been influenced by selection bias, as the main use of the Canadian score has been adapted to our study. When analyzing the home prescriptions of newly admitted patients, we decided to exclude intravenously administered drugs and enoxaparin at the doses typically used for thrombosis prevention. This choice might have generated some false positives because enoxaparin at a dose of 6000 IU/0.6 mL was not excluded but is sometimes used as for thrombosis prevention in obese patients. Secondly, we used ATC codes to select stays with at-risk drugs. Stays by epileptic patients were selected if they featured a drug with the ATC code N03; however, this selection might also generate false positives because drugs with the ATC code N03 are sometimes also used to treat patients with neuropathic pain. There was also a risk of false negatives because epileptic patients can be treated with drugs coded as ATC N03. Thirdly, the time interval of 12-36 h between admission and the medication review was rather restrictive; some drugs might have been left out of our analysis, that is, drugs taken before admission but erroneously omitted from the medication review or prescribed later in the hospital stay. Fourthly, the patient risk score was adapted for the CDSS used in our hospital, and the parameters were related to local clinical pharmacy practices (eg, when deciding whether or not to issue a PI); hence, our results cannot be extrapolated to other health systems. Lastly, the clinical pharmacist might have forgotten to track some of the PIs logged in the CPOE system. The study described the construction of the score, and presentation of definitive results is needed in the future.
Study strengths
Firstly, the large number of clinical pharmacists in our team enabled us to collect and analyze a significant amount of data over a short period of time. The use of a computer-based tool and the inclusion of the patient risk score in the CDSS facilitated the extraction of study data. During the study, the clinical pharmacists were blinded to the patient risk score calculated in the PharmaScore module, and so the score did not influence the medication reviews potentially prompted by a PI.
Conclusion
With the growing number of wards linked to EHRs databases and an increase in the clinical pharmacist’s various activities, better time management is essential. Our results showed that (1) having a patient risk score threshold helps the clinical pharmacist to target hospital stays likely to trigger a PI, (2) this task is complex, and (3) the results obtained can be improved further. Definition of an optimal score threshold is complex because the process is based on data from clinical practice; for optimal use, it must reflect conditions on the ward. In a context of reduced human resources, this score could be used to sort out hospital stays, enabling pharmacists to prioritize themselves by maximizing their capacity to see at-risk drug prescriptions. Extensive collaboration with the CDSS’s developers is now required to evaluate the routine use of the patient risk score by clinical pharmacy teams.
Acknowledgments
We thank the technical team of Keenturtle for their assistance and for the development of the score and the clinical pharmacy team (S. Belaiche, M. Dambrine, S. Genay, S. Gilliot, H. Henry, K. Laaziri, A. Leleux, M. Masse, F. Moreau, M. Perez, N. Simon, A. Toulemonde, H. Tribouillard, B. Valentin). We also thank David Fraser PhD (Biotech Communication SARL, Ploudalmézeau, France) for copy-editing assistance.
Author contributions
Laurine Robert (Conceptualization, Data curation, Formal analysis, Methodology, Writing—original draft), Nathalie Vidoni (Conceptualization, Data curation, Formal analysis, Methodology, Writing—review & editing), Erwin Gérard (Conceptualization, Methodology, Validation, Writing—review & editing), Emmanuel Chazard (Conceptualization, Formal analysis, Methodology, Validation, Writing—review & editing), Pascal Odou (Conceptualization, Methodology, Validation, Writing—review & editing), Chloé Rousselière (Conceptualization, Formal analysis, Methodology, Supervision, Validation, Writing—review & editing), and Bertrand Décaudin (Conceptualization, Methodology, Validation, Writing—review & editing)
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Conflicts of interest
The authors have no competing interests to declare.
Data availability
Data underlying this article cannot be shared publicly. Data will be shared on reasonable request to the corresponding author following ethical review and execution of appropriate data use agreements.
References
Commission staff working document—accompanying document to the :#Proposal for a Regulation of the European Parliament and of the Council amending, as regards pharmacovigilance of medicinal products for human use, Regulation (EC) No. 726/2004 laying down Community procedures for the authorisation and supervision of medicinal products for human and veterinary use and establishing a European Medicines Agency and the : #Proposal for a Directive of the European Parliament and of the Council amending, as regards pharmacovigilance, Directive 2001/83/EC on the Community code relating to medicinal products for human use #Summary of the impact assessment {COM(2008) 664 final} {COM(2008) 665 final} {SEC(2008) 2670}.
