Determination of Clinically Acceptable Analytical Variation of Cardiac Troponin at Decision Thresholds

Abstract

Background

Clinical decision-making for risk stratification for possible myocardial infarction (MI) uses high-sensitivity cardiac troponin (hs-cTn) thresholds that range from the limit of detection to several-fold higher than the upper reference limit (URL). To establish a minimum analytical variation standard, we can quantify the effect of variation on the population clinical measures of safety (sensitivity) and effectiveness [proportion below threshold, or positive predictive value (PPV)].

Methods

From large datasets of patients investigated for possible MI with the Abbott hs-cTnI and Roche hs-cTnT assays, we synthesized datasets of 1 000 000 simulated patients. Troponin concentrations were randomly varied several times based on absolute deviations of 0.5 to 3 ng/L and relative changes of 2% to 20% around the low-risk threshold (5 ng/L) and URLs, respectively.

Results

For both assays at the low-risk thresholds, there were negligible differences in sensitivity (<0.3%) with increasing analytical variation. The proportion of patients characterized as low risk reduced by 30% to 29% (Roche) and 53% to 44% (Abbott). At the URL, increasing analytical variation also did not change sensitivity; the PPV fell by less than 3%. For risk stratification, increased delta thresholds (change between serial troponin concentrations) increased sensitivity at the cost of a decreased percentage of patients below the delta threshold, with the largest changes at the greatest analytical variation.

Conclusions

At the low-risk threshold, analytical variation up to 3 ng/L minimally impacted the safety metric (sensitivity) but marginally reduced effectiveness. Similarly, at the URL even relative variation up to 25% minimally impacted safety metrics and effectiveness. Analytical variation for delta thresholds did not negatively impact sensitivity but decreased effectiveness.

Introduction

The use of a single high-sensitivity cardiac troponin (hs-cTn) concentration measurement from an emergency department presentation blood draw to risk-stratify patients for possible myocardial infarction (MI) is now an established procedure and guideline recommended (1–3). Two large meta-analyses established the safety of allocating patients to low risk for MI or major adverse cardiac event based on a cTn concentration lower than a prespecified concentration threshold (4, 5). These were for the Abbott ARCHITECT hs-cTnI and the Roche hs-cTnT assays. Low-risk thresholds for other troponin assays have been assessed in much smaller, generally single health jurisdiction, cohorts. The recommendations are not to stratify as low risk a patient if the blood draw was made within 2 or 3 h of symptom onset (there is no consensus on the exact timing). Additionally, either through clinician judgment (1) or use of an accelerated diagnostic pathway incorporating a risk score and/or electrocardiogram, other patients with low troponin concentrations may not be considered eligible for stratification as low risk (6–9). These and other patients may be discharged from the emergency department after a second blood draw where there is minimal change in troponin concentration.

The safety (acceptable risk) requirement for low-risk troponin thresholds is high, usually >99% sensitivity (and >99.5% negative predictive value) (10). Many thresholds were developed on datasets with approximately 100 or fewer events (MI or major adverse cardiac event), meaning that the thresholds developed were unlikely to be optimized and were very much dependent on the one or 2 lowest concentrations in a dataset (e.g., for 99% sensitivity with 100 events, the second-lowest concentration becomes the threshold) (11).

At low thresholds, the analytical variation of the test can be large compared with the variation at the upper reference limits (URLs; with 99th percentile concentration from a healthy population being the URL recommended for troponin). Some of the low-risk thresholds are less than the limit of quantitation (LoQ) and may have CVs of >20%, with variation at the limit of detection (LoD) difficult to determine due to how laboratories report the data (often simply < LoD) (12).

For laboratory professionals, there are currently few options to assess quality of an assay at such low concentrations. Quality control (QC) materials are not universally available at these specific thresholds, and laboratories have constructed their own material to facilitate optimal monitoring (13). The International Federation of Clinical Chemistry and Laboratory Medicine has recommended using a QC material with a reference value between the LoD and the lowest of the sex-specific 99th percentiles (14).

What an acceptable precision metric (i.e., SD or CV) for this low QC material should be is unknown, as it pertains to clinical impact. Previous simulation work has shown that there can be misclassification due to both imprecision and, in particular, bias (15). However, bias is not always vigorously assessed, with lot-to-lot comparisons assessing this metric typically based on a few samples at important decision thresholds (16). For laboratory professionals and clinicians, an easier concept is likely to be interpretation of variation in absolute and relative terms, as these concepts are imbedded in many accelerated diagnostic pathways. What is unknown is whether higher absolute variation near the low-risk threshold and higher relative variation near the URL will reduce safety (sensitivity) or effectiveness [percentage below threshold or positive predictive value (PPV)]. Similarly, to what extent would absolute variation affect the safety and effectiveness of a delta threshold (difference between serial measurements) below the URL? To address these questions, we utilized a large high-fidelity dataset and a validated data simulation technique to assess the impact of absolute and relative hs-cTn variation.

Materials and Methods

We created two 1-million patient synthesized datasets (one for each troponin assay investigated) from research datasets drawn from the Christchurch, New Zealand data repository created from 4 studies: ADAPT (17), ADAPT-RCT (18), EDACS-RCT (19), and SPACE-24 (20). The inclusion and exclusion criteria were substantially similar for all 4 studies: Patients aged ≥18y presented acutely from the community to an emergency department with possible cardiac symptoms that prompted the attending physician to investigate for possible acute coronary syndrome. In accordance with American Heart Association case definitions, these symptoms included the presence of acute chest, epigastric, neck, jaw, or arm pain or discomfort or pressure without an apparent noncardiac source (21). Exclusions were ST-elevated myocardial infarction on presentation, need for admission for medical conditions other than cardiac, absence of informed consent, or anticipated problems with follow-up (e.g., non-New Zealand resident). Blood was drawn into lithium-heparin or EDTA tubes, centrifuged immediately, and stored at −80°C before testing. Two adjudicators independently assessed outcomes according to the Third Universal Definition for Myocardial Infarction (22). Where there was disagreement, a third senior cardiologist adjudicator was used.

Assays Used for cTn Measurement

We used 2 assays for cTn measurement. The hs-cTnI Abbott ARCHITECT i2000 assay (Abbott Diagnostics) has a manufacturer reported limit of blank of 0.7 to 1.3 ng/L, a LoD of 1.1 to 1.9 ng/L, an overall URL of 26 ng/L, and a sex-specific URL of 16 ng/L for females and 34 ng/L for males (23). The low-risk threshold is 5 ng/L (5). The hs-cTnT Cobas e411 assay (Roche Diagnostics) has a manufacturer-reported limit of blank 3 ng/L, LoD 5 ng/L, a sex-specific URL of 9 ng/L for females and 17 ng/L for males, and an overall URL 14 ng/L (23). The low-risk threshold outside the United States is 5 ng/L (4).

Data Simulation Process

Synthetic datasets were created by sampling from probability distributions of variables, obtained from the research data, so that the statistical features of research data were preserved (24). The approach is similar to that used by the US Census Bureau and the same as was used previously to determine low-risk stratification thresholds of a troponin assay (25). The synthesized cTn distributions were conditional on other variables. The classification tree with the density smoothing option for the continuous variables in the R package, synthpop, was used (24). Multiple synthesized datasets were created separately for each hs-cTn assay such that the final combined synthetic dataset for each assay contained >200 000 MI; 150 000 MI and 850 000 non-MI were then randomly drawn this data for each assay to form the 1-million patient datasets used in this analysis. The choice of 15% MI prevalence is arbitrary. However, it will not affect the analysis of the primary safety metric, sensitivity, which is independent of prevalence. The effectiveness measure of the proportion less than a low-risk threshold is likely to be higher than in populations with lower prevalence, but the relative difference in effectiveness brought about by analytical variation will not change with prevalence.

The research data used for synthesis included the components of the Emergency Department Assessment of Chest Pain accelerated diagnostic pathway (19, 26), namely sex; age; history of hypertension; history of dyslipidaemia; history of smoking; history of coronary artery disease; family history of cardiovascular disease; diaphoresis; pain with palpation; pleuritic pain; pain radiating to arm, shoulder, neck or jaw; new ischemia on electrocardiogram; and presentation and 2 h cTn concentrations along with the primary outcome, index admission MI. Apart from age and cTn concentrations, all were Boolean variables.

To estimate the effect of analytical variation, for each synthesized patient the synthesized cTn concentrations was considered to be the true (baseline) concentrations. Then, to evaluate the performance of the low-risk threshold and the delta change less than the URL on each synthesized concentration, 6 more cTn concentrations were created by randomly sampling across a normal distribution with a mean equal to each baseline concentration and a SD equal.to the absolute variations of 0.5, 1, 1.5, 2, 2.5, and 3 ng/L. To evaluate the performance of the URL, the SDs were set at 2%, 5%, 10%, 15%, 20%, and 25% of the overall URL for the assay. The rnorm function with inputs of baseline concentration and SD in base R was used.

For example, a synthesized patient with a true concentration of 9 ng/L may have imprecision concentrations after applying the random function with SDs 0.5, 1, 1.5, 2, 2.5, and 3 and mean of 9, of 8.84, 9.61, 7.95, 8.65, and 8.12 ng/L (rounded to 9, 10, 8, 9, and 8 ng/L), whereas another synthesized patient with a true 9 ng/L has an imprecision concentration of 9.44, 9.37, 9.07, 11.00, and 5.49 ng/L (rounded to 9, 9, 9, 11, and 5 ng/L). As the datasets used to synthesize data had concentrations reported <LoD, we allowed synthesized concentrations to be <LoD.

Metrics for Assessment of Effect of Analytical Variation

Safety was assessed by sensitivity at both the low-risk threshold and the overall URLs. At the low-risk threshold, the primary effectiveness measure was the proportion less than the threshold. These are those patients who (if they existed in the real world) may be eligible for early discharge after clinical features are taken into account. For clinicians, a higher proportion means earlier reassurance for patients that they are not having an MI and improvements in patient flow in the emergency department. At the URL, the primary effectiveness measure was the PPV; if the URL is used as a decision threshold for further investigation, a higher PPV minimizes the proportion of patients unnecessarily undergoing further investigation. The values of all metrics were reported as well as the relative change in safety and effectiveness, using the metrics of the dataset with the true concentrations for each assay.

Results

There were 2707 (hs-cTnT) and 2662 (hs-cTnI) patients in the research dataset. Approximately 37.5% of patients were female and the mean (SD) age was 63 (13) years (Table 1). There were 489 (18.1%) and 475 (17.8%) MI in the hs-cTnT and hs-cTnI research datasets, respectively. The distributions of cTn concentrations around the low-risk threshold and URLs are shown in Supplemental Figs. 1 and 2.

The 2 synthesized datasets had very similar demographics, patient histories, and presenting symptoms to their corresponding research datasets (Table 1). The distributions of troponin concentrations for those with and without an MI diagnosis were also similar (Supplemental Figs. 3 and 4). Random selections from the synthesized datasets of cohorts the same size as the research data shows, appropriately, similar numbers of patients near the low-risk threshold but with different numerical values (Supplemental Figs. 5 and 6).

Effect of Analytical Variation Around the Low-Risk Threshold

For both assays there was negligible difference in sensitivity (<0.3%), even with high absolute variation (Table 2 and Fig. 1A). As the variation increased, the proportion less than the threshold decreased (decreased effectiveness) at a greater rate for the hs-cTnI assay than the hs-cTnT assay (Fig. 1C). The absolute reduction in proportion below threshold was 8.6% (relative 16%) for hs-cTnI compared to 0.86% (relative 3%) for hs-cTnT. The starting proportion (the proportion with true, baseline concentrations for each assay) was much greater for the hs-cTnI assay (53%) than the hs-cTnT assay (30%) (Table 2).

URL Thresholds

For both assays, there were negligible differences in sensitivity, even with high relative variations (higher SDs) (Table 2 and Fig. 1B). As the variation increased, the specificity and PPV decreased, at a greater rate for the hs-cTnT assay than the hs-cTnI assay (Table 2 and Fig. 1D). This may be because true PPV was much lower and true proportion above the URL much higher for the hs-cTnT assay than the hs-cTnI assay.

Deltas Less Than the URL

Larger delta thresholds had lower sensitivity and greater effectiveness (Fig. 2). The sensitivity increased with increasing absolute variation, whereas the effectiveness was reduced. This is because analytical variation increases, on average, the size of the differences (delta) between serial measurements (Supplemental Fig. 7). Therefore, for any delta threshold there will be a greater proportion of individuals exhibiting deltas above the threshold, which results in increased safety (because some of those individuals would have an MI) and decreased effectiveness (because most of those individuals would not have an MI).

Discussion

At the threshold used for stratification to low risk, both assays maintained similar safety performance (i.e., sensitivity) even with high analytical absolute variation. The effectiveness was reduced at higher absolute values of analytical variation, but the proportions that could be allocated to low risk remained high and clinically useful. Similarly, at the URLs, where for high-sensitivity assays there is expected to be imprecision of <10%, there was a negligible change in sensitivity and little change in specificity or effectiveness. Even with greater imprecision, more commonly observed with previous-generation non-high sensitivity (contemporary) assays, the safety and effectiveness metrics were maintained.

For emergency department algorithms using a delta threshold to identify a change between troponin results, greater analytical variation did not result in a worse safety metric (reduced sensitivity), which offers some reassurance when using different assays with different analytical variation <URL. The trade-off is that effectiveness is reduced. In clinical practice, the effect of higher analytical variation would be more patients being considered for further testing (most likely just another troponin test). It should be noted that when designing clinical pathways, the use of larger delta thresholds will compensate for the lower effectiveness of higher analytical variation, but this will be at the cost of lower sensitivity.

In clinical practice, low-risk troponin thresholds are not recommended to be used alone for clinical decision-making and should not be used to stratify a patient as low risk if troponin is measured within 2 h or 3 h of symptom onset. Single-sample rule-out has been demonstrated to be safe in randomized trials and clinical practice (7, 8, 27, 28). In large meta-analyses for both the Roche and Abbott assays using 5 ng/L as a decision threshold, about 50% of the false negatives were identified as early presenters in whom troponin was measured within 3 h of symptom onset (4, 5). Uncertainty in establishing the time of onset of myocardial injury, or a small degree of myocardial injury in which case troponin takes longer to be elevated (20), may mean that most of the other false negatives are also early presenters, although it is possible that some present several days after the actual onset of injury by which time blood troponin concentrations have fallen below the low-risk threshold.

Three prior studies have addressed analytical variation issues. First, Apple et al. simulated the distribution of troponin and considered the effect of reducing the CV from 25% to 10% on the rate of false positives about the 99th percentile of a healthy population (29). They noted that lower imprecision would lower the 99th percentile (63 ng/L for 10% CV compared with 70 ng/L for 25% CV in their simulation). For serial sampling, the greater CV resulted in more false positives above the 99th percentile; however, the rate of only 3 to 5 per 1000 patients was low. While this simulation differs from our study in that the 99th percentile was different for the two CVs modeled, it supports our conclusion that greater analytical variation around the URL is likely to reduce the PPV. Second, Lyon et al. used data from the 1137-patient RING and ROMI study to simulate misclassification at the 2 and 5 ng/L thresholds of the Abbott assay (15). Monte Carlo simulation models have previously demonstrated that at or near the 99th percentile, Abbott hs-cTnI results are seldom misclassified for MI. Around 5 ng/L, there was little influence of CV, from 1% to 40%, on the misclassification. These results are congruent with the present study, which uses different methodology and is based on datasets several times larger. Third, in a theoretical exercise with no input from actual troponin measurements, in a hypothetical population with cTn concentrations corresponding to the low-risk threshold, Aakre et al. demonstrated how CVs from 2.5% to 20% would reclassify, after rounding to integers, patients at thresholds from 1 to 9 ng/L (30). They noted that, given the low prevalence of MI at these thresholds, the probability of misclassification was low even at the highest CV considered. What was not assessed, but has been in the present study, is the change in sensitivity and effectiveness for a typical cohort where there are many patients with concentrations below the low-risk threshold. They also summarized the analytical imprecision needed to determine change in troponin concentration (delta) with sufficient certainty for baseline concentrations from 1 to 14 ng/L with delta thresholds from <2 to <8 ng/L. For example, at a 5 ng/L baseline an analytical variation of <17.2% imprecision for a delta of <3 ng/L was needed to avoid misclassifying more than 5% of patients. Our study adds to these studies by its utilization of measurements made by 2 assays and by determining the effect of analytical variation on metrics used to judge the safety and effectiveness of troponin thresholds.

Biases may arise from lot-to-lot variation. This has been measured as 2 to 3 ng/L (worst observed SDs) on the Abbott assay and may invalidate algorithms that use minor absolute change criteria (31). There are also differences in the means and SDs of low-QC material between 3.8 (SD = 0.5) ng/L and 5.4 (SD = 3.3) ng/L. While monitoring lot-to-lot variations of the Roche hs-cTnT assay, Haagensen et al. found the number of patients who may be ruled out at a 5 ng/L threshold may vary from 15% to 30% (32). While the variation here may not affect safety, it could change effectiveness of a low-risk threshold. Commercially available QC materials near low-risk thresholds are needed to support the use of assays for low-risk stratification.

In the United States, the FDA has mandated that no hs-cTn results be reported below the LoQ (often associated with a concentration achieving a CV of 20%) because of concerns about patient misclassification. For some assays, the LoQ is above the threshold established and in use elsewhere. In fact, the use of a higher threshold increases the likelihood of false negatives (16). As the proportion less than the threshold will be higher, the negative predictive value may not be largely effected but the sensitivity will be lower. While the Roche assay threshold of 6 ng/L at the LoQ appears to be safe (33), this may not be the case for future assays or even in other patient populations (34). Our results, along with the evidence from a large meta-analysis and clinical studies, supports the opinion expressed by others that the FDA could permit reporting of hs-cTn concentrations below the LoQ (4, 16).

Below the URL, changes in troponin concentrations over 13 h are employed in some algorithms to risk stratify patients (1). These changes may be very small (13 ng/L). While small deltas will be safe, their effectiveness at risk stratification becomes lower with greater analytical variation. Kavsak and colleagues have demonstrated that below 10 ng/L across multiple assays, the maximum analytical variation was +/−3 ng/L and approximately 30% around the URL (35). At the thresholds of 5 ng/L in this study, this kind of variation is still safe but leads to reduced effectiveness. Aakre and colleagues demonstrated that to employ diagnostic pathways that incorporated a baseline troponin with a delta [e.g., the European Society of Cardiology pathways (1)], the higher the baseline troponin threshold concentration and/or the lower the delta threshold, the smaller the analytical imprecision required of the assay (30). For clinical pathways using a delta, we recommend that if changes >3 ng/L are used to risk stratify to higher risk classes, then the SD cannot be higher than 3 ng/L and optimally should be near 1 ng/L as previously demonstrated for long-term performance for hs-cTn assays near 5 ng/L (36, 37).

For high-sensitivity assays, where at the URL the CV has to be ≤10%, there was little effect on the effectiveness metric, the PPV. However, at higher CVs, possibly encountered with contemporary assays and many point-of-care assays, the effectiveness will be less. This simply highlights another advantage of the high-sensitivity assays.

As hs-cTn point-of-care assays for use both within emergency departments and prehospital settings have been introduced, efforts have been made to establish a road map for analytical characterization. Clinical performance studies in intended users are part of that road map, and this study helps support the interpretation of those results in light of understanding the effect of analytical imprecision on key clinical indicators.

Limitations

This study did not take into account other factors that are known to affect hs-cTn concentrations (e.g., hemolysis, biotin, blood collection tube, etc.) or lead to false elevations (e.g., fibrin, macrocomplex, etc.) (38, 39). It also did not model the impact of biases between different analyzers, between lots of reagents and calibrators, and between operators, nor did the study separately account for inter- or intraindividual components of variation.

The synthesized datasets all had the same prevalence of MI (15%). This means that, in a clinical setting where prevalence is lower, the proportion eligible for discharge following a troponin result less than the low-risk threshold will be higher. Consequently, where effectiveness is reduced because of analytical variation, the numbers eligible for early discharge will be reduced even further.

The use of synthetic data was necessitated because there are not datasets available with very large numbers of patients who had an MI and had low troponin concentrations reported. While the comparison of research and synthetic data suggest the synthetic data is representative of real patients, this cannot be known with certainty.

Conclusion

At low-risk thresholds, the analytical variation typically found with high-sensitivity troponin assays does not meaningfully impact test performance but has a marginal effect on the effectiveness of using those thresholds to risk stratify. Similarly, when the URL has the imprecision typical of contemporary assays, the safety metrics were not negatively impacted and effectiveness was only marginally reduced. Analytical variation for algorithms using a change of troponin threshold to risk stratify did not negatively impact the safety metric but decreased effectiveness.

Supplemental Material

Supplemental material is available at Clinical Chemistry online.

Nonstandard Abbreviations

hs-cTn, high-sensitivity cardiac troponin; MI, myocardial infarction; URL, upper reference limit; LoQ, limit of quantitation; LoD, limit of detection; QC, quality control; PPV, positive predictive value.

Author Contributions

The corresponding author takes full responsibility that all authors on this publication have met the following required criteria of eligibility for authorship: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved. Nobody who qualifies for authorship has been omitted from the list.

John Pickering (Conceptualization-Equal, Data curation-Lead, Formal analysis-Lead, Investigation-Equal, Methodology-Equal, Writing—original draft-Lead, Writing—review & editing-Equal), Peter Kavsak (Writing—review & editing-Supporting), Robert Christenson (Writing—review & editing-Supporting), Richard Troughton (Writing—review & editing-Supporting), Chris Pemberton (Writing—review & editing-Supporting), A. Mark Richards (Writing—review & editing-Supporting), Laura Joyce (Writing—review & editing-Supporting), and Martin Than (Conceptualization-Equal, Formal analysis-Equal, Investigation-Equal, Methodology-Equal, Writing—review & editing-Supporting).

Authors’ Disclosures or Potential Conflicts of Interest

Upon manuscript submission, all authors completed the author disclosure form.

Research Funding

None declared.

Disclosures

J.W. Pickering reports consultancy with Upstream Inc., Luminoma, Siemens Healthineers, Quidel Corporation, and Radiometer Health. P. Kavsak reports grants from Abbott Laboratories, Ortho Clinical Diagnostics, Randox Laboratories, Roche Diagnostics, and Siemens Healthcare Diagnostics; consulting fees from Abbott Point of Care, Beckman Coulter, Quidel, Roche, and Siemens Healthcare Diagnostics; honoraria from Abbott Laboratories, Siemens Healthcare Diagnostics, and Thermo Fisher Scientific; and travel support from Randox Laboratories, and McMaster University has the patents as an inventor “Method of Determining Risk of an Adverse Cardiac Event” (EP 3 341 723 A1) and “Quality Control Materials for Cardiac Troponin Testing.” R.H. Christenson reports grants or contracts from Siemens Healthineers, Roche Diagnostics, Becton Dickinson, Abbott Diagnostics, QuidelOrtho, and Beckman-Coulter; consulting fees for Siemens Healthineers, Beckman-Coulter, Roche Diagnostics, Abbott Diagnostics, QuidelOrtho, Sphingotech, Babson Diagnostics, and Becton Dickinson; and honoraria from Siemens Healthineers, Beckman-Coulter, Roche Diagnostics, QuidelOrtho, and Becton Dickinson and is editor-in chief for The Journal of Applied Laboratory Medicine, Association for Diagnostics & Laboratory Medicine. R.W. Troughton reports consulting fees and research grants from Roche Diagnostics, Merck, American Regent, and Bayer. C.J. Pemberton reports public research funding from the Health Research Council, Heart Foundation, and Ministry of Business, Innovation and Employment of New Zealand. A.M. Richards has received consulting fees from Roche Diagnostics for participation on the advisory board, travel support from Roche Diagnostics, has stock or stock options from Upstream Medical Technologies, has participated as Data Safety Monitoring board member for the STAREE trial and as Data Safety Monitoring board chair for the CRITICAL ACS trial, and is chair of Christchurch Heart Institute Trust. M.P. Than reports research funding from Abbott, Beckman, Radiometer, and Siemens Healthineers; provision of equipment and materials for research from Siemens Healthineers; consulting fees from Abbott, Radiometer, Roche, Siemens, and Upstream Medical Technologies; honoraria from Abbott, Roche, and Siemens; travel support from Siemens; and participation on advisory boards for Abbott, Radiometer, Roche, Siemens Healthineers, and Upstream Medical Technologies.

Role of Sponsor

No sponsor was declared.

References