1.2 Classification and outcome measures

Classification is needed to:

If it is to be useful it needs to be:

The field of trauma and orthopaedics is ripe with various classification systems and different outcome measures. It is important for a clinician to be conversant with them as they help in communication of ideas, sharing of opinion, and measurement of clinical outcomes. Many organizations perceive them as useful indicators of quality of practice, which is crucial in delivery of quality healthcare. In addition, the day-to-day practice needs to be open to external scrutiny, analysis, comparison for clinical improvement, legislation, and performance assessment by healthcare providers, healthcare system payers, and government and voluntary regulatory bodies. These reasons necessitate the availability of good classification systems and robust outcome measures.

Classification may be defined as a system whereby one can arrange or organize any number of people or things (seen as a division or a group) based on type, quality, characteristics, etc. (Box 1.2.1).

Any classification system should be clinically useful, reproducible, validated, easy to use, and, most importantly, have appropriate divisions to separate from each other all the (known) types or grades of the condition, it is meant to classify.

Biomedical studies are either experimental (the investigator assigned the exposures) or observational (investigator did not assign the exposures). An experimental study can be a randomized or a non-randomized controlled trial depending upon whether the allocation to a particular group was at random or not. An observational study can be analytical or descriptive depending upon whether there is a comparison group or not. The analytical studies can be further classified into cohort study (exposure precedes outcome), case–control study (outcome precedes exposure, e.g. identify an outcome and work backwards as to what might have caused the outcome), and a cross-sectional study (exposure and outcome occur at the same time).

Most reports of outcome in orthopaedics are observational studies. Whilst these studies give useful indications of outcome, in most cases they are open to a variety of bias. Bias may be defined as a non-random, systematic error in the design or conduct of a study that may result in mistaken inference about association or causation. In the orthopaedic literature, common types of bias include recall, selection, sampling, and publication bias.

The double blind, prospective, randomized clinical trial (RCT) is the gold standard of an experimental study design (Table 1.2.1). Randomization ensures that allocation to the comparison groups is unbiased. An RCT usually requires large numbers of patients whose diagnoses and severity grading is relatively similar (narrow entry criteria). Stirrat and colleagues in 1992 spelled out some of the problems that can arise with RCTs in surgery: ‘Placebo operations are unethical. Blinding of the patient is usually difficult if not impossible. Surgical skills vary between surgeons and also the “learning curve” plays an important role when new operations are being compared to existing ones’ (Box 1.2.2).

Outcome measures are vital to the setting of standards of care and their measurement, as well as in the assessment of disease or injury severity. The British Paediatric Association Outcome Measures Working Group defines an outcome measurement as a subtype of health status measurement where changes in the measure are known (or at least believed) to be largely attributable to a health service intervention.

Outcomes can be binary (death), continuous (change in blood pressure), a count (number of episodes), or a score (knee society score, Oxford hip, knee, or shoulder score). Also, the outcome can be objective (mortality rate) or subjective (pain, quality of life). To assess outcomes, condition-specific and general questionnaires are widely available, but their sensitivity to change and validity is open to question (Box 1.2.4).

An outcome measure should be quick, simple to use, reliable, specific to the question being investigated, cost-effective, and applicable. In most cases, this ‘ideal’ instrument does not exist, although many measures have come into general use without meeting these criteria (Box 1.2.3).

Outcome measures may be broadly classified into those used to measure doctors’ assessment, and those used to measure the patient’s own assessment of their problem. In most cases, it is appropriate and perhaps necessary to record both types of outcome. These two measures tend to assess different aspects of a condition and should normally be presented as separate outcomes. In the past, patient-based outcome measures have been dismissed as being too unreliable (subjective test vs. objective data) but in fact these assessments tend to follow closely the main indication for the original indication.

Two essential requirements of an outcome measure are that it measures what it is supposed to and that this measure is made with the minimum of error. The former is called ‘validity’ and the latter ‘reliability’. Content validity examines the ability of the instrument to measure all aspects of the condition for which it was designed so that it is applicable to all patients with that condition. One problem with increasing the content is that the reliability tends to decrease; however, validity at the expense of some reliability is the rule.

Another important issue with an outcome measure is ‘agreement’. Various reliability coefficients are available. An intraclass correlation coefficient (ICC) should be used for continuous data to measure agreement between or within methods or raters. The Pearson correlation is based on regression analysis and is a measure of the extent to which the relationship between two variables can be described by a straight (regression) line. Pearson’s product moment correlation coefficient tends to over-estimate the agreement and is unable to distinguish data when there is a systematic error. It is always worth plotting the results to find systematic biases when comparing data. The best way to plot the data is by placing the mean of points on the x-axis and the difference on the y-axis. This method has been extended by Altman and Brand for normally distributed data to include lines at two standard deviations above and below the mean line, these being termed the ‘limit of agreement’.

If the instrument is being assessed for its reliability as a clinical test, then sensitivity, specificity should be used. These are derived from the proportions of a 2×2 table.

An outcome instrument should be able to detect changes with time and this is termed as ‘responsiveness’. It is important to ensure that the instrument can detect a clinically important change, even if this is quite small. This has two implications—firstly the clinically important change should be and can be defined and secondly this change must be considered while designing an experiment so that power calculations can be made accordingly.

Outcome measures can be substantive (what one really wants to know) or surrogate (what one often ends measuring instead). For a substantive measure (e.g. time taken for an implant to fail), the follow-up is usually prolonged, and during the life of the study one tends to lose participants either due to withdrawal (by patient or surgeon) or due to an adverse event (death/physical or mental impairment) or inability to keep track of patient’s movements (lost to follow-up). For these reasons, surrogate measures are commonly used in clinical practice. However, one has to remember that the association between the surrogate and substantive outcome may be altered by the intervention and also unless the trial is powered to detect a difference in the substantive outcome, the effect of the intervention on the surrogate outcome may determine the clinical practice which can prove to be harmful.

Inclusion of health-related quality of life (HRQL) is increasing in popularity as an outcome measure after any intervention. There are two key dimensions to it: primary (physical, psychological, social, etc.) and additional (neuropsychological, personal productivity, pain, etc.). HRQL provides a method of measuring intervention effects as well as the effects of the untreated course of diseases, in a manner that may be helpful to both the investigator as well as the individual. Most outcome measures tend to assess one or more of HRQL.

Another commonly used measure in orthopaedics is ‘survival analysis’ (Box 1.2.5). This is important in studies where participants are entered over a period of time and therefore have various lengths of follow-ups. These methods permit the comparison of the entire survival experience during the follow-up and may be used for the analysis of time to any dichotomous response variable such as a non-fatal event or an adverse effect. The graphical presentation of the total survival experience is called the survival curve and the tabular presentation is called the life table. Two commonly used methods are Kaplan–Meier and Cutler–Edere method. For a Kaplan–Meier estimate, one needs to know the exact time of entry into the trial and also the exact time of the event or loss of follow-up. In this method, the follow-up period is divided into intervals of time so that no interval contains both deaths and losses. In addition, one makes two assumptions. Firstly, it is assumed that at any time patients who are censored have the same survival prospects as those who continue to be followed. Secondly, it is assumed that the survival probabilities are the same for the subjects recruited early and late in the study. For some studies, all that is known is that within an interval of time, a known number of deaths and losses occurred amongst the known number of participants at risk. In such cases, the Kaplan–Meier method can not be used. In the Cutler–Edere method, the assumption is made that the deaths and losses are uniformly distributed over an interval to overcome this problem.

For expressing survival data of a particular implant, the life-table method is usually used. In this method, generally the cumulative survival is calculated at regular time intervals (usually yearly). The life-table method has advantages as it reports number of implants followed, number of failures, loss to follow-up, and confidence intervals. Establishing patients who are lost to follow-up are important in interpreting a survival analysis. Each individual lost may have been a failure and may therefore have dramatically altered the final result of the analysis. This can be addressed by including a worst-case scenario calculation in all survival studies to display the possible results of the loss to follow-up on the study. Confidence intervals give some sense of uncertainty in the estimated treatment or intervention effect. In a survival study, wide confidence intervals suggest that the numbers at risk were small and the results must be treated with caution. The method of Peto is usually used to calculate confidence intervals.

This brief chapter can not summarize all the relevant classification systems and outcome measures used in trauma and orthopaedics. The reader should refer to standard orthopaedic textbooks for the same. An excellent in-depth critique on this subject is available in Outcome Measures in Orthopaedics and Orthopaedic Trauma, by Pynsent et al.