Economic evaluation of occupational health and safety programmes in health care

Abstract

Background

Evidence-based resource allocation in the public health care sector requires reliable economic evaluations that are different from those needed in the commercial sector.

Aims

To describe a framework for conducting economic evaluations of occupational health and safety (OHS) programmes in health care developed with sector stakeholders. To define key resources and outcomes to be considered in economic evaluations of OHS programmes and to integrate these into a comprehensive framework.

Methods

Participatory action research supported by mixed qualitative and quantitative methods, including a multi-stakeholder working group, 25 key informant interviews, a 41-member Delphi panel and structured nominal group discussions.

Results

We found three resources had top priority: OHS staff time, training the workers and programme planning, promotion and evaluation. Similarly, five outcomes had top priority: number of injuries, safety climate, job satisfaction, quality of care and work days lost. The resulting framework was built around seven principles of good practice that stakeholders can use to assist them in conducting economic evaluations of OHS programmes.

Conclusions

Use of a framework resulting from this participatory action research approach may increase the quality of economic evaluations of OHS programmes and facilitate programme comparisons for evidence-based resource allocation decisions. The principles may be applicable to other service sectors funded from general taxes and more broadly to economic evaluations of OHS programmes in general.

Introduction

In conducting economic evaluations of occupational health and safety (OHS) programmes in public health care systems, the principles and frameworks developed for the private, for-profit sector do not accommodate the unique characteristics of this sector. From an economic point of view, a universal publicly funded health care system has distinct characteristics, as ‘shareholders’ and ‘customers’ receiving services are one and the same and the mandate is the provision of cost-effective quality care. Many entities are affected by the consequences of OHS programming in this sector, including health authorities, worker associations and workers’ compensation providers [1–3].

Economic evaluation of OHS initiatives in the peer-reviewed literature has been largely focused on the analysis of insurance costs, taking the perspective of the employer and relying on workers’ compensation expenses as the key and often the only measure of outcomes. This narrow perspective might be justified based on insurance costs representing the most immediate and direct cost implications of OHS incidents. Productivity losses associated with time off work or while at work, employee turnover and the industrial relations climate are rarely considered, though generally acknowledged as being outcomes of economic importance to the organization. Few studies consider other consequences, and fewer still the full range of relevant resources and outcomes associated with an OHS programme [4,5]. In fact, economic evaluations of OHS initiatives are clearly lagging in quality when compared with economic evaluations of new drugs and medical technologies [6–8].

To identify an appropriate framework for economic evaluations in the health care sector, there is a need for collaboration across multiple academic disciplines and the various stakeholders. Managers, unions, OHS professionals, OHS researchers and health economists have not traditionally collaborated together on OHS evaluations and might have inherent misunderstandings or mistrust of each other’s work [9,10]. Furthermore, an unbridled increase in health care expenditure has called into question the sustainability of the public health care system in Canada and has prompted two governmental commissions in the last decade [11,12].

There is clearly a need to develop methodological guidance, for researchers and practitioners, for economic evaluations of OHS programmes in health care. The aim of our project was to work collaboratively with public health care stakeholders in a Canadian province to develop an economic evaluation framework for OHS programmes. It had two specific objectives: to define key resources (costs) and outcomes (consequences) to be considered in economic evaluations of OHS programmes in public health care and to define how they should be integrated into a comprehensive framework.

Methods

The primary methodology was participatory action research supported by mixed qualitative and quantitative methods. It included a literature review, working group, key informant interviews, a Delphi panel, nominal group discussions and framework trials as illustrated in Figure 1 and described below. The Behavioral Research Ethics Board at The University of British Columbia reviewed and approved the project (H09-01314).

A review of scientific literature was conducted at the beginning of this project. It was based on a systematic review of OHS interventions with economic analysis [10] and an economic evaluation methods textbook for the OHS field [13]. The review provided an overview of ‘state of the art’ economic evaluations of OHS interventions. Key concepts from the review were incorporated into a background paper. This was a lay language summary of economic evaluations intended as a source of information to assist working group, interview and panel participants in this study.

A working group comprised of representatives of managers, unions, health care professionals and researchers met regularly over a period of 2½ years. Members were asked to consider and discuss results from each phase in the development of the framework and to approve the final version. The working group used nominal group techniques, wherein it was directed through a series of questions and asked for final agreement on the answers to those questions.

Twenty key informant interviews were conducted with decision makers within health care. The sample included directors and managers of health care organizations (9 people), health care unions (5), health care insurers (3) and the Ministry of Health Services (3). Individuals were chosen through a snowballing technique, whereby people interviewed were asked who else should be included in data gathering. Two interviews were conducted by telephone and all were recorded and transcribed for thematic analysis. Additionally five health care workers were interviewed to ask about their experience with OHS programmes, what resources and outcomes should be considered in making decisions about OHS programmes and how to best integrate the impacts of a programme on worker health and on patient care.

From interactions with the working group and the interviews, we learned that economic evaluation terms had different meanings between stakeholders, often different to those used in the academic literature. Economic evaluation, cost-benefit analysis and business case were sometimes used interchangeably and the word ‘cost’ was often used whenever something was expressed in dollars, irrespective of whether it was a resource or an outcome of an OHS programme. Therefore, the research team carefully defined every term in the framework and used the terms ‘resources’ and ‘outcomes’ of an OHS programme, rather than the terms ‘costs’ and ‘consequences’ (used in economics literature), to increase clarity. From the literature review and interviews, 61 candidate resources and outcomes were identified. After in-depth discussion, researchers determined a final list of 11 resources and 31 outcomes with specific definitions.

The Delphi panel comprised 46 members, 41 of whom completed all three panel rounds (10 worker, 12 employer and 11 insurer representatives, 3 Ministry of Health Services’ representatives and 10 patient representatives). Their task was to prioritize the 11 resources and 31 outcomes [14] using brief summaries printed in cards (Q-cards). They were asked to read the background document and the Q-cards and rank resources and outcomes according to their opinion as to which one should be considered first, second, third etc. by managers or directors when they make decisions about OHS programmes.

An environmental scan was conducted to identify data sources available to support the economic evaluation of OHS programmes. Three categories of information were collected: the type of data, data accessibility and the data administrator.

The first draft of the framework was created using the 11 resources and 31 outcomes and critiqued by the working group. The next iteration was presented to senior decision makers who provided input on its clarity and relevance, resulting on a refined draft.

Decision makers identified the importance of trialling the framework before it could be accepted and in agreement with the working group, the researchers chose three different circumstances to trial the framework: a hands-on evaluation of a ‘ceiling lift’ coaching programme at a health authority, with guided data collection and full economic analysis by research team economists; an in-house application of the framework to a ceiling lift programme in another health authority, where researchers provided ongoing phone assistance, and a team within the health authority was in charge of the evaluation; and a hands-off approach where the framework draft and one phone call were provided to the lead person on musculoskeletal injury prevention with a provincial health and safety initiative.

Results

Table 1 lists the 11 resources required by an OHS programme in order of relative priority according to the Delphi panel. Three had top priority: OHS staff time, training the workers and programme planning, promotion and evaluation. For accounting purposes, resources can be broadly grouped into three categories: worker training and education, labour (staff time devoted to plan, implement, and operate the programme) and equipment and supplies.

Table S1 (available as Supplementary data at Occupational Medicine Online) lists the 31 outcomes of an OHS programme in order of priority. Five had top priority: number of injuries, safety climate, job satisfaction, quality of care and work days lost.

Figure 2 illustrates a way to integrate most but not all of these outcomes, connecting them to the delivery of high quality health care. It is proposed that the outcomes of an OHS programme come about through three main paths: by preventing occupational injuries and illnesses, by changing the process of delivering care and by changing the safety climate of the organization (defined as the sum of all workers’ opinions and experiences about the overall level of safety in the workplace).

There were minor differences in ranking resources, but major differences in ranking outcomes between stakeholder groups. In resources, OHS staff was initially ranked as number one by employers and as number four by workers. The opposite applied to training the worker, initially ranked as number four by employers and as number one by workers. All groups eventually concurred that these two resources have top priority in making allocation decisions in OHS in health care.

When it came to outcomes employer and insurer representatives in the panel rated the number of injuries, work days lost and quality of care as the top outcomes. Ministry representatives ranked quality of care, number of injuries and safety climate as the top outcomes, whereas employees ranked job satisfaction, meaningful return to work and safety climate as the top three. Patient representatives ranked safety climate, job satisfaction and quality of care as their top three outcomes. It is notable that worker and patient representatives highly valued the effect of an OHS programme on safety climate and job satisfaction.

After considering panel results, the working group of researchers and stakeholders recommended a framework that combined resources and outcomes in a cost-effectiveness analysis (total amount of resources used to prevent one injury, illness or work day lost) and a partial cost-benefit analysis of monetary outcomes (overall societal savings or costs over 5 years) with separate specification of non-monetary outcomes (impact on the process of care, on the quality of health care and on the safety climate of the organization). Table 2 lists the seven principles that are the foundation of the proposed framework for economic evaluation of OHS programmes in health care. Table 2 also includes guiding questions that can be used to assess whether an evaluation adhered to the principles.

The trials of the framework under different circumstances had different results. The hands-on approach where researchers took responsibility for analysis resulted in an evaluation that showed a ceiling lift coaching programme prevented an additional 68 injuries with resources of $798690 (2006 Canadian dollars) and monetary outcomes of $800470 (E. Tompa, submitted forpublication). The in-house evaluation was compromised by competing time demands on the designated staff and it was eventually abandoned. The hands-off approach lead to greater attention in choosing appropriate comparator facilities and the writing of a ‘business case’ for the provincial programme was eventually handed out to external consultants.

Three important messages for implementation of the framework in conducting economic evaluations arose from the trial runs. Firstly, a minimum of three team members are required when a decision maker commissions an economic evaluation: a local content expert familiar with the OHS programme, a data analyst familiar with data sources and a person with background in economic evaluation. Secondly, choosing an appropriate comparator that would otherwise have the same outcomes as intervention units/facilities is crucial and the units/facilities need to be large enough and observed long enough. Thirdly, the evaluation needs effort and time commitment. The core personnel listed above need to have sufficient time available to them, commensurate with their tasks and skills. If this is not possible, outside collaboration with consultants or university researchers should be sought.

A monograph presenting the final framework, refined after the trial runs, is available online at the WorkSafeBc web site [15]. It contains a full description of each of the seven principles, details on the assessment of resources and outcomes and a guide on conducting measurements and calculations needed to complete the economic evaluations.

Discussion

The participatory action research process described in this paper resulted in a methods framework for use in economic evaluations of OHS programmes in the public health care sector. The principles and methods may be applicable to other publicly funded service sectors, such as education and municipal services.

From the literature review, we learned that economic evaluations of OHS interventions published in peer-reviewed journals have methodological shortcomings and provide fair to moderate evidence of the cost-benefit of a few interventions. From the key informant interviews, we learned not to assume common perspectives or language among stakeholders and researchers. Though decision makers talked of conducting economic evaluations, there was little scientific rigour in the use of the terms and a blurring of what constitutes a cost versus a consequence. This prompted the researchers to speak of resources and outcomes instead and to provide specific definitions for each resource and outcome.

The Delphi panel had minor differences in ranking resources, but major differences in ranking outcomes of an OHS programme. Our results suggest that patients, workers and Ministry of Health Services would not support economic evaluations of OHS programmes that focus strictly on monetary outcomes and do not consider effects on the quality of health care provided and the wellbeing of the workforce. The panel was clear that quality of patient care is a priority outcome that ought to be considered in any economic assessment. Worker and patient representatives preferred early indicators of workforce wellbeing, such as safety climate and job satisfaction, rather than late indicators such as the quality of life of workers that are injured at work. Except for worker representatives, panel members ranked direct financial impacts on injured or sick workers, such as lost income and out-of-pocket expenditures, as low priority outcomes.

Three resources (OHS staff time, training the workers and programme planning, promotion and evaluation) and five outcomes (number of injuries, safety climate, job satisfaction, quality of care and days lost) had top priority across the whole stakeholder panel. It could be argued that any economic analysis of an OHS programme in public health care that does not consider these priority resources and outcomes is seriously flawed.

From the environmental scan, we learned the importance of knowing what data are available before commencing an evaluation. From the working group, we learned that senior decision makers would not commit to ongoing half-day or full day discussions, but that their input was better gathered in one-on-one interviews and presentations of a draft framework.

The project’s main strengths are the involvement of representatives from all stakeholder groups relevant to OHS programmes in health care and its grounding by engaging decision makers who review and consider economic analyses in reaching decisions about resource allocation. This may also be seen as a weakness, as solutions tailored to a local environment may not have universal validity or be broadly applicable to other settings. That said, a review of the seven principles supporting the framework suggests these are general enough to inform the conduct of economic evaluations in other settings. Furthermore, when the guiding questions associated with the principles are asked of evaluations published in the peer-reviewed literature, many methodological weaknesses become readily apparent, in particular the lack of appropriate consideration of multiple perspectives (Principle 1) and the lack of explicit reporting of assumptions and their likely impact on the results (Principle 6).

Another possible weakness is that the snowballing technique commonly used in qualitative research and used here to select interview participants may have resulted in over-representation of participants who shared particular views.

The framework resulting from this project is primarily aimed at practitioners and other OHS stakeholders, rather than researchers. As such, the framework monograph (available at the WorksafeBC website [15]) avoids technical terminology and includes multiple examples.

The principles themselves are worded as if they were directed to staff in health care organizations charged with conducting economic evaluations and the framework monograph provides specific guidance on measurement and calculations needed to complete the evaluation.

Decision makers can use the guiding questions in Table 2 to judge to what degree a given economic evaluation adheres to the principles and thus how much its results can be trusted. More widely, sector stakeholders can use the principles and guiding questions to assess and decide jointly whether to support and act on the results of a given evaluation.

Although the framework is primarily directed at practitioners and stakeholders, it raises some interesting research questions. It would be important to test whether application of this framework indeed results in evaluations that are of improved quality and more compar able to each other. It would be also interesting to assess the similarities and differences between this framework, directed primarily to practitioners, and frameworks used by researchers.

In conclusion, we present a participatory action research approach that resulted in a framework for conducting economic evaluations of OHS programmes in public health care. We believe that using this framework will increase the quality of economic evaluations and facilitate programme comparisons for evidence-informed resource allocation. With minor adaptations, the framework may be useful in other public service sectors.

Funding

Funding for this research has been provided by the WorkSafeBC Research Secretariat (RS2009-OG12).

Conflicts of interest

None declared.

References