Evaluation

Once the assessment and planning phases have been conducted, and interventions have been selected for implementation, the final stage of designing a workplace health program involves decisions concerning the monitoring and evaluation of program activities. Just as assessment data are critical for evidenced-based program planning and implementation, so too is evaluation to prove that workplace health interventions have been effective and build the business case for continuing them. Ideally, evaluation plans should be developed as part of the planning process and before beginning the intervention activities.

What is program evaluation

The definition and approach to program evaluation described here are based on the Centers for Disease Control and Prevention (CDC) Framework for Program Evaluation in Public Health.¹  

As noted in the framework, effective program evaluation  systematically examines the implementation and results of strategies and interventions  with the aim of using findings to improve those actions.  As such, it’s important that the evaluation approach be useful, feasible, ethical, and accurate. The framework guides professionals through a series of steps that are intended to ensure their  program evaluations meet these standards and are most likely to yield results that will be used for program improvement.

The framework is summarized in the figure below. It includes six steps and four standards for effective program evaluation.

CDC framework for program evaluation

[A text description of this chart is also available.]

 

Besides producing practical, useful evaluation designs, the framework encourages an approach aimed to integrate evaluation with program planning and routine program operations. The early steps of engaging stakeholders and describing the program yield insights for planners and implementers as well as evaluators. By involving diverse program stakeholders such as business leaders, managers, and employees, not just evaluation experts, the evaluation design can be a driving force for planning health strategies, improving existing programs, and demonstrating the results of resource investments.

Evaluation helps define the value of the program

The level of corporate investment in workplace health programs will depend on the perceived value of these programs. Decisions about the value of a workplace health program may vary company by company or even by stakeholder and decision maker within a single company. That’s why the framework includes stakeholder engagement as the first step—to identify early on the different preferences and needs stakeholders will bring to assessing value and success of a program. Value consists of three dimensions: merit (was the program a quality program), worth (cost-effectiveness, did the benefits exceed the costs), and significance (did it accomplish something important). A program may score well on all these dimensions or just some. A program may have merit but be judged as not worth the cost, perhaps because it is driving an outcome that is not very important to key decision makers.

Defining the program and forming the evaluation

In the framework’s early steps, it is important to define the program in sufficient detail so that those conducting the evaluation can engage key stakeholders, identify their preferences and needs, and use that information to decide which parts of the program must be evaluated to determine if the activity is worth continuing or improving.  A series of questions are part of these tasks:

• What is the “program?” (What are the key activities of the program and what are the key outcomes it aspires to achieve?)
• What is the larger “environment” of the program? (What resources are available to conduct the activities?  What elements of the larger context or situation might help or hinder program success?)
• What aspects of the program will be considered by decision makers (e.g., senior leaders) when judging program performance? (e.g., Quality implementation of activities?, Achieving outcomes?, Which outcomes?)
• What benchmarks (i.e., type or level of performance), if any, must these activities or outcomes reach for the program to be considered successful?
• What kinds of evidence on performance are most credible to key decision makers in making judgements about how the program has performed?
• How will the lessons learned from the inquiry be used? (e.g., To help improve program effectiveness in the next round? To make a decision on program continuation?)

Evaluate both outputs and outcomes over time

Clearly, these questions encompass both program implementation and program effectiveness.  Either can be the focus of a given evaluation, but usually aspects of both need to be addressed.  Ideally, evaluation planning has been integrated with implementation planning, and it is a simple task to identify the main processes (e.g., activities or outputs) employed by the program as well as the intended short-term and long-term outcomes (which may be stated in the implementation plan as program objectives or goals) for the planned programs.

Process measures

Process measures examine all the steps and activities taken in implementing a program and the outputs they generate, such as the number and type of educational materials for a stress management class that are developed and given to employees. They are useful for keeping implementation of the program on track and also for determining if program implementation met the quality and other standards to which the program aspired.  This is important so that, if a program does not achieve its intended outcomes, it can be determined if the program was the wrong approach or if it was a strong program that simply was not implemented correctly.  Process measures also can assess issues such as the costs of operating a program, the numbers of employees reached, the most successful program locations, or comparisons of the program’s design and activities to others.  

Outcome measures

Outcomes are events or conditions that indicate program effectiveness.  They generally are displayed as short-, intermediate, or long-term.  Long-term measures, in the context of workplace health promotion, typically relate to things like reductions in disease or injury and the costs associated with them. These are often similar to the goals of the program and these long-term outcomes often take years to observe.

Short- and intermediate term measures, by contrast, relate to the intermediate steps and “drivers” necessary to achieve the long-term outcomes, such as individual employee reductions in healthy lifestyle risks such as tobacco use, or process changes such as implementing a new health-related policy or benefit at the organizational level that supports lifestyle changes.

Whatever program components have been included in the evaluation, it is important that they be both measurable and realistic. “Measureable” means that the activity or outcome has been phrased in a way that lends itself to data collection. “Realistic” means that the outcomes are ones which a well-implemented intervention can be reasonably expected to produce.  When goals/outcomes are unrealistic it is often because 1) the program has not been in place long enough to produce the outcome of interest or 2) the program simply is not broad or intense enough to produce it.

Finally, once the key outcomes have been identified and written as measureable and realistic, identify when each will be measured (e.g., every 6 months, every 12 months, etc.), assign this tracking responsibility to a specific individual, and require regular reports.  Some outcomes may be measured only early or late in the program, while others may be measured several times, as long as the program is active.

Baseline measures

Some subset of the process or outcomes measures may need to be collected from the very start of the intervention as “baseline” measures.  The need for baseline measures is one key reason for designing the evaluation plan before implementation begins because they establish a starting place and frame of reference for the workplace health program. These can usually be developed from data collected during the initial assessment activities and summarized in the assessment final report.

Baseline measures determine where the organization currently is on a given health problem (e.g., the percent of employees who use tobacco) or issue (e.g., the percent of employees who are aware of recommended physical activity guidelines) and inform the benchmarks/targets against which program managers and decision makers will assess program performance.

Baseline measures can also be used to describe the current level of program activities and allow measurement of the program’s progress (e.g., process measures) over time such as the number of new physical activity classes offered to employees or the establishment of a new health benefit.

Outcomes of interest

The evaluation guidance so far has been general guidance that can apply to any outcome.  The evaluation module has been organized by the specific health topics listed above, and for each one, potential measures for the following four main outcome categories of interest to employers and employees have been developed.

Worker productivity

• Workplace health programs that improve employee health by reducing, preventing or controlling diseases can affect worker productivity
• Improvements in physical, mental, and emotional health enhance stamina, concentration, and focus leading to greater work output
• The cost savings of providing a workplace health program can be measured against:
  • Absenteeism among employees due to illness or injury
  • Reduced overtime to cover absent employees
  • Costs to train replacement employees

Health care costs

• An investment in employee health may lower health care costs, insurance claims, or worker’s compensation costs
• A workplace health program has the potential to both keep healthy employees in a “low-risk” category by promoting health maintenance, and to target unhealthy employees in higher-risk categories, therefore lowering overall health insurance costs
• Potential evaluation measures for health care costs following an assessment of health care and pharmaceutical claims include:
  • Changes in the number and type of health insurance claims over time (e.g., hospitalizations, outpatient visits, or pharmacy claims)
  • Changes in health care utilization (e.g., use of preventive health services such as cancer screening)
  • Quality of care indicators (e.g., percent of employees with high blood pressure that is being controlled through medication)
  • Changes in direct medical costs due to illness or injury

Health outcomes

• Participation in health promotion activities through a workplace health program allows individuals to develop knowledge, self-management and coping skills as well as build a social support network among coworkers, supervisors, and family. These skills can facilitate adopting healthy behaviors in the short-term and lead to changes in physical, mental, and/or emotional health in the long-term
• Potential evaluation measures can compare the health status of program participants over time such as:
  • Targets for the optimal levels of health status (e.g., percent of employees reporting good or excellent health before and after a health promotion program or intervention)
  • Acceptable levels of disease, injury or disability (e.g., percent of employees diagnosed with colorectal cancer before and after a screening program or number of employees who experience a fall at work before and after a falls prevention intervention)
  • Prevalence of risk factors (e.g., percent of employees who are overweight/obese before and after a weight management program)

Organizational change

• Creating a culture of health, where health promotion is a valued part of the normal work day environment, will address employers’ health concerns for their employees and enhance their competitiveness by engaging employees, and presenting the company as an attractive place to work. A culture of health is achieved when the organization’s structures, policies, procedures, or practices are aligned to support or maintain health
• Potential evaluation measures that relate to this type of organizational change include documenting the process and effects of changes (e.g., changes in employee morale) made to organizational structure, policies, procedures, or practices   

 Top of Page