|
|
Volume 5: No.
2, April 2008
TOOLS AND TECHNIQUES
Adapting Logic Models Over
Time: The Washington State Heart Disease and Stroke Prevention Program Experience
Marilyn Sitaker, MPH, Jan Jernigan, PhD, Susan Ladd, MS, Miriam Patanian, MPH
Suggested citation for this article: Sitaker M, Jernigan J, Ladd S, Patanian
M. Adapting logic models over time: the Washington State Heart Disease and
Stroke Prevention Program experience. Prev Chronic Dis 2008;5(2).
http://www.cdc.gov/pcd/issues/2008/
apr/07_0249.htm. Accessed [date].
Abstract
Logic models are graphic representations of the relationship between program activities and their intended effects and are used for both program planning and evaluation. Logic models can provide an important foundation for program evaluation by identifying evaluation questions that most appropriately assess program processes and outcomes and by guiding measurement decisions. We demonstrate how logic models can be used to plan program evaluation by describing the adoption of logic modeling by
the Washington State Heart Disease and Stroke Prevention Program (WaHDSPP) and by specifying the changes in process and use of logic models since the program’s initial funding. Our paper describes how a logic model was used in generating the program evaluation plan for the WaHDSPP, including the identification of evaluation questions and development of indicators to track progress effectively.
We describe the use of evaluation results, as well as steps state programs can take to use
logic models in program evaluation.
Back to top
Introduction
The benefits of logic models for program planning and evaluation are evident. As a result, descriptions of logic models are becoming more commonplace in program and evaluation literature, and funding announcements now routinely encourage their development and use. For example, when the Centers for Disease Control and Prevention’s (CDC’s) Division for Heart Disease and Stroke Prevention (DHDSP) released
its 5-year Funding Opportunity Announcement (FOA) for 2007 for state
programs to address heart disease and stroke, applicants were required to develop logic models. However, although application requirements now more commonly require logic models, the subsequent use of these models once grants have been awarded may be inconsistent. We describe the Washington State Heart Disease and Stroke Prevention Program’s (WaHDSPP’s) experience with program logic models and their use in evaluation planning. We also describe how the development and use of program
logic models evolved over time. By describing how the WaHDSPP developed its logic model and used it to construct an evaluation plan, we hope to demonstrate the link between development and use. Subsequently, we hope to demonstrate the utility of logic models in public health programs and stimulate the construction of better and more usable evaluation plans.
Back to top
Overview of the WaHDSPP
The goal of the WaHDSPP is to build statewide support for programs targeting people who have heart disease or history of stroke, as well as people who are at high risk for developing either condition (e.g., people with hypertension, people with high blood cholesterol, people with diabetes). The WaHDSPP first received funding from CDC’s DHDSP in 2003. From 2003 through 2006, the program was staffed by a full-time program manager and half-time epidemiologist. During this time, the
program relied heavily on cross-program staff from the Washington State Department of Health, internal and external partners, and the WaHDSPP advisory council to implement its activities.
Back to top
History of Logic Model Use by the WaHDSPP for Planning and Evaluation
Logic models were not a requirement for funding by CDC, and during the first
2 years (2003–2004) of the program, no logic model existed to describe or assist in evaluating the WaHDSPP. Activities conducted by the WaHDSPP were chosen on the basis of their alignment to CDC DHDSP program priorities, the existing program capacity, relationships with internal and external partners, and previous experience. During this time, the WaHDSPP focused on capacity building and needs assessment
activities, including developing The Burden of Heart Disease and Stroke in Washington State (1) (referred to hereafter as “the Burden document”), forming a statewide advisory council, and conducting an environmental scan of extant population resources for prevention and control of heart disease and stroke. Using the CDC DHDSP framework included in
A Public Health Action Plan to Prevent Heart Disease and Stroke (2) (referred to hereafter as "the Action Plan") and findings from the environmental
scan and the Burden document, the advisory council developed the Washington State Public Health Action Plan for Heart Disease and Stroke Prevention and Management (3) (referred to hereafter as “the State Plan”). Because projects had not been implemented and the logical link to expected outcomes of these capacity activities had not been identified through a logic model, the WaHDSPP did not engage in any program evaluation activities during 2003–2004.
Although not a requirement for continued funding, logic model use was encouraged by the DHDSP to assist in program planning and evaluation, and the DHDSP provided training and technical assistance to states for the development and use of logic models. Recognizing the benefits of logic models, the WaHDSPP developed a simple logic model that described its planned activities and expected outcomes as part of the 2005–2006 continuation application. Each activity in the 2005–2006
logic model (Figure 1) was linked to an essential public health service objective from the WaHDSPP State Plan. However, as the only staff members for the WaHDSPP, the program manager and epidemiologist produced the logic model in isolation, with no input from cross-program staff or partners.
[ View enlarged image and descriptive text ]
Figure 1. Washington State Specific Logic Model Framework for Heart
Disease and Stroke Prevention Program. (HDSPP indicates Heart Disease and Stroke
Prevention Program; CDC, Centers for Disease Control and Prevention; and HD&S,
heart disease and stroke.)
To assist in evaluation planning, a logic evaluation plan (LEP) was developed in Excel (Microsoft Corporation, Redmond, Washington), linking evaluation measures to each activity and outcome in the logic model. The 2005–2006 LEP for one activity, the Washington State Collaborative, is shown in Figure 2. The evaluation included some process measures, as well as measures of progress toward short-, medium-, and long-term outcomes. The LEP not only provided simple measures for assessing
program activities and outcomes, it also helped the WaHDSPP identify and address measurement gaps. For example, to measure the impact of the Washington State Collaborative on the medium-term outcome better management of chronic conditions, the WaHDSPP developed and implemented a statewide survey of health care providers to assess implementation of the planned care model by primary care physicians and to assess providers’ use of evidence-based guidelines.
[ View enlarged image and descriptive text ]
Figure 2. Washington State Heart Disease and Stroke
Prevention Program (WaHDSPP) Logic Evaluation Plan, 2005–2006. (WSC
indicates Washington State Collaborative; CVD, cardiovascular disease; FHCQ,
Foundation for Healthcare Quality; HBP, high blood pressure; BRFSS,
Behavioral Risk Factor Surveillance System; HD, heart disease; PA,
physical activity; and CHARS, Comprehensive Hospital Abstract Reporting
System.)
Planning for 2006–2007 improved previous years’ efforts for several reasons. First, the planning process was a collaborative effort. Second, evaluation and surveillance data, collected as part of the previous years’ evaluation efforts, were used in program planning efforts. Given the involvement of internal and external partners in the WaHDSPP, the program staff realized the potential benefits of the partners’ input in planning program activities and evaluation
efforts. As a result, partners and key stakeholders were invited to participate in the planning retreat for the 2006–2007 continuation application. The involvement of partners and stakeholders responsible for implementing program activities added depth and richness to the discussion. This collaborative process allowed for an in-depth discussion of the barriers to and facilitators of implementing program activities, factors that
affect outcome attainment, and future program directions.
These discussions were immensely helpful in developing work plans that linked to successful outcomes and that were both feasible and appropriate, given WaHDSPP resources.
The WaHDSPP program manager scheduled time during the planning retreat to review surveillance, evaluation, and assessment results from the previous year, ensuring that these data were taken into account in the proposed program activities. Involving partners and stakeholders in this process provided an opportunity to discuss results with people who were the most knowledgeable about the activities, which led to a better understanding of program activities by the entire planning committee and
more realistic approaches to program improvement. In addition, being part of the program planning effort increased buy-in and “ownership” of the WaHDSPP. Once
work plans were developed, the logic model and the LEP were updated to reflect proposed activities and outcomes. Collaborating on the logic model revisions provided new clarity in the short-, intermediate-, and long-term objectives for both staff and partners, and this clarity facilitated the selection of indicators.
The planning retreat for 2007–2008 focused on responding to the FOA released by the DHDSP and included newly hired WaHDSPP staff. As during the previous year,
key partners and stakeholders were involved in planning, and evaluation and
surveillance data were used to guide planning discussions. One new feature in the planning process involved categorizing proposed activities to align with the spectrum of prevention outlined in the Action Plan. The Action Plan, developed
collaboratively by the DHDSP and its national partners, describes a comprehensive approach to addressing heart disease and stroke, from preventing risk factors to
preventing recurrent cardiovascular events. The FOA encouraged states to use the Action Plan to guide development of their applications. By comparing proposed activities
with the Action Plan, the WaHDSPP was able to identify a gap in its interventions, leading to the development of a new objective for the assessment of rehabilitation
capacity.
Having additional program staff available to develop work plans for the application
enabled partners to collaborate more closely than in years past to revise the logic model. The updated logic model (Figure 3) demonstrated the WaHDSPP’s alignment with the Action Plan, categorizing activities and their outcomes as either primary prevention, acute event, or secondary prevention, and reflected more mature thinking on the part of the WaHDSPP about the specific changes expected as a result of
program activities.
[ View enlarged image and descriptive text ]
Figure 3. Washington State Heart Disease and Stroke
Prevention Program (WaHDSPP) Logic Evaluation Plan, 2007–2008. (HDSPP indicates
Heart Disease and Stroke Prevention Program; CDC, Centers for Disease
Control and Prevention; EMS TAC, Emergency Medical Service Technical
Advisory Committee; NCQA, National Committee for Quality Assurance; EPHS,
essential public health services; MIS, management information system; HD&S, Heart Disease and Stroke; MI, myocardial
infarction.)
A second feature of the 2007–2008 planning process involved the use of the CDC DHDSP’s
Developing an Evaluation Plan (4). This guide is one in a series developed by the DHDSP to assist states in their evaluation efforts. The WaHDSPP used the guide to systematically develop evaluation questions, leading to more well-rounded evaluation plans for individual activities and for the overall program (Table). The evaluation terms used by the program were revised to be consistent with
those provided by CDC; the term measures was replaced with indicators and
medium-term outcomes were referred to as intermediate outcomes. The evaluation plan replaced the LEP and provided more details on methods and data sources that would be used to answer key evaluation questions.
Back to top
Relationship Among Activities, Evaluation Plans, and the Logic Model
This section describes an outreach activity proposed for 2007–2008 and how it is being evaluated on the basis of the logic model. The African American Awareness and Screening Project takes place in barbershops and hair salons that have predominantly African American clients in two Washington counties with large African American populations. The project, which began in 2006, consists of
training barbers and stylists to provide information about hypertension and the signs and
symptoms of heart attack and stroke to their clients. Blood pressure readings are taken, and clients with high blood pressure are encouraged to see a health care provider.
Links to the program logic model
As shown in the 2007–2008 logic model, the African American Awareness and Screening Project directly contributes to the supporting (i.e., short-term) outcome of increased public awareness of heart disease and stroke signs and symptoms. This activity is also linked to the intermediate outcomes of increased recognition of symptoms and reduced time between recognition of symptoms and taking action to call 911 and to greater control of risk factors (e.g., high blood pressure, elevated
cholesterol levels), on the basis of the theory that increased awareness of high blood pressure and high cholesterol levels will lead to improved control of these risk factors.
Evaluation approach
Evaluation planning focused on the stage of development of the activity. Because the African American Awareness and Screening Project was implemented in 2006, evaluation efforts targeted the implementation of the activity and assessment of the outcomes of increased public awareness of signs and symptoms of heart attack and stroke, as well as control of risk factors.
Furthermore, the logic model assisted in developing the evaluation questions. By looking at the logic model to determine
expected activities and outcomes, WaHDSPP staff were able to develop and prioritize the questions that the evaluation should answer, including the fidelity of the project, the impact of the project, and lessons learned and implemented.
The evaluation involves a mixed-methods approach, including both qualitative and quantitative data, and includes 1) the identification of facilitators of and barriers to the project through key stakeholder interviews, 2) a review of quarterly progress reports to evaluate fidelity to the original project plan, and 3) focus group sessions with participating barbers and stylists to evaluate trainings, project implementation, and perceptions about what worked and what did not work. Outcome
evaluation will be conducted through review of screening results and follow-up to determine how many screened participants with high blood pressure were treated by a health care provider.
Back to top
Lessons Learned and Future Directions
Reflecting on the past 6 years of program activity, WaHDSPP staff identified several key lessons that have facilitated the use of programmatic logic models:
- Logic models have assisted the WaHDSPP in developing its theory of change. By stating the theory of change, the program can better identify intermediate steps — and related indicators — that precede long-term outcomes and identify the incremental steps that precede short-term outcomes, allowing the program to monitor progress in a more proximal manner.
- The WaHDSPP has used the annual occasion of preparing a continuing application as a time for collective reflection, not only on how to better refine existing activities or develop new ones, but also as a time to critically examine and improve the logic model and evaluation tools.
- The logic model now categorizes activities and outcomes according to their place in the spectrum of prevention, allowing WaHDSPP staff to visually track the expected impacts resulting from program activities over time and to think more clearly about how achievements in impacts at one stage of prevention may facilitate achievements at another stage by shrinking the size of the vulnerable population (5).
- Periodic reflection allows the program to incorporate new knowledge, such as information and resources from the CDC DHDSP, peers, and the literature to enhance evaluation planning.
- Logic modeling is integral to program planning, implementation, and evaluation. The model set forth in CDC’s six-step
Framework for Program
Evaluation in Public Health describes this collaborative approach (6).
- Evaluation and logic model development should be conducted in partnership with program stakeholders, with results feeding directly back into ongoing program planning and progress monitoring. Logic models and evaluation plans are dynamic tools to guide the program in carrying out activities but should always be developed and refined in partnership with activity work plans and the key staff and partners involved in the work.
These lessons have been incredibly valuable in informing and improving the direction of the WaHDSPP and demonstrate how logic models are useful in public health program planning and evaluation.
Back to top
Author Information
Corresponding Author: Susan Ladd, MS, Behavioral Scientist, Division for Heart Disease and Stroke Prevention, Centers for Disease Control and Prevention, 4770 Buford Highway NE, Mailstop K-47, Atlanta, Georgia 30341. Telephone: 770-488-5448. E-mail: sladd@cdc.gov.
Author Affiliations: Marilyn Sitaker, Miriam Patanian, Washington State Department of Health, Olympia, Washington; Jan Jernigan, Division for Heart Disease and Stroke Prevention, Centers for Disease Control and Prevention, Atlanta, Georgia.
Back to top
References
- Sitaker M, Kemple A, Hartwick N. The burden of heart disease and stroke in
Washington State. Olympia (WA): Washington State Department of Health, Community and Family Health; 2004. http://www.doh.wa.gov/cfh/heart_stroke/publications/ hdsp_burden.pdf.*
Accessed August 9, 2007.
- Centers for Disease Control and Prevention.
A public health action plan to prevent heart disease and stroke. Atlanta (GA): U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; 2003. http://www.cdc.gov/dhdsp/library/action_plan/index.htm.
- Washington State Department of Health.
Washington state public health action plan for heart disease and stroke prevention and management. Olympia (WA): Washington State Department of Health, Community and Family Health. http://www.doh.wa.gov/cfh/heart_stroke/state_plan.htm.*
Accessed August 9, 2007.
- Centers for Disease Control and Prevention.
Developing an evaluation plan. Atlanta (GA): U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Division for Heart Disease and Stroke Prevention; 2006. http://www.cdc.gov/DHDSP/state_program/evaluation_guides/ evaluation_plan.htm.
- Milstein B, Jones A, Homer JB, Murphy D, Essien J, Seville D. Charting plausible futures for diabetes prevalence in the United States: a role for system dynamics simulation modeling. Prev Chronic Dis 2007;4(3).
http://www.cdc.gov/pcd/issues/2007/jul/06_0070.htm.
-
Centers for Disease Control and Prevention.
Framework for program evaluation in public health. MMWR Recomm Rep 1999;48(RR-11):1-40.
Back to top
|
|
