ORIGINAL RESEARCH

Evaluation of the Healthy Schools Program: Part I. Interim Progress

Margaret Beam, PhD; Ginny Ehrlich, MPH, MS; Jessica Donze Black, RD, MPH; Audrey Block, MSW, JD; Laura C. Leviton, PhD

Suggested citation for this article: Beam M, Ehrlich G, Donze Black J, Block A, Leviton LC. Evaluation of the Healthy Schools Program: Part I. Interim progress. Prev Chronic Dis 2012;9:110106. DOI: http://dx.doi.org/10.5888/pcd9.110106.

PEER REVIEWED

Abstract

Introduction
Federal and state policies identify schools as a setting to prevent childhood obesity, but schools need better health-promoting strategies. The objective of this study was to evaluate interim progress in schools receiving hands-on training from the Healthy Schools Program, the nation’s largest school-based program aimed at preventing childhood obesity. The 4-year program targets schools with predominantly low-income, African American, or Hispanic students.

Methods
In 2010 we assessed schools that enrolled in the 2007-2008 and 2008-2009 school years. School representatives completed an inventory of 8 content areas: policy and systems, school meals, competitive foods and beverages, health education, physical education, physical activity outside of physical education, before- and after-school programs, and school employee wellness. Schools’ baseline inventory was compared by t test with the most recent inventory available.

Results
Schools made significant changes in all content areas, and effect sizes were moderate to large.

Conclusion
Participating schools improved environmental policies and practices to prevent childhood obesity. The program is a resource to implement recent federal and state policies.

Top of Page

Introduction

Childhood overweight and obesity are serious problems for the United States, where prevalence has nearly doubled during the past 20 years (1). More than 95% of school-aged youth in the United States attend school, and it is the place where children and youth spend the most time outside the home (2). As a result, national, state, and local policy makers focus on the schools for obesity prevention. In 2010, the report to the President from the White House Task Force on Childhood Obesity (3) and the passage of the Healthy, Hunger-Free Kids Act (4) continued the policy momentum created with passage of the Child Nutrition and WIC Reauthorization Act of 2004 (5). The 2004 act required that all school districts participating in the federally funded school meals programs develop a school wellness policy addressing school nutrition and physical activity (5).

Although the policy momentum for school wellness efforts is clear, research suggests that the 2004 law had modest effects at best (6). Local district policies varied greatly in comprehensiveness, and several studies found no significant changes in schools’ nutrition and physical activity environments after passage of these policies (7-10). Given that the Healthy, Hunger-Free Kids Act requires most public school districts to continue the wellness efforts mandated by the 2004 law, we sought to identify strategies that will help promote these efforts in schools.

The objective of this study was to evaluate the progress of schools participating in the Healthy Schools Program (HSP) as they implement health-promoting policy, practice, and environmental changes. The 4-year program targets schools with predominantly low-income, African American, or Hispanic students. HSP is the largest program in the nation devoted specifically to school-based obesity prevention. HSP is part of the Alliance for a Healthier Generation, founded by the American Heart Association and the William J. Clinton Foundation to reduce prevalence of childhood obesity in the United States. The program is funded by the Robert Wood Johnson Foundation along with local, state, and philanthropic partners. This article addresses 2 questions: 1) how much improvement do schools make when HSP gives them hands-on training and technical assistance? and 2) which improvements are more prevalent? This study does not include the thousands of schools that have signed up exclusively for on-line participation in HSP. A companion article (11) addresses training and technical assistance (TTA) factors associated with school progress.

Top of Page

Methods

Study design

Schools recruited during the 2007-2008 and the 2008-2009 school years completed baseline and follow-up inventories of policy, practice, and environment indicators. Baseline was then compared with the most recent inventory completed by schools as of December 2010. HSP is a 4-year program, so this interim evaluation represents 2 snapshots of progress for schools at various stages in the program. Institutional review board approval was obtained from the RMC Corporation Human Subjects Protections Committee.

The Healthy Schools Program

HSP builds schools’ capacity to implement evidence-based policies and programs that promote and provide access to healthy foods and physical activity. HSP provides TTA at no cost, online tools for step-by-step practice standards and continuous improvement, and a national recognition program that recognizes schools reaching certain standards. HSP normally provides TTA for 4 years, although some schools complete TTA early. Regional HSP relationship managers recruit the schools, whose principals then designate representatives to undergo TTA. During a 4-year period, the relationship managers lead the school representatives through 9 highly structured train-the-trainer sessions that focus on a 6-step change process. These steps are 1) formation of a school wellness council; 2) completion of an assessment, the HSP Inventory; 3) local prioritization and action planning; 4) technical resource development and brokering; 5) implementation support; and 6) monitoring and evaluation of progress through updates to the HSP Inventory. Schools complete training at their own pace.

Through these sessions and other contacts, school representatives train relevant school personnel to achieve best practices in 8 content areas, described in the HSP Framework (www.healthiergeneration.org/schools.aspx?id=3470). These content areas are 1) school-level health policy, infrastructure, and systems; 2) school nutrition programs; 3) competitive foods and beverages (those sold outside of the school meals programs); 4) physical activity opportunities before, during, and after school; 5) physical education programs; 6) health education programs; 7) before- and after-school programs; and 8) school employee wellness programs. Training content depends on schools’ individual action plans, but relationship managers encourage schools to attempt new improvements over time.

Relationship managers contact schools between these sessions and arrange for TTA by 7 national content experts who tailor technical assistance to school needs and action plans. Through the HSP website, participating schools can access an online database containing more than 800 resources and 8 online topical tool kits aligned to the framework and vetted by the American Heart Association’s Science Advisory Committee. School representatives receive a biweekly electronic newsletter with information about grant opportunities, new wellness resources, school successes, and current research.

Program schools

We pilot-tested the HSP program in 224 schools in 13 states in 2006-2007 and performed factor analysis to refine the outcome measures. In the 2007-2008 and 2008-2009 school years, we recruited schools in these and 20 additional states and the District of Columbia. We recruited schools in the additional states because their adult obesity rate was 25% or higher, or they had the highest statewide obesity rates among children aged 10 to 17 years (12,13). The relationship managers targeted schools with predominantly low-income or racial/ethnic minority populations and recruited from feeder webs (elementary and middle schools that feed into 1 high school). Although we recruited individual schools, not school districts, 4 entire urban school districts also adopted HSP.

Recruitment consisted of signing a contract and attending an initial training session. Schools were HSP participants if, within 1 year of recruitment, they continued to take part in TTA, submitted an action plan, or submitted follow-up information about progress.

Data collection

The HSP Inventory is an online tool that measures schools’ progress over time (Appendix). Submission is voluntary and no incentives are provided, but we encourage school representatives to update their inventories at least annually to assess progress and realign priorities. The school representatives collected the necessary information from food service directors, physical education teachers, principals, and others. We chose online submission because it yielded data at a reasonable cost and provided schools with a tool for continuous quality improvement. School staff provide better evaluation information when feedback is useful to them for purposes such as quality improvement (14). Schools updated their inventories at their own pace. Most schools completed their baseline inventory when they started implementing HSP, and the median school had submitted its most recent follow-up inventory 6 months before December 2010 (range, 0-32 mo). The mean (standard deviation) number of months between baseline and follow-up inventories was 18.6 (8.6).

Outcome measure

An expert panel comprising thought leaders in education, health, and health promotion in academia, government, and professional associations reviewed and approved inventory items. We revised some inventory items between 2007 and 2008 to reflect emerging research on best practices, but 83 items were common to both versions. The inventory collects information for 8 content areas; item responses in each content area are dichotomized and summed to create 8 content area index scores and a total index score. The 9 indices are policy and systems (6 items), school meals (17 items), competitive foods and beverages (12 items), physical activity outside of physical education (6 items), physical education (14 items), health education (10 items), before- and after-school programs (6 items), school employee wellness (12 items), and total (all 83 items). Reliability was excellent for the total index (Cronbach α = .90) and fair to good for 6 content areas (Cronbach α  = .63-.87). The indices for policy and systems and physical activity outside of physical education had lower Cronbach α (.57 and .52, respectively). In 2009, we directly assessed validity in 21 pilot-year schools that submitted a revised HSP Inventory. School meals and competitive foods and beverages could not be directly assessed. Agreement with the inventories ranged from 55% to 78%; discrepancies most often were due to out-of-date inventories.

Data analysis

Using SPSS 18 (SPSS, Inc, Chicago, Illinois), we conducted t tests to compare the means of the indices for the baseline and follow-up inventories. To gauge the extent of change we supplemented the significance tests by examining Cohen’s d effect sizes and percentage of schools improving in each content area. We used listwise deletion to handle missing data.

Top of Page

Results

Relationship managers recruited 80% of the schools they approached to participate. Of 1,909 schools recruited, 1,514 (79%) continued or completed HSP (Table 1). Of participants, 42 completed the pilot-year’s inventory at baseline and were ineligible for the study, leaving 1,295 (86%) of continuing and completed schools that provided both a baseline and follow-up inventory.

In both the baseline and follow-up inventory groups, the primary race/ethnicity of students at most schools was African American or Hispanic; most students were eligible for free or reduced-price lunches (Table 2). More than half the schools were in the South. We recruited urban schools in large numbers. A higher proportion of the schools that provided follow-up data were in the Midwest and South compared with the participating schools overall. Schools providing follow-up data also included more elementary schools, fewer schools in cities, and more rural schools than did overall HSP participants. All differences were significant.

The mean differences in the scores for baseline and follow-up inventories were significant for all content areas, and generated effect sizes of this magnitude are normally regarded as moderate to large (Table 3). The index for the total of all content areas showed the largest effect, followed by the school employee wellness content and school meals areas.

Top of Page

Discussion

Schools that provided baseline and follow-up inventories to HSP made moderate to large changes in policies, practices, and environment. In contrast, recent national surveys of schools indicate few improvements to prevent obesity and little change in competitive foods and beverages or physical education (17,18). The program recruited 80% of the schools it approached, and these were schools with few resources but students at greatest risk of obesity. If these schools exhibit significant changes after receiving TTA, then it is likely that HSP will also benefit other schools.

We may be underestimating the extent of change since half the inventories were more than 6 months old. Sustainability is a question for further research. Nonresponse from 12% of eligible schools is of concern, but it is likely that they too made progress. In 2008, we audited progress in schools that participated exclusively online and did not complete inventories. The majority of those schools made improvements (data not shown).

The most common improvements were in the areas of school employee wellness and school meals. Employee wellness changes were easy to make, giving schools an early “win” to maintain momentum. The schools were also motivated to change school meals to respond to recent public demands. Less common were changes in physical education and policy and systems, which are most often controlled by the school districts. Schools also changed less in the area of physical activity outside of physical education. Lower reliability for the physical activity outside of physical education category and the policy and systems category may be partly responsible. The least common changes were in before- and after-school programs, which schools implement with community partners and which did not focus on the school day.

This program evaluation had several limitations. We could not validate information about school meals and competitive foods and beverages. Schools progressed through the program and provided data at their own pace, so the evaluation is a snapshot of accumulated changes. In the absence of a control group we cannot assert any causal relationship between HSP and school progress. These schools might have made progress in the absence of HSP, but that is unlikely given national surveys that indicate little progress (17,18).

Schools participating in Healthy Schools Program TTA improved policies, practices, and environments related to childhood obesity prevention. These findings help to identify the TTA needed to implement the 2010 Healthy, Hunger-Free Kids Act (4), which requires minimum standards to promote less calorie-dense offerings in all foods sold in school and requires changes in schools to promote both physical activity and healthy eating.

Top of Page

Acknowledgments

This evaluation was supported by grant 61099 from the Robert Wood Johnson Foundation.

Top of Page

Author Information

Corresponding Author: Laura C. Leviton, PhD, Senior Adviser for Evaluation, The Robert Wood Johnson Foundation, Route 1 and College Rd East, Princeton, NJ 08543-2316. Telephone: 609-627-5952. E-mail: llevito@rwjf.org.

Author Affiliations: Margaret Beam, Audrey Block, RMC Research Corporation, Portland, Oregon; Ginny Ehrlich, Alliance for a Healthier Generation, Portland, Oregon; Jessica Donze Black, Pew Charitable Trusts, Washington, DC.

Top of Page

References

1. Ogden CL, Carroll MD, Curtin LR, Lamb MM, Flegal KM. Prevalence of high body mass index in U.S. children and adolescents, 2007-2008. JAMA 2010;303(3):242-9. DOI: 10.1001/jama.2009.2012.
2. Table 2. Single grade of enrollment and high school graduation status for people 3 years old and over, by sex, age (single years for 3 to 24 years), race, and Hispanic origin: October 2010. US Census Bureau. http://www.census.gov/hhes/school/data/cps/2010/tables.html. Accessed January 17, 2012.
3. Solving the problem of childhood obesity within a generation. White House Task Force on Childhood Obesity. Report to the President. http://www.letsmove.gov/sites/letsmove.gov/files/TaskForce_on_Childhood_Obesity_May2010_FullReport.pdf. Accessed July 28, 2011.
4. Healthy, Hunger-Free Kids Act of 2010, Pub. L. No. 111-296, 42 U.S.C. 1751 (December 13, 2010).
5. Child Nutrition and WIC Reauthorization Act of 2004, Pub. L. No.108-265 Sect. 204 (June 30, 2004).
6. Belansky E, Chriqui JF, Schwartz MB. Local school wellness policies: how are schools implementing the congressional mandate? Robert Wood Johnson Foundation Research Brief. http://www.impacteen.org/briefs.htm. Accessed March 24, 2011.
7. A foundation for the future II: analysis of local wellness policies from 140 school districts in 49 states. School Nutrition Association. http://www.schoolnutrition.org/Content.aspx?id=8504. Accessed July 28, 2011.
8. Belansky ES, Cutforth N, Delong E, Litt J, Gilbert L, Scarbro S, et al. Early effects on the federally mandated local wellness policy on school nutrition environments appear modest in Colorado’s rural, low-income elementary schools. J Am Diet Assoc 2010;110(11):1712-7. DOI: 10.1016/j.jada.2010.08.004.
9. Belansky ES, Cutforth N, Delong E, Ross C, Scarbro S, Gilbert L, et al. Early impact of the federally mandated local wellness policy on physical activity in rural, low income elementary schools in Colorado. J Public Health Policy 2009;30 Suppl 1:S141-60. DOI: 10.1057/jphp.2008.50.
10. Moag-Stahlberg A, Howley N, Luscri L. A national snapshot of local school wellness policies. J Sch Health 2008;78(10):562-8. DOI: 10.1111/j.1746-1561.2008.00344.x.
11. Beam M, Ehrlich G, Donze Black J, Block A, Leviton L. Evaluation of the Healthy Schools Program: Part II. The role of technical assistance. Prev Chronic Dis 2012;9:110105. DOI: http://dx.doi.org/10.5888/pcd9.110105.
12. Behavioral Risk Factor Surveillance System survey data. Atlanta (GA): Centers for Disease Control and Prevention; 2007.
13. National Center for Health Statistics. National Survey of Children’s Health. Hyattsville (MD): US Department of Health and Human Services; 2007.
14. Rossi PH, Lipsey MW, Freeman HE. Evaluation: a systematic approach. Thousand Oaks (CA): Sage; 2003.
15. National Center for Educational Statistics. Common Core of Data. US Department of Education. http://nces.ed.gov/ccd/. Accessed April 14, 2011.
16. Census regions and divisions of the United States. http://www.census.gov/geo/www/us_regdiv.pdf. Accessed July 28, 2011.
17. Turner L, Chaloupka FJ, Chriqui JF, Sandoval A. School policies and practices to improve health and prevent obesity: National elementary school survey results: school years 2006-07 and 2007-08. Vol 1. Chicago (IL): University of Illinois at Chicago. http://www.bridgingthegapresearch.org. Accessed July 28, 2011.
18. Johnston LD, O’Malley PM, Terry-McElrath YM, Freedman-Doan P, Brenner JS. School policies and practices to improve health and prevent obesity: national secondary school survey results, school years 2006-07 and 2007-08. Vol. 1. Executive summary. Ann Arbor (MI): Institute for Social Research. http://www.bridgingthegapresearch.org. Accessed July 28, 2011.

 

Top of Page

Tables

Table 1. School Participation Status in the Healthy Schools Program, 2007-2010

Status in 2010	Initial Yeara		Total
	2007-2008	2008-2009
Continuingb	549	932	1,481
Inactivec	38	37	75
Droppedd	85	138	223
School closed	36	30	66
Completed programe	23	10	33
Transitioned to onlinef	0	31	31
Total recruited	731	1,178	1,909

^a Initial year is based on the date of initial sessions and memorandum of understanding.
^b Schools that continued training and technical assistance and provision of inventories or action plans through December 2010.
^c Schools that did not continue training and technical assistance or update an inventory or action plan in a year.
^d Schools that chose not to continue the Healthy Schools Program.
^e Schools that completed the Healthy Schools Program.
^f Schools in Alaska that lost their relationship manager and transitioned to online participation exclusively.

 

Table 2. Characteristics of Schools Participating in the Healthy Schools Program (HSP) and Schools Providing Both Baseline and Follow-up HSP Inventory Data, 2007-2010^a

Characteristic	Number and Percentage With Characteristic
	2007-2008		2008-2009		Total
	Recruited Schools (n = 731), n (%)	Data Provided (n = 501), n (%)	Recruited Schools (n = 1,178), n (%)	Data Provided (n = 794), n (%)	Recruited Schools (n = 1,909), n (%)	Data Provided (n = 1,295), n (%)
School level
Elementary	438 (60)	306 (61)	735 (62)	518 (65)	1,173 (61)	824 (64)
Middle	138 (19)	99 (20)	186 (16)	121 (15)	324 (17)	220 (17)
High	113 (15)	77 (15)	152 (13)	101 (13)	265 (14)	178 (14)
Other/missing	42 (6)	19 (4)	105 (9)	54 (7)	147 (8)	73 (6)
Students eligible for free or reduced-price lunch, %
0-24	51 (7)	16 (3)	74 (6)	47 (6)	125 (7)	63 (5)
25-49	135 (18)	98 (20)	173 (15)	118 (15)	308 (16)	216 (17)
50-74	171 (23)	134 (27)	356 (30)	245 (31)	527 (28)	379 (29)
75-100	336 (46)	234 (47)	428 (36)	285 (36)	764 (40)	519 (40)
Missing	38 (5)	19 (4)	147 (13)	99 (13)	185 (10)	118 (9)
Primary race/ethnicity
White	211 (29)	133 (27)	432 (37)	312 (39)	643 (34)	445 (34)
African American	268 (37)	206 (41)	372 (32)	245 (31)	640 (34)	451 (35)
Hispanic	200 (27)	135 (27)	242 (21)	171 (22)	442 (23)	306 (24)
Other	20 (3)	10 (2)	52 (4)	27 (3)	72 (4)	37 (3)
Missing	32 (4)	17 (3)	80 (7)	39 (5)	112 (6)	56 (4)
School locale
City	284 (39)	206 (41)	477 (41)	281 (35)	761 (40)	487 (38)
Suburb	201 (28)	111 (22)	298 (25)	230 (29)	499 (26)	341 (26)
Small town	83 (11)	69 (14)	125 (11)	97 (12)	208 (11)	166 (13)
Rural	139 (19)	102 (20)	223 (19)	157 (20)	362 (19)	259 (20)
Missing	24 (3)	13 (3)	55 (5)	29 (4)	79 (4)	42 (3)
Regionb
Northeast	152 (21)	78 (16)	191 (16)	122 (15)	343 (18)	200 (15)
Midwest	200 (27)	155 (31)	235 (20)	162 (20)	435 (23)	317 (25)
South	338 (46)	260 (52)	636 (54)	453 (57)	974 (51)	713 (55)
West	41 (6)	8 (2)	116 (10)	57 (7)	157 (8)	65 (5)

^a Source: National Center for Educational Statistics (15).
^b Region is based on US Census categorization (16).

 

Table 3. Comparison of Baseline and Follow-Up Scores for Inventory Content Areas in Schools That Completed a Baseline and Follow-Up Inventory (n = 1,295), Healthy Schools Program, 2007-2010

Content Area	No. of Items	Baseline, Mean (SD)a	Follow-up, Mean (SD)a	Cohen’s d Effect Size	Percentage That Improvedb
Total	83	40.4 (10.6)	47.9 (11.5)	0.83	79
School employee wellness	12	4.1 (1.9)	5.5 (2.2)	0.63	58
School meals	17	11.1 (3.1)	12.6 (2.9)	0.60	56
Health education	10	4.3 (2.5)	5.3 (2.4)	0.50	49
Competitive foods and beverages	12	3.9 (2.8)	5.1 (2.9)	0.48	48
Physical education	14	7.2 (2.3)	7.9 (2.0)	0.42	44
Physical activity outside of physical education	6	2.7 (1.5)	3.2 (1.5)	0.49	41
Policy and systems	6	4.2 (1.4)	4.7 (1.2)	0.51	39
Before and after school programs	6	2.9 (1.9)	3.6 (1.9)	0.41	37

Abbreviation: SD, standard deviation.
^a Baseline and follow-up scores are sums of the desired responses.
^b Percentage of schools that improved on 1 or more items. All gains were significant (P < .001).

Top of Page

Appendix. Healthy Schools Inventory

Items Common to 2007-2008 and 2008-2009 Inventories

This appendix is available for download as a Microsoft Word file [DOC - 203k]

Top of Page

---

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions.