The country is filled with programs that aim to reduce childhood obesity. As we have often pointed out, however, most of these programs are doomed to have little impact, even if they work, because they don’t incorporate any evaluation tools, and so it very unlikely they will get the financial support needed to roll out across the country.
The Wall Street Journal Health Blog has caught on to the same issue but profiles one program that is doing some self-evaluation. The piece is titled Progress From Portland on Improving Kids’ Healthy Behaviors:
…while many cities and towns have programs and events to attack childhood obesity, few really know whether they’re making a difference.
One program that has been tracking its impact reports new signs of progress. Earlier this year, the WSJ profiled the Let’s Go! program in Portland, Maine, which has rallied schools and other sites in the state to get more families to adopt its daily 5-2-1-0 message: every day, eat at least five servings of fruit and vegetables, get two hours or less of screen time, get at least one hour of exercise and consume zero sugary drinks.
New data from an independent telephone survey of about 800 parents in the greater Portland area show some steady improvements in healthy behavior. In 2011, 31% of children had adopted at least three of the 5-2-1-0 behaviors, compared with 22% in 2007.
Specifically, in 2011, 26% of parents reported that their children ate the recommended number of servings of fruits and vegetables, compared with 16% in 2007. In addition, 45% of parents said their child’s daily recreational screen time was 2 hours or less, compared with 38% in 2007.
Unfortunately, it is not enough to simply do an evaluation. It is essential that the evaluation be meaningful. This one, like so many others, throws off more smoke than light. To note, just two of the more obvious issues:
-
Surveying people about their own virtuous behavior encourages people to lie.
All surveys have problems as people can make errors and their perceptions may not be accurate. Still, it may be possible to get honest answers if the question is neutral: Does your family prefer vanilla or chocolate ice cream? In this case, though, the surveyors might as well call up the parents and ask: Are you a good parent? The 5-2-1-0 program is not about neutral behavior. The whole point, promoted incessantly, is that these are virtuous behaviors that good parents ought to promote and require. That people say they do so is not really surprising and is quite possibly not accurate. The difference in the 2007 and 2011 survey results may just testify to the impact of four years of promotion of such behavior has on parental guilt. -
Behavioral change doesn’t matter unless it leads to an outcome difference.
Potential funders for national rollouts of these types of programs don’t actually care about these specific behaviors. If kids spend more than two hours a day in front of computer screens because they are preparing their entries for the Siemens or Intel National Science competitions and still make adequate time for physical fitness, nobody will care that they fail this part of the 5-2-1-0. Even an issue such as increasing produce consumption is, beyond the produce industry, only likely to be of interest if it results in healthier children.
This type of evaluation almost seems designed to avoid the kind of real-life testing that matters. What you want to do, of course, is weigh and measure students who are going to be exposed to the program before they start the program, then reweigh and measure after a fixed interval. You want to do the same thing with students who are demographically similar but are not being exposed to the program.
How often one measures depends on what one wants to know. You can measure the effect this school year, or after five years of exposure to the program, or what the impact of a program done in elementary school and then stopped in 5th grade is on various health parameters when the students graduate middle school or high school. You can study if exposure to such a program in grade school affects health status at age 20, etc., etc.
It is hard to know which is scarier. That decisions on what programs to create or extend are being based on such empty research or that even with powerful incentives to exaggerate, the parents still report that only 26% of their children eat at least 5 servings of fruits and vegetables (only 5!) each day.
