Design implementation and evaluation
Design the implementation
Now that you have your accountability plan and key indicators, it is time to get specific about your plan for implementation. The template below will help you think through the specific mechanisms of implementing policy change.
Understanding organizations and systems is critical to understanding and making decisions about where and how interventions should be applied. A policy implemented in a single setting may show promising results, but implementing multiple policies and programs at the same time with different actors responsible for each aspect, can help ensure sustainable behavior change.
In Somerville, MA, Shape Up Somerville (SUS) is a citywide public health program that is implemented across multiple levels of influence and systems through interventions before, during, and after school to include changes to built environment such as bike lanes, regional mass transportation, and physical education equipment in schools and gyms to increase physical activity, wellness policies and school gardens, nutrition education during and after school, and healthy restaurant programs to support healthy food and healthy eating habits.
Use the template below to plan your implementation.
Policy implementation planning template:
List the specific desired outcomes: In the previous step, you worked with department staff to establish specific, measurable outcomes and interim indicators. Prioritize the indicators that will demonstrate progress. Use the space below to capture the desired outcomes of the policy change.
Example: Shape up Somerville began with the goal of preventing obesity in 1st through 3rd graders.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Specify the target population: Are you designing an intervention that targets only certain sites? Are you planning on implementing a program or initiative for a specific target audience? Clearly specify the population you plan to target.
If you plan to conduct a rigorous impact evaluation, as noted by the Wilson Sheehan Lab for Economic Opportunities, it is important to establish concrete, objective eligibility criteria and ensure that these criteria are applied consistently by all staff during screening and intake. It is also vital to identify the individuals who are eligible for a particular program or service, even if everyone eligible cannot be served by existing resources. This helps cities identify any unmet needs and guide future resource allocation, while also creating the possibility of a rigorous impact evaluation by identifying a potential comparison group. Finally, when conducting screening and intake, it is imperative to collect and maintain basic information (e.g., name, date of birth, gender) about all program-eligible individuals, not just those who participate in the program.
Example: In the Shape Up Somerville example, the program began by targeting 1st through 3rd graders.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Define the location: Where will the policy change take place? Will implementation take place at one site or multiple sites?
Example: Shape Up Somerville interventions were delivered in schools.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Clarify implementation partners and articulate their roles: What are the key organizations, institutions, or people responsible for implementation? The project team should be as clear and specific as possible in providing a description of the concrete activities that implementation partners are meant to carry out. Practice profiles may be a helpful tool to help define what is required of the actors who are responsible for implementing the intervention.
Example: In Somerville, mayoral leadership and engagement of multiple city agencies (public health, primary care, schools, Cambridge Health Alliance, etc.) streamlined and connected the various Shape Up Somerville initiatives and grants, which helped to keep the talent and energy focused on healthy eating and active living for all sectors of Somerville. SUS partners ultimately included schools, nonprofits, City and state agencies, and national foundations.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Describe the intervention: What are the steps, activities, or processes? Remember that the mechanism, the specific steps or process that lead to behavior change, is key to understanding the effectiveness of the policy change and sharing the results with others.
Example: Goals of Shape Up Somerville were to influence every part of an early elementary schoolchild’s day – before, during, and after- school to promote healthy eating, physical activity, and their weight status. Intervention included: food service enhancements to improve breakfast and lunch and staff professional development; walk to school activities; SUS classroom curriculum and professional development; school wellness policy development; SUS after-school curriculum and professional development; outreach and education to home through materials, forums, events; “SUS approved” restaurants; community outreach and capacity building through policy development, trainings, media placements.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Clarify timeline and milestones for delivery and evaluation: How frequently will the services/intervention be delivered? How long will the program run? What is the time to expected impacts (e.g., 12 months after program completion)?
Example: Shape Up Somerville, led by Tufts University Friedman School of Nutrition Science and Policy, conducted a quasi-experimental (nonrandomized) intervention study among 1st through 3rd grade students in Somerville (intervention) and two comparison communities for 9-months during one school year (fall 2002 – spring 2003). The study showed positive association of the intervention on change in BMI. SUS released reports summarizing evaluations, progress to date, plans for additional projects, and impacts ten and fifteen years into the initiative.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Plan your analysis: The previous section addressed establishing an accountability framework to monitor progress and implementation. Determine which indicators will be discuss through that process, who will share the data and analysis, and how feedback from those conversations will be incorporated by the project team.
Example: The Mayor’s office implemented a data-driven decision making approach, called SomerStat, which was synergistic with the approaches taken by the research and evaluation teams deployed for the specific grants.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Plan for sustainability: If the policy and program change is successful, how will the project team sustain and expand to additional sites or populations?
Example: Although most of the partnerships forged in the early years were maintained, new grants and collaborations expanded attention to older public school students, active transportation and the Green Line subway extension, the built environment, farmer’s market efforts such as “Grown in Somerville” with Union Main Street, economic development, and health disparities, as well as data collection, monitoring and surveillance.
Your turn:
____________________________________________________________________________
____________________________________________________________________________
Pilot planning template
If you plan to conduct a pilot, first determine the purpose of the pilot. Do you want to observe whether the policy change results in the desired behavior change? Are you concerned about feasibility, scalability, or cost? Do you want to test whether your desired results can be achieved through this policy approach? Clarify the goal of the pilot and the questions you hope to answer. This will help you identify the key indicators that you will use to measure whether the pilot is successful. Before beginning, establish a baseline of current performance and set your targets accordingly based on what you hope to accomplish through the pilot.
Next, establish the pilot plan for your project team, including milestones for team check ins to determine whether to move ahead or change course. This is separate from the meetings that will occur as part of the accountability framework. Use the space below to create your timeline and develop your meeting schedule.
| Task | Person responsible | Start date | Check in date | Deliverable |
|---|---|---|---|---|
Plan your evaluation
While you are designing the implementation, it is also time to begin to develop your plan for analysis. As noted by the Wilson Sheehan Lab for Economic Opportunities, decisions about program set up - including outreach, screening, intake, data tracking, and enrollment - can determine whether or not a rigorous impact evaluation will be possible. As discussed, city leaders may find it helpful to include evaluation partners in early conversations about program set up. During previous stages, you may have worked with a researcher to plan your proposed policy change. You should be clear about the purpose of the evaluation and develop an evaluation plan. Below are additional considerations.
Causation vs. trend analysis
In the "Establish a framework to monitor progress" stage, there was guidance on tracking trends over time. It is important to note here that monitoring data is different from analyzing a causal effect. Trend analysis will illustrate that changes are occurring and may lead to questions that could be answered by your evaluation, but trend analysis alone does not provide evidence about what is driving those changes. You may see changes in your outcome indicators based on variables outside of the City's control. For instance, a change in federal trade policy can have an impact employment rates in your City outside of your City's new job training and placement program.
To address causality, you must compare the outcomes of individuals who received the intervention with those who did not. Without a comparison group of similar individuals who were not exposed to the intervention, it is impossible to know which factors are responsible for the outcomes. Randomized Control Trials (RCTs), in which individuals are randomly assigned to "treatment" and "control" groups, are the gold standard in proving causality. If you can demonstrate that outcomes improved for those who received the intervention and did not improve at all or improved less for a comparison group, you can be somewhat certain that the intervention was the cause.
Examples of other, non-causal impact evaluations that are rigorous and reliable include certain quasi-experimental studies using comparison group designs, smart interrupted time series analyses that model trends over time in ways that allow estimation of intervention effects, well designed pre-post test analyses, needs assessments, implementation studies; etc. For more information on different types of evaluations, see the Sage Research Methods website.
Look for differences in groups
Build in time to also examine the pilot data to see if groups are experiencing different results. Reviewing performance based on outcome measures for a population in aggregate may hide the subtle ways that some people may be affected differently, so it is necessary to carefully analyze the data for disparate outcomes. For instance, the Moving to Opportunity (MTO) study that examined the impact of housing vouchers on families who lived in high-poverty neighborhoods found that children who moved before the age of 13 had better education and higher incomes as adults. However, adults did not see improved education and employment outcomes.
Disaggregating data into more specific categories of race and ethnicity can help understand trends and disparities in a more granular way, which can help decision makers target policies and interventions in a more equitable way.
Earlier in the Roadmap, we worked to identify stakeholders and who might be positively and negatively impacted by a change in policy. Use the results of that exercise to involve the community in interpreting data analysis and to gather qualitative feedback from the community to call attention to issues. For instance, an increase in youth detention may be explained by inefficiencies or lack of access to transportation.