Review the evidence and collect details

Once you have a thorough understanding of the problem, data, and actors, you will need to conduct your own research on available evidence and learn about the landscape of policy options. Your goal will be to develop an accessible list of information about policies that have helped solve similar problems in other places. This chapter will guide you through the steps of preparing your reviewers, understanding levels of evidence, and collecting the details on evidence that you will need for discussion with decision makers in your city.

Select and prepare your reviewers

Assign a person or team to review relevant interventions that have demonstrated success in other jurisdictions. Their goal will be to produce a review of all of the relevant programs, policies, or initiatives that your team could consider implementing. They will be conducting a literature review, supplemented by interviews with experts and peers who have information about potential programs.

Since the people you assign to this task are providing the baseline of potential options to implement, it is important that you choose people you are confident will be thoughtful and thorough in their work. It is also critical that they are aware of the importance of source credibility. Ensure your reviewers have familiarized themselves with rules for critically analyzing information sources before they get started. If they are including news articles as part of their search, they should also be aware of methods for evaluating news credibility.

Ask your reviewers to develop a plan and timeline to complete the review. This starts with solidifying a governing research question. One way to know you are creating a detailed research question is to use the acronym "PICO," reminding you to include reference to the population, intervention (if you have one in mind), comparison, and outcome elements in your research question. For example, if you know that you have a population of teenagers who are homeless, you have learned from your stakeholders that many of these teenagers identify as LGBT, and you have heard something about the approach of creating drop-in centers for delivering support, you could ask:

For LGBT youth facing homelessness (population), how effective are drop-in centers (intervention) relative to standard social services (comparison) at settling them in permanent housing (outcome)?

If you do not have an intervention in mind, you can leave out the intervention and comparison, and simply ask:

How can we help LGBT youth facing homelessness (population) obtain permanent housing (outcome)?

Once your reviewers have their research questions, use the questions to create a list of keywords they can use in their literature search. This will help keep their collection of articles reasonably focused. Have them describe what their search process will be, including listing the databases they will review and their plan for storing the information they will collect. (Here are some additional suggestions for structuring a review plan.)

What your reviewers will be doing is a lightweight version of a rapid review. To the extent that they would like to develop a methodology for specific search protocols, including critical appraisal processes, they can certainly go ahead and do so.

Use different levels of evidence for the most complete picture

As discussed in the "Working with a range of evidence" chapter, the Nesta Standards of Evidence framework -- reproduced below -- is helpful to understand the level of confidence that a program or intervention is having a positive impact. This approach lets you put interventions that you have heard about locally, but which have not been studied, together with those with a demonstrated causal effect. This is helpful since a number of policies, programs, and practices that you will want to consider may not yet have much of a research record.

Nearly any piece of information about an intervention can be classified within the standards of evidence. For example, if someone has a credible hypothesis about how an intervention could produce benefit, this type of evidence would be classified as Level 1. This type of evidence could include interventions in other cities that you read about in the news or hear about at a conference. For instance, when USDA allowed schools and school districts to provide free breakfast and lunch to all students through the Community Eligibility Provision (CEP), many districts implemented this option even before there was evidence of effectiveness. This would fit into Level 1, since it builds on the logic that providing access to meals will decrease food insecurity.

You may also find pieces of information that qualify as Level 2 evidence. For instance, as more school districts implemented CEP, principals testified about positive changes they observed in students, such as a more positive demeanor that can lead to better academic performance. Data also showed that the students described were receiving more meals than before the program's implementation. However, it was too soon to build evidence to show a causal relationship between the increase in free meals and increased academic performance.

“Lower-level” evidence can consist of nuanced, but valuable, elements that relate to an intervention’s potential fit with a community. Qualitative information such as testimony from people who have experienced an intervention and those who have facilitated or implemented it, will always be useful to consider and provides detailed, local context.

Experimental studies -- described starting at Level 3, with controlled experiments, and going through Level 5 with fully fleshed out protocols and processes for replication -- still occupy the higher levels of this description of evidence. This prioritization shows the value of rigorous study for evaluating the impact of a particular policy or program, rather than other factors that could be influencing outcomes of interest. Remember that multiple types of evidence can complement rigorous impact evaluations and give city leaders additional insights and broader perspective.

Access information

Encourage your reviewers to begin their search for promising practices in government and non-profit clearinghouses and in academic policy journals. Work published in these sources has already been subjected to a high level of professional scrutiny and generally contains a great deal of detail, which means that your reviewers are more likely to be able to pull out the information you need to make a good decision.

Clearinghouses:

Here are some of the "clearinghouses" (collections of program evaluations) for evidence-based policies and programs:

• Results First Clearinghouse Database, The Pew Charitable Trusts.
  Collects together eight other clearinghouses to provide information on interventions in the following categories: adult criminal justice, education, child welfare, health, juvenile justice, mental health, substance abuse, and youth development. Individual clearinghouses included in Results First are:

  • What Works Clearinghouse - for education interventions

  • California Evidence-Based Clearinghouse - for child welfare interventions

  • Crime Solutions - for crime prevention interventions

  • Blueprints for Healthy Youth Development - for youth development and violence prevention

  • Social Programs That Work - for social services programs

  • National Registry of Evidence-based Programs and Practices - for substance abuse and mental health programs (inactive)

  • Promising Practices Network - improving outcomes for children (inactive)

  • What Works in Reentry Clearinghouse - programs supporting people returning from incarceration

• UK What Works Centres.
  The British government has chartered a network of subject-specific organizations which collect, synthesize, and report out evidence on the effectiveness of policy, programs, and practices.

  • National Institute for Health and Care Excellence (NICE) - health and social care

  • Sutton Trust/Educational Endowment Foundation - educational achievement

  • College of Policing What Works Centre for Crime Reduction - crime reduction

  • Early Intervention Foundation - early intervention

  • What Works Centre for Local Economic Growth - local economic growth

  • Centre for Ageing Better - quality of life for older people

  • What Works Centre for Wellbeing - wellbeing

• Wilson Sheehan Lab for Economic Opportunities (LEO), University of Notre Dame
  Provides links to their evaluations of interventions in criminal justice, self-sufficiency, education, health, and housing and homelessness.

• Clearinghouse for Labor Evaluation and Research, US Department of Labor
  Provides research on workforce and labor-related policies and programs.

• J-PAL Evaluations, Abdul Latif Jameel Poverty Action Lab.
  Studies on poverty-reducing interventions from around the world.

• Campbell Systematic Reviews, Campbell Collaboration
  Systematic review of policy interventions by scholarly "coordinating groups" in seven issue areas: crime and justice, education, international development, methods, social welfare, knowledge translation and implementation, and disability.

• Benefit-Cost Results, Washington State Institute for Public Policy.
  A Washington State-chartered program to establish the likelihood of relative costs and benefits for programs across the following areas: juvenile justice, adult criminal justice, child welfare, pre-K to 12 education, children's mental health, health care, substance use disorders, adult mental health, public health and Prevention, workforce development, and higher education.

• Effective Interventions, Centers for Disease Control and Prevention. Repository for evidence-informed HIV prevention/care interventions (with steps on how to implement)

Journals and indexes:

Once you have reviewed relevant information accessible through clearinghouses, it is worth taking a look at individual studies published as scholarly articles. Your reviewers will do this by searching article indexes and academic journals. Unfortunately, a number of useful academic journals and indexing platforms may not be generally accessible to your reviewers if you do not have a subscription to them. To address this, consider partnering with a local college or university to provide you with a list of articles from the following sources. (This is also an excellent opportunity to seek research support from any academic stakeholders you identified previously.)

Indexes:

Google Scholar "includes most peer-reviewed online academic journals and books, conference papers, theses and dissertations, preprints, abstracts, technical reports, and other scholarly literature, including court opinions and patents." It allows you to specify a publication range in your search and provides links to likely sources for all listed articles.

Policy File Index "is a unique resource for U.S. public policy research in that it grants users access to timely, updated information from over 350 public policy think tanks, nongovernmental organizations, research institutes, university centers, advocacy groups, and other entities."

Policy Archive works "to collect and disseminate summaries and full texts, videos, reports, briefs, and multimedia material of think tank, university, government, and foundation-funded policy research."

Another source for high-quality research is the reports of the Congressional Research Service. You can find their reports aggregated here.

Specific journals:

Often, the most useful academic journals for unearthing policy or program reviews will be in each specific subject area -- that is, in education journals for education policies. You can use a general index like Google Scholar to begin to learn which journals are most often useful for your search. However, you can also search the following two generalist policy publications for issues and articles related to your topic:

• Public Administration Review
• Journal of Policy Analysis and Management

News sources, subject matter experts, and interviews:

Although it is valuable to know what has been explored through a scholarly research framework, it is also useful to learn what kind of interventions are being tried in the field right now and how people feel about these trials. Search subscription news aggregators like LexisNexis or free news aggregators like Google News or Yahoo News for your issue area plus the word "policy" or "program" to see what kinds of interventions are being covered in the news.

Do not forget to contact the advisory group members who were chosen for their subject matter expertise. In addition to providing good material through interviews themselves, they may have ideas about other subject matter experts you should contact to ask about potential interventions.

Peer cities and networks:

In addition to reading about the work of other cities, it is ideal if you are able to speak with people who are involved with current implementations of interesting interventions. In your city, you likely have relationships with your neighbors, but it can be helpful to look outside of your region for solutions as well. To find counterparts in other cities who are working on a program you would like to learn more about, you can contact staff in intermediary organizations (like GovEx or your professional associations) who might be in a position to put you in touch immediately or who might see another good route for setting up a conversation. Here is a list of networks that may be helpful:

• International City/County Management Association (ICMA)
• National Association of Counties (NACO)
• National League of Cities
• US Conference of Mayors
• What Works Cities

Collect the details

Once your reviewers have found evidence on possible policies or programs to implement, they should capture key details that will help decision makers understand the potential effectiveness and applicability of each potential intervention.

For each source that has provided information you want to convey, use the template below to capture as many of the following details as possible. (You do not need to complete this for sources that provide duplicative information or are otherwise not that useful.)