Monitoring and evaluation

Three questions that guide how to assess your policy impact

By 3 March 2016

by Julia Coffman

For Think Tanks and others acting to influence public policy, assessing one’s impact in a complex and crowded political environment can be challenging. When trying to understand if one’s policy work is making a difference, we recommend answering three key questions.

Question 1: Who are your audiences?

Influencing efforts ultimately is about communicating effectively to individuals or groups so that they learn, think about, or do something differently regarding a policy issue. While the research and content that Think Tanks deliver is a critical element of the influence process, “so is engaging with journalists, advocates, policymakers, and others who might interpret and use the findings”, according to Ruth Levine.

Audiences are the groups and individuals that Think Tanks or other influencers target and attempt to affect or persuade. They represent the main actors in the policy process and fall into three categories:

  1. The public (or specific segments of it)
  2. Policy influencers (e.g., media, community leaders, the business community, thought leaders, political advisors, other advocacy organizations, etc.)
  3. Decision makers (e.g., elected officials, administrators, judges, etc.).

Influence strategies may focus on just one audience or target more than one simultaneously.

Within these categories, it is important to identify specifically who is being targeted. “Those who aspire to inform decision making with research and evidence should avoid talking generically about ‘policy makers’”, for example. While there are three broad categories of audiences that might be engaged, who is being targeted within those categories?

Think Tanks with theories of change and related communication strategies will already have clear answers to the audience question.

Example Audiences


Question 2: How do you want audiences to change?
Changes are the outcomes in audiences that Think Tanks and other influencers aim for in order to progress toward a policy goal. There are three broad categories of changes, and they fall along a continuum based on how much an audience is expected to engage on a policy issue in order to achieve the influence that Think Tanks are after.The continuum starts with basic awareness or knowledge. Here the goal is to make the audience aware that a problem or potential policy solution exists. The next point is will. The goal here is to raise an audience’s willingness to take action on an issue. It goes beyond awareness and tries to convince the audience that the issue is important enough to warrant action, and that any actions taken will in fact make a difference. The third point is action. Here, influence efforts actually support or facilitate audience action on an issue. Again, influence strategies may pursue one change with an audience or more than one simultaneously.
Example Audience Changes

Keep in mind that in order for policy change to occur, somebody ultimately needs to do something differently than they are doing right now. Influence efforts need to move somebody toward action. Decades of research have shown that just making people more aware of an issue or problem generally is not enough to mobilise them to act. Education by itself is not equivalent to motivation, and new knowledge does not automatically result in attitude or behaviour change.

Question 3: How will you capture audience changes?

Once the first two questions have been answered, assessments of policy impact can focus on measuring whether changes in the identified audience have occurred. Definitions of each outcome and the measures that might indicate whether those changes have occurred can help to operationalise audience change into measurable indicators.

Capturing audience changes can involve a familiar list of traditional data collection methods, such as surveys, interviews, focus groups, or polling. But because the influence process in a political environment can be complex, and audience outcomes can be hard to measure (e.g., public will or political will), innovative methods have been developed specifically for assessing policy influence efforts. Think Tanks may find bellwether interviews, champion tracking, and policymaker ratings particularly useful.

Finally, isolating an organisation’s impact is difficult in a complex policy context that involves multiple actors. To increase the chance that any audience changes detected can be plausibly linked back to Think Tank efforts, it is critical to be as specific and precise as possible when answering the first two questions and to be sure that they link closely to the communications strategies that Think Tanks are using.

 

This article was originally published in the Aditi bulletin, an online publication, compiled by the Centre for Study of Science, Technology and Policy (CSTEP), to connect with Think Tanks in the South Asian region. While the Think Tank Initiative conducts annual regional workshops to facilitate discussion between Think Tanks, many organisations felt that more avenues/platforms were required to facilitate effective knowledge exchange. CSTEP produced the bulletin in order to serve that need; to share articles, experiences, interviews and books written by and for Think Tankers.

The bulletin has recently released its fourth issue. Each issue centres upon a theme. Previous subjects include: Domestic funding and philanthropy; Importance of Communication; Engaging with Policy Makers; Policy Impact.

The next issue is due in mid-May 2016 and CSTEP are inviting young researchers and individuals who are well versed in their respective domains to submit contributions. A French version of Aditi has also been published recently (the issue focussing on Policy Impact) and there are plans for Spanish versions of forthcoming issues. Therefore submissions are welcome in all three languages.