This 39-page guidance note by the Overseas Development Institute (ODI) is about designing a monitoring and evaluation (M&E) framework for policy research projects. It explains what aspects of policy research projects you should monitor and evaluate, why, when and how. It offers plenty of practical examples, explainers and an overview of key questions and methods.

The guidance note starts with a foundation chapter which introduces a theory of change approach, identifies the purpose of evaluation and helps you identify knowledge roles and functions for your partners and personnel. It then guides you through the development of a framework, which is built around six key areas. The guidebook recommends following a three-step structure in preparation for the evaluation of each key area:

1) clarifying the purpose, intensity and timing; 

2) defining key questions; 

3) identifying appropriate approaches, methods and indicators. 

Let’s now explore the six key areas in more detail:

1. Strategy and direction: ‘Are we doing the right thing?’ 

The first area involves monitoring and evaluating whether your project is being strategic and if it is leading to the desired goals. It means revisiting your strategies, reviewing the theory of change, analysing potential unwanted consequences and deciding whether your strategy needs modifying. 

The guide provides you with examples of key questions to ask, for instance:

• Is the project’s theory of change appropriate, logical and credible? 
• Are the right stakeholders being engaged? 

After you choose the M&E questions, use them to guide the selection of appropriate and feasible approaches, methods and indicators. In this case, they are usually qualitative rather than quantitative and revolve around reviewing reports and other key documents.

2. Management and governance: ‘Are we implementing the plan as effectively as possible?’ 

This area studies how the research project is managed, if it provides value for money, how decisions are formed and what the capacity and performance of partners are. Your example questions could be:

• Is the work plan realistic in terms of timing, staffing and resources?
• How decisions are made, with what criteria and how are they documented? 

You will most likely use qualitative methods, such as reviewing agendas and minutes of internal meetings.

3. Outputs: ‘Are outputs audience-appropriate and do they meet the required standards?’ 

Outputs are the goods and services that your project produces. These can be reports, articles, policy briefs, websites, blogs, workshops, events and so on. Outputs are generally the main focus of research projects’ M&Es because they are visible and tangible. The extent of the evaluation will depend on your resources, but, in general, it is a good idea to focus on the quality, relevance, credibility, accessibility and quantity of your outputs. 

Example questions include:

• What outputs have been produced? 
• To what extent are the outputs being delivered in a way that represents value for money?

Approaches, methods and indicators for assessing outputs are usually straightforward and quick to gather (for example, a list of the type, number, quality and relevance of outputs produced, a collection of web statistics or after-action reviews). 

4. Uptake: ‘Are people accessing and sharing our work?’ 

You can start monitoring and evaluating uptake once outputs have been delivered to your audience. You should look at what outputs are picked up and used and what are the responses to them.

You can use the following example questions:

• What outputs have been used by stakeholders and how?
• How can uptake be improved and strengthened? 

Most commonly, you will assess uptake by asking for direct feedback, collecting web and social media statistics and looking at event attendance lists and feedback. 

5. Outcomes and impacts: ‘What kinds of effects or changes have the work contributed to?’

In this area, you study the long-term changes in behaviours, policies, capacities or practices that your research has contributed to. Evaluating outcomes and impacts is usually time-consuming and the scope and extent will depend on your budget.

Here are a few examples of key questions to ask:

• To what extent has research influenced policy?  
• To what extent has research shifted public agendas – what gets discussed and how it is framed?

Common approaches to measuring impact include capacity assessments, stakeholder interviews, outcome mapping, stories of change and contribution analysis.

6. Context: ‘How does the changing political, economic, social and organisational climate affect our plans and intended outcomes?’

Lastly, it is important to monitor and evaluate the political, economic and organisational context and changes happening within it. It can help you frame your research and communicate it better, explain why uptake is happening or discover why you are seeing (or not seeing) the outcomes you expected. 

Typical M&E questions in this area are:

• What political, economic or organisational changes are taking place? How do the changes affect the research plans and activities? 
• What influences decision-making? 

Methods for studying context tend to be qualitative and require in-depth analysis.

To sum up, this guidance note is a great resource for understanding the development of a monitoring and evaluation framework for policy research. Many choices about the scope, intensity and timing of the six M&E areas will be individual and dependent on your resources as well as your capacity, experience and skills. The authors of the document recommend staying realistic and practical about what can be done, rather than trying to do everything. 

Apart from the detailed and practical description of the six areas for M&E, the resource also features a list of further reading and an annex with an overview of the key questions, typical approaches and example indicators for the M&E areas. 

This article is part of our initiative, R2A Impact Practitioners. To find out more, please click here.