For many DFID-funded RPCs (Research Programme Consortia*) the concept of Theories of Change (ToC) can appear to be the elephant in the room when conceptualising new (or even revising existing) research programmes. As Duncan Green has pointed out, ToC has become the new “fuzzword” that everyone’ s bandying about, but when you ask people what is a ToC or what theirs looks like, there tends to be an awkward silence.
This, I suggested at today’s ToC workshop hosted by DFID and Research for Development, stems from confusion created by:
- An over reliance on, and confusion with, logframe modelling, which often does not follow a logical pathway (ironically)
- The tendency to first align ToC with M&E and assigning indicators, without first plotting clear pathways to change
- Lack of clear guidance and support in developing a ToC, specifically for research uptake and policy influence
DFID today acknowledged that the process which sees all new RPCs developing a ToC could be improved, and talked about moves underway to provide better guidance for DFID-funded programmes. This is good to hear, after all DFID have played an instrumental role in seeing ToC become the new “fuzzword” within the development sector and it’s important that this potentially valuable process is defined clearly, allowing the fuzz to be removed.
DFID is listening, so perhaps there is a real window of opportunity to ensure the ToC process is refined, not convoluted with the development of logframes, and guided by the development community. Based on the discussion yesterday and some of my thoughts over the last 12 months in thinking about ToC, I have suggested 10 things DFID and its future guidelines might want to consider to improve the process .
1. ToC needs to come before the logframe in the programme cycle. Currently DFID RPCs put together the logframe before they put together the ToC. This often results in an overly complex ToC because there is a natural tendency to use the same logic (or rather the illogic) as in the logframe. The result is that the focus on actors along the various pathways and the changes you want to bring about in them are often missed.
2. Make sure you have some evidence and contextual understanding of how change might happen before you begin your ToC. This ensures that those assumptions within the ToC can be better ‘unpacked’ and understood, and would ensure that when you actually produce your ToC it provides not just a hypothetical overview of change, but one which is strategically relevant (Duncan Green talked about the need to separate out the hypothetical from the strategic and this process would support this).
3. Provide better on-going training and support
4. DFID need to ask for annual revisions of ToC so that it is seen as an ‘emergent’ process, and more reflective of changing contexts, rather than being something that is written at the outset and seldom returned to.
5. Make sure RPCs focus more on the “and a miracle occurs” moment (familiar to most people through this cartoon) or in other words the interactions and activities that are required to see outputs and outcomes materialise and the assumptions underlying these. Simon Batchelor (IDS) spoke of making sure every step is “stress” tested as a means of reviewing causality, which is another useful way of thinking about this.
6. Ensure the initial focus is on building clear pathways of change, and ensure the M&E element follows as something leading on from this
7. Make sure the research process is included in the change pathways (a great deal of assumptions exist in relation to the direction and outputs associated with the research itself, and often these never actually emerge).
8. Engage researchers from the start by making sure ToC is not something done for researchers but something that includes them (along with every other member of the team)
9. Promote changes to the incentive structure for research uptake so that researchers can think more in terms of ‘impact’. This is a biggy! The incentive structure seems so engrained across academia, and to talk about changes solely in the context of development research appears untenable. Thankfully, Duncan Green encouraged us to think again, and suggested a number of incentives DFID could direct at DFID-funded research programmes. Here are two of a few ideas that emerged:
- Make ToC adaptable so that RPCs can take advantage of shocks and events happening in the real world to get their research into policy and practice. What’s more they should get credit for doing this.
- Have a league table for researchers leading on advocacy and engagement through various different communications tools (i.e. social media, blogging, podcasts, radio etc.)
10. Don’t be too prescriptive about what a ToC should look like. Each ToC could include a variety of different pathways (or theories), as such diagrams and narratives will often be very different. As long as these are “stress” tested and the indicators assigned to these justified, then this should be seen as appropriate. Creative thinking is what we need!
It would be great to hear people’s thoughts and experiences on developing a ToC for DFID-funded programmes, along with any ideas on how this process could be improved.
Duncan Green (Oxfam) also blogged about the workshop here: Can theories of change help researchers (or their funders) have more impact?
*DFID-funded research programmes
Paula Fray
Shubha Jayaram
Sue Martin
Maria Balarin
James Harvey
Emily Hayter
Susan Koshy
Ronald Munatsi
Ajoy Datta
Hi Andrew,
I think your post might be relevant, but I find it difficult to read because of the many abbreviations that are not explained. Can you help me out here?
best regards,
Femke Merkx
Good suggestions. For #4 rather than asking for annual revisions, perhaps allowing/encouraging it would strike the right balance between endless tinkering (which can divert time and effort from actually using the ToC) and excessive rigidity or out-of-sight-out-of-mind invisibility.
I’d also like to add #11 Ensure the diagram articulates both theories of how change occurs (eg deterrence; capacity building) and theories of how the activities will trigger these changes (eg publicly increasing the number of inspections to increase the likelihood of detecting non-compliace). This is particularly important when trying to make sense of the evidence across multiple projects, where projects might have the same theory about how change occurs (eg increasing incentives) but different theories about what activities will trigger these (eg cash payments, public recognition, chocolate) or sometimes similar activities (eg football) but different theories about subsequent change processes (eg increased physical fitness; increased social capital).
Patricia, thanks for your comments. I would be a little worried that just “encouraging” programmes to revise the ToC might fall flat unless this is incentivised in some way. Although, agree it’s about finding a balance and ensuring people are engaged in the process, which again is easier said than done!
On your second point, I think you are completely right! Having the activites there ensures a full and causal pathway is in place. However, it’s worth adding, ToC is a useful process that in feeds into the development of the logframe. How do we need to make sure that that the process of developing a ToC does not become too complex and resource dependent?
Patricia,
thanks for your comments. I would be a little worried that just
“encouraging” programmes to revise the ToC might fall flat unless this
is incentivised in some way. Although, agree it’s about finding a
balance and ensuring people are engaged in the process, which again is
easier said than done!
On your second point, I think you are completely right! Having the
activites there ensures a full and causal pathway is in place. However,
it’s worth adding, ToC is a useful process that in feeds into the
development of the logframe. How do we need to make sure that that the
process of developing a ToC does not become too complex and resource
dependent?
Hi Patricia, thanks for your comments. I would be a little worried that just “encouraging” programmes to revise the ToC might fall flat unless this is incentivised in some way. Although, agree it’s about finding a balance and ensuring people are engaged in the process, which again is easier said than done!
On your second point, I think you are completely right! Having the activites there ensures a full and causal pathway is in place. However, it’s worth adding, ToC is a useful process that in feeds into the development of the logframe. How do we need to make sure that that the process of developing a ToC does not become too complex and resource dependent?
I direct the DFID-funded RPC known as “STRIVE: Tackling the structural drivers of HIV” and we have had a lot of success using the TOC process. I think it can be an especially useful exercise when designing research projects — especially projects that attempt to develop and evaluate demonstration projects. If forces groups of participants to articulate how they think change will happen and to agree on what the evidence of change will be.
I think where people get tripped up is deciding at what “level” to develop their theory of change — especially when working within a large consortium. Are we developing a theory of getting research into practice? How the overall RPC will affect a range of outcomes? What the logic is for a particular small intervention.
The harder question I think is how to reconcile the Theory of Change approach with the DFID log frame, which emerges from a different logic and approach to project development. I’d be happy to share our final TOC — which includes our strategy and assumptions regarding when and how research inflruences practice, but is actually designed to address the multiple impacts we hope the consortium will achieve.
Hi Lori,
This insight is really interesting and it’s great to hear that the ToC process
was realy useful for your programmme. It would be good to feature different ToC
on Research to Action and hear how different prorammes have gone about the
process. Perhaps you might consider a guest post that features your ToC and
expands (if only a little) what is contained in your comment?
Reconciling the ToC with the DFID logframe is a difficult task, and I would
much prefer it if these were seen as distinct tools for managing slightly
different things. Having said that it would make life muh easier if they did fit
together a little more succinctly.
Hope we hear some more from you!
Hi
Andrew, Patricia, Lori and others,
This
looks like a really useful and timely discussion. Some very good suggestions
here, which chime with my own experience in facilitating the
development of theories of change for programmes and evaluation capacity work.
I have
also recently completed a review for DFID on how theory of change is being used
in international development – by donors, civil society organisations and
programmes, including research programmes. The report and resource
documents are now available on R4D (links at the end of the post).
One of
the main messages to come from the review is that theory of change is
challenging to work with because it requires a commitment to take a reflective,
critical and honest approach to answer difficult questions about how our
efforts might influence change, given the political realities, uncertainties
and complexities that surround all development initiatives, including development
research.
At the same time, by encouraging on-going questioning of
what might influence change in the context and drawing on evidence and learning
during implementation, theory of change thinking can be incredibly inspiring,
as Lori suggests.
Improving the fit between programme, context and stakeholders, moving beyond technocratic responses
towards more realistic and feasible interventions that are responsive to
dynamic contexts are some of the things people said.
How to keep this reflection and dialogue going, especially between
donors, implementing agencies and stakeholders, given the demands of log-frames, M&E, performance
management and institutional incentives was at the heart of discussions
at a DFID workshop in May 2012 to discuss the draft report with contributors
and others (see Duncan Green’s commentary http://www.oxfamblogs.org/fp2p/?p=10071).
Although there are no easy answers, the review report and its supporting resource documents – although hefty – presents and discusses people’s
rich experience in tackling some of the key conceptual, practical and
institutional challenges, including some of the issues people have flagged
here. There are examples and practical suggestions, and more in-depth Box Examples to illustrate people’s experience, from donors to
implementing agencies and projects.
The appendices include a collection of links to resources and guides
to working with theory of change and annotated examples of a range of theories
of change developed for different purposes, including for research programmes.
The review has informed DFID’s own approach to working with theory
of change. The report includes extracts from some of DFID’s new draft internal guidance
to working with theory of change throughout the project cycle (business case and beyond).
DFID’s draft guidance reflects a lot of the learning about:
– the importance of keeping theory of change an open, on-going reflection and discussion rather
than a prescriptive methodology
– using theory of change thinking alongside the log-fame instead of falling into the trap of making it a ‘log-frame on steroids’
– the key points about reviewing the theory of change at least annually to keep the thinking going throughout.
Looking forward to lots of discussion on this!
Isabel
LINKS:
Summary overview: http://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/Theory_of_Change_Review_Report_Summary.pdf
Full Report (with Appendices 1-2) – a big file: http://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/DFID_ToC_Review_VogelV7.pdf
Appendix 3 – annotated examples – also a big file: http://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/Appendix_3_ToC_Examples.pdf
these are really constructive insights for improving the use of ToC’s. I also support Patricia’s comments. I’m new to ‘Development’ and have been astounded at the institutionalisation of logframes and how hard it can be to make sense of them, but I have been working with ToC’s and Contribution Analysis for awhile and I’d like to add a couple of suggestions for improvement. For #4 to encourage DFID to greater appreciate the dynamics of change and for programme deliverers/funded agencies to become more reflective, build in learning loops at times which both complements internal/external reporting up/out to better balance accountability demands with learning. Taking Patricia’s point – for most organisations doing this annually would make sense. For #5 between outputs and outcomes include ‘Reach’, that is, who is the ‘target group’/intended beneficiaries for your efforts/action, to assist with unpacking assumptions. Also for dealing with the ‘magic happens/leaps of faith’ that occur in logic models, log frames etc, using spheres of control/influence may help in thinking through complicated/complex change and Lori’s challenge of where to locate the ToC. I have used this with public sector managers in the context of partnerships and I have had very positive reactions and usually big sighs of relief. I am happy to provide more information on this if you are interested.
My final comment comes from my experience in facilitating evaluative capability building. When people are coming together to build/create ToCs, think about how it is facilitated in order to build a shared understanding of the issues/challenges/strengths/evidence/aspirations and the pathways for change and using the opportunity to strengthen group relationships, whether that is with teams, organisations or partnerships.
these are really constructive insights for improving the use of ToC’s. I also support Patricia’s comments. I’m new to ‘Development’ and have been astounded at the institutionalisation of logframes and how hard it can be to make sense of them, but I have been working with ToC’s and Contribution Analysis for awhile and I’d like to add a couple of suggestions for improvement. For #4 to encourage DFID to greater appreciate the dynamics of change and for programme deliverers/funded agencies to become more reflective, build in learning loops at times which both complements internal/external reporting up/out to better balance accountability demands with learning. Taking Patricia’s point – for most organisations doing this annually would make sense. For #5 between outputs and outcomes include ‘Reach’, that is, who is the ‘target group’/intended beneficiaries for your efforts/action, to assist with unpacking assumptions. Also for dealing with the ‘magic happens/leaps of faith’ that occur in logic models, log frames etc, using spheres of control/influence may help in thinking through complicated/complex change and Lori’s challenge of where to locate the ToC. I have used this with public sector managers in the context of partnerships and I have had very positive reactions and usually big sighs of relief. I am happy to provide more information on this if you are interested.
My final comment comes from my experience in facilitating evaluative capability building. When people are coming together to build/create ToCs, think about how it is facilitated in order to build a shared understanding of the issues/challenges/strengths/evidence/aspirations and the pathways for change and using the opportunity to strengthen group relationships, whether that is with teams, organisations or partnerships.
these are really constructive insights for improving the use of ToC’s. I also support Patricia’s comments. I’m new to ‘Development’ and have been astounded at the institutionalisation of logframes and how hard it can be to make sense of them, but I have been working with ToC’s and Contribution Analysis for awhile and I’d like to add a couple of suggestions for improvement. For #4 to encourage DFID to greater appreciate the dynamics of change and for programme deliverers/funded agencies to become more reflective, build in learning loops at times which both complements internal/external reporting up/out to better balance accountability demands with learning. Taking Patricia’s point – for most organisations doing this annually would make sense. For #5 between outputs and outcomes include ‘Reach’, that is, who is the ‘target group’/intended beneficiaries for your efforts/action, to assist with unpacking assumptions. Also for dealing with the ‘magic happens/leaps of faith’ that occur in logic models, log frames etc, using spheres of control/influence may help in thinking through complicated/complex change and Lori’s challenge of where to locate the ToC. I have used this with public sector managers in the context of partnerships and I have had very positive reactions and usually big sighs of relief. I am happy to provide more information on this if you are interested.
My final comment comes from my experience in facilitating evaluative capability building. When people are coming together to build/create ToCs, think about how it is facilitated in order to build a shared understanding of the issues/challenges/strengths/evidence/aspirations and the pathways for change and using the opportunity to strengthen group relationships, whether that is with teams, organisations or partnerships.