Authors: Agnes Becker, Annie Holmes, Becky Wolfe

Working in academia, we are increasingly being asked to demonstrate the impact of research, either as part of the Research Excellence Framework (REF) or as a funder requirement. But do we have the skills to meet these requirements?

On 22 September 2015, we held a workshop highlighting tools the London School of Hygiene & Tropical Medicine could use to demonstrating impact and resulting in a set of recommendations to help us better demonstrate impact at the School.

Workshop: Toolbox and genius bar

The workshop followed a panel session learning from REF 2014 and from the Wellcome Trust and DFID on how to better demonstrate impact  and highlighting some of the challenges faced when demonstrating impact. Focussing on practical solutions to address some of these challenges at the School, the workshop invited experts to showcase tools to plan, track and describe the impact of research.

Planning for impact: Theory of change tool

Alexandra Chitty, Research Uptake Officer at the Sanitation and Hygiene Applied Research for Equity (SHARE) Consortium based at the School, spoke about SHARE’s use of a theory of change and outcome mapping to guide its work and demonstrate its impact.

Advantages: The theory of change tool

• is a participatory process that requires your whole team to think about the change you want to see as a result of your research, how it will be reached and what will be used to measure progress along the way
• allows you to plan for realistic, plausible pathways to multiple impacts at the start of your project
• provides guidance on the evidence needed to demonstrate impact

SHARE’s theory of change (and resulting outcome mapping exercises) helped SHARE to ensure its research influences improvements to sanitation and hygiene policies and practices around the world. One example was the Inclusive Water, Sanitation and Hygiene (WASH) project which shed light on the barriers that disabled, older and chronically ill people in Uganda and Zambia face when accessing WASH, and designed and tested an approach that would address these. Through various activities and with the efforts of multiple actors, the project informed DFID’s Disability Framework. Read Alexandra Chitty’s blog on using a Theory of Change to plan for impact.

Limitations: The theory of change tool

• is subjective as it only reflect views of those in room when it is formulated – though regular review is recommended
• has limits to what can be achieved through its participatory planning process for large projects due to the diversity of views
• does not capture the opportunistic nature of influencing policy
• does not allow you to plan for many different sets of pathways to impact.
• Download the Theory of Change presentation

Tracking impact: RAPID Outcome Mapping Approach tool

Jessica MacKenzie, Research Fellow with the Research and Policy in Development (RAPID) programme at the Overseas Development Institute, used a climate change project example to show how to use the Outcome Mapping Approach.

Advantages: Outcome Mapping Approach

• is a manageable, step-by-step approach
• allows you to identify stakeholders whose behaviour you want to change to achieve impact
• encourages you to suggest behavioural changes you’d expect, like and love to see in your stakeholders
• allows you to use these changes to set progress markers for tracking and monitoring
• encourages you to course-correct based on progress markers

The climate change project used this tool to map behavioural changes (progress markers) and were able to show an increase in the percentage of these markers being achieved over time. By 2015 the project had managed to meet 60% of it’s expect to see progress markers, 40% of its like to see markers and 10% of its love to see markers. In achieving these indicators, the project helping to improve the capacity of climate negotiators from the world’s poorest countries to better negotiate at the United Nations Framework – Convention on Climate Change.

Limitations: Outcome Mapping Approach

• takes time at the start of the project to agree your stakeholders, behavioural changes and progress markers with your team
• Download the RAPID Outcome Mapping Approach presentation

Tracking impact: Altmetrics tool

Katherine Christian and Natalia Madjarevic from London-based startup Altmetrics spoke about using online metrics to track impact. Alternative metrics, or altmetrics, are indicators of impact and engagement with research that extend beyond traditional methods of influence measurement, such as citations. They include citations in news outlets, social media, reference managers, policy documents and post-publication peer review, as well as other sources such as Wikipedia and youtube.

Advantages: Altmetrics

• allow you to track your research in real-time
• support expert judgement, such as peer review, for grant applications and CVs
• can help to identify potential impact case studies
• can be used as evidence for impact case studies, along with qualitative data, particularly to show research is informing and provoking debate

Limitations: Altmetrics

• cannot prove impact
• cannot demonstrate high quality research
• Download the Altmetrics presentation

Describing impact: Research Impact Framework tool

Shirine Voller, IDEAS Project Manager at the London School of Hygiene & Tropical Medicine, described how she used this tool to describe the impact maternal and newborn health projects had had in Ethiopia, Nigeria and India.

Advantages: Research Impact Framework

• is a standardised framework
• is light touch
• is a way of identifying case studies of what leads to impact

Limitations: Research Impact Framework

• evidence is anecdotal, self-reported and not verified
• Download the Research Impact Framework tool presentation

Describing impact: Using the School’s Press Office

Katie Steels, Media Manager at the School, used an example of a twins experiment that revealed a genetic link with mosquito bites to show how the Press Office can be used to plan for, track and describe impact. Getting in touch with the Press Office as soon as you submit a paper will help them to plan a media strategy, such as writing a press release, coming up with a targeted list of journalists, and putting together a social media plan. By registering yourself as part of the Press Office’s expert database you may be asked to give reactive comments on current health stories in the news. Katie and her team will put together a media coverage and impact report for you to use as evidence of impact.

• Download the Press Office support presentation

Recommendations and next steps

The School’s Senior Leadership Team have recognised there needs to be more support for staff on how to demonstrate impact. The recommendations that came out of the workshop could help to develop this support. They centred mostly on ways to change the culture of the School so that demonstrating impact is automatically  part of every research project as well as collating a central repository of tools staff can use.