Authors: Agnes Becker, Annie Holmes, Becky Wolfe

At the London School of Hygiene & Tropical Medicine we have a proud history of improving health worldwide. However, we are increasingly being asked to demonstrate the impact of research, either as part of the Research Excellence Framework (REF) or as a funder requirement.

What can we do to better demonstrate the non-academic impact of our research?

On 22 September 2015, we held two sessions aiming to answer this question as part of the School’s annual symposium, attended by staff from all over the world:

• A panel session to highlight the importance of demonstrating impact.
• A workshop highlighting tools the School could use to demonstrate impact.

The result of the two sessions is a set of recommendations to help us better demonstrate impact at the School.

What is “impact”?

One challenge is the different definitions of research impact. Two relevant definitions come from the REF (where researchers were asked to prepare case studies demonstrating the impact of their research as part of the national audit of research quality which influences government funding to universities for research) and from the UK Department for International Development (DFID):

• REF: ‘Impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ (HEFCE et al 2012a)
• DFID: Impact is ‘the demonstrable contribution that excellent research makes to society and the economy’ (based on HM Treasury’s definition, which operates across UK government)

Panel: Fame, funding and the REF

The REF has brought the question of rigorous evidence of impact to everyone’s attention – what did we learn from the process? – and do funders require the same level of evidence for impact?

Bringing together researchers and funders the panel session, introduced by Professor Anne Mills, Deputy Director and Provost of the School, highlighted challenges in demonstrating impact and some suggestions on how to address them.

Top tips from researchers and learning from the REF

Dr Cathy Zimmerman of our Gender Violence & Health Centre presented on a draft paper, entitled “Rigged or rigorous? The role of non-academic partners”. This suggested:

• Plan for impact from the start but be flexible for opportunities
• Achieve impact by collaborating with partners – local groups and NGOs – to shape research questions so the evidence is useful to them
• Be bold about your findings
• Download Cathy Zimmerman’s presentation

Hilary Hunter, Planning and Governance Manager, presented some of the lessons learned from coordinating case studies for REF 2014:

• Identify potential impact case studies early by raising awareness in the School
• Collect evidence for impact during research projects, rather than retrospectively, which will help overcome the frequent challenge during REF 2014 of finding strong links between research and impact
• Tell a story with a strong narrative

Alison Grant, Professor of International Health, presented on lessons learned from submitting a REF case study for 2014. Her top tip:

• When writing a REF case study, start at the end and work backwards. Identify the right impact at the start – e.g. in the right time period – then construct the narrative.

Funders and demonstrating impact

The panel were joined by speakers representing two of the School’s major funders each with different requirements for demonstrating impact: The Wellcome Trust and the Department for International Development (DFID).

Kevin Dolby, Evaluation Advisor at The Wellcome Trust, said there are no specific requirements for Wellcome Trust grantees to demonstrate future impact at the time of application and this allows them to take risks. Wellcome have a broad definition of research impact, which includes academic impact as well as non-academic impact, and recognise that impact takes a long time, e.g. up to 20 years down the line. However, case studies are the only effective way to tell the stories of impact of their grants. Kevin spoke about how Wellcome are looking for ways to reduce the burden of reporting impact for researchers by:

• Investigating how REF case studies can be re-used  to further demonstrate The Wellcome Trust’s impact (although the last REF did not require impact case studies to name their funder, which was a shame)
• Looking at how existing research information metadata can help track impact at Wellcome and beyond
• Download Kevin Dolby’s presentation

Nathanael Bevan, Evaluation Advisor at DFID, spoke about why demonstrating impact is so important to DFID: “DFID spends £350 million of taxpayers money on research funding…we need to demonstrate it is reducing poverty not only to the taxpayers but also to the countries we work in.” However, DFID recognise that impact is not linear and advocates using a tool called the theory of change which takes into account different types of impact, e.g. contributing to a body of knowledge or changing attitudes. Using this approach DFID aim to pull together a portfolio of impact evidence with short term individual impacts and a longer term collective impact. Nathanael gave a few tips for researchers to consider when demonstrating impact:

• Think about impact from the start (See DFID’s research uptake guidance: https://www.gov.uk/government/publications/research-uptake-guidance)
• Build relationships with partners from the start to tailor research questions to their needs
• Publications, citations and downloads are not enough to tell an impact story, qualitative evidence is also needed (DFID are considering using a grant to an independent researcher to collect qualitative data to verify claims of grantees)
• Tell DFID about impact stories regularly to build into their portfolio of impact

Challenges

A number of challenges were highlighted by several participants, including:

• The difficulty of attributing impact to one institution or funder when research is a collaborative endeavour
• Using a narrative to demonstrate impact in case studies can be reductionist and linear – many projects have more than one impact achieved through multiple routes.
• How do you demonstrate the impact of negative findings? Will we start to skew how we demonstrate impact so only positives are shown?
• Topics such as clinical trials can tell a linear narrative story more easily than other topics such as health systems strengthening.
• Impact takes a long time to achieve, particularly for lab research.

Next steps

The School needs to do more to support staff in demonstrating impact, although this shouldn’t become the sole driver for research. Communications staff and academics need to work together to create a central resource on how best to demonstrate impact for REF, funders and to showcase the work of the School. The School’s Senior Leadership Team have recognised this need and will be working towards developing support for staff.

• Read the blog about the workshop on tools the School could use to demonstrate impact.