The tricky business of research impact evaluation

Posted on December 12, 2012 by

0


Stroke AssociationAs both government and charity funding sources are squeezed by the recession, it’s becoming increasingly important for those engaging in medical research to demonstrate the value of their work.

In the higher education sector, the new Research Excellence Framework (REF) requires institutions to demonstrate the wider impact of their research on diverse areas such as public policy, healthcare services and quality of life. As medical research charities, we are not exempt; in fact demonstrating value for money to our supporters should be considered a vital part of our work.

Anyone who has tried to tackle the issue of research impact will know it is no easy task. This year the Stroke Association undertook an evaluation of its last 20 years of research funding. Our portfolio totalled more than £40 million across almost 500 grants and fellowships. Here I share a few of the challenges we faced and suggest some points to consider when conducting an impact evaluation of your own.

Choose impacts that matter to your audience

There is no best way to go about looking at research impact. Before you embark on such a project you should consider your target audience and what kinds of impacts they will care about. These answers will shape the ways you conduct your evaluation.

At the Stroke Association we put the stroke survivor at the heart of everything that we do so we were looking to show impact that they would find meaningful. This doesn’t mean the number of publications or citations our researchers produce, but real tangible differences in stroke care and services. We collected case study examples to demonstrate impact across the whole stroke care pathway and used patient and researcher stories to bring these impacts to life.

Consider broad impacts

A primary aim of medical research is to advance patient care but it can also have positive effects in other sectors. Make sure you consider broad impacts to realise the full value of your funding.

In addition to improvements in stroke prevention and treatments, we looked at:

  • The role of our funding in strengthening the stroke research community, for example by bringing and retaining talented individuals in the field, by creating new collaborations and by developing new research infrastructure.
  • The amount of follow-on funding our researchers attracted from elsewhere.
  • Our impact on the economy through commercial development or reduced health service costs.
  • Any changes to public policy or medical education that resulted from our research.

The path from research to impact is rarely linear

The collaborative nature of science makes it very difficult to attribute specific impacts to a piece of research. This is particularly true of basic research that tends to contribute to the accumulation of knowledge in a given field rather than directly advancing medical care.

During the evaluation process it helps to acknowledge the contributions that a piece of research has made rather than looking for direct cause and effect.

Delve deep into your archives

The translation of research evidence into clinical practice can take a very long time. This means that the full impact is often only realised several years or decades after the research was conducted.

To find your most significant outcomes you really need to delve back into your archives. The REF guidelines suggest a period of 15 years is appropriate for evaluating the impact of medical research.

Starting with a complete set of grant records and current contacts for your past research alumni will make this process much easier.

Boosting researcher compliance

Researchers are the most likely people to be the aware of their research outcomes so gathering their opinions should be a vital part of any evaluation. However, getting them to comply with your information requests can be difficult.

Try to make the process beneficial for them, i.e. by offering help with media and publicity if their research is successful. Alternatively you could make impact reporting a compulsory part of the grant contract.

Objectivity

Researchers may have a subjective take on the importance of their research and corroborating its role is a given advancement can be difficult. Consulting impartial opinion leaders is always a good idea, especially those involved in developing clinical guidelines.

Numbers matter

Although we decided that publication and citation numbers were not that important to our target audience, we still wanted to include quantitative impacts where possible. This generally involved searching through published statistics and audit data to find relevant numbers. For example, using stroke audit records from before and after a piece of research was conducted to demonstrate the scale of its impact.

Communicating the results

Once you’ve conducted your evaluation, you will want to consider the best ways to communicate the findings. This will again depend on the main objectives of your evaluation and your target audience.

Our main aim was to raise the public profile of our stroke research and to stimulate further donations so we opted for a magazine-style report. It was written for a lay audience using lots of graphics and accessible yet emotive language. This document will be shared with several of our stakeholder groups but it is important to realise that one document is unlikely to meet all the needs of the organisation. For example, we intend to produce a second more quantitative report for our Trustees and Research Advisory Committees to inform our future research strategy.

Our new report ‘Saving Lives: 20 years of investing in vital stroke research’ can be downloaded from our website. If you have any questions about the report or our research impact evaluation, please email Clare Walton.

Posted in: Research