* * * * * * *

I’ve been spending a lot of time recently reading and thinking about evaluation and its role in research projects, public engagement with research and in achieving research impact.  As an impact ‘professional’, it is important that I give the academics I work with the most useful, practical and accurate advice possible so that they can deliver impact through their work.

In my opinion, there is insufficient emphasis on the importance of good evaluation practice in

·         running a good research project,

·         undertaking valuable and relevant public engagement with research, and

·         demonstrating the results of that engagement. 

This is particularly evident recently in the growing recent emphasis on public engagement as a response to the UK’s impact agenda.

·         Research funders, including the research councils and universities are running seminars and conferences for sharing good practice in public engagement

·         The NCCPE (National Co-ordinating Centre for Public Engagement) has continued its seeming inexorable rise to prominence, and

·         RCUK is launching a Catalyst for Public Engagement scheme to follow on from the soon-completed Beacons for Public Engagement, seeking to embed public engagement within institutions.

Certainly, increasing the quantity and quality of public engagement with research at HEIs has to be good, but it is a secondary outcome of the impact agenda. This renewed emphasis on public engagement with research distracts from the true purpose and meaning of impact: public engagement can be regarded as a precursor step towards impact - and certainly necessary if we take the meaning of public engagement in its broadest sense.

But impact is about more than public engagement; it is about the results of that engagement. Those results can only be elucidated with well-planned and effective evaluation. Evaluation is arguably therefore the most important ‑ but often the most neglected ‑ piece of the impact puzzle.

There is much material freely available on the web on the principles and practices of evaluation, so I won’t go into any of it here. But I want to highlight the value of sharing practice and concerns between disciplines and even sectors for effective impact evaluation.  Museums, galleries and other cultural institutions have much experience in evaluating the wider public value of their exhibitions and outreach activities. Arts and humanities researchers can learn much about evaluating impact from museums and galleries, not least because these kinds of organisations are often key partners for academics in their public engagement work. 

Much can also be learned from the social sciences in terms of methodologies and methods for analysing qualitative data. One argument often coming from the anti-impact camp in the arts and humanities is that our impacts are not easily quantifiable. Nevertheless, there are ways of turning qualitative feedback into the kinds of figures that satisfy funders and stakeholders.

This is why the focus from funders on public engagement with research fails to fully respond to the impact agenda. Funders should require that the UK’s impact plans, the “Pathways to Impact”, incorporate not only plans for engaging the public but also plans for evaluating the project.  This means that Funders have a duty to suitably fund the costs of carrying out a proper evaluation. And central to any effective self-evaluation is the question: how will you know you’ve been successful?

A basic rule of evaluation is that often when time and money are short, evaluation is the first thing to go. While researchers can learn the basics of evaluation themselves, this extra time imposition on already overburdened academics can make evaluation unappealing and practically impossible. My advice to academics is to delegate – if you want to give out questionnaires at a public engagement event – pay an undergrad to hand them out and collect them at the end. Don’t expect to be everywhere and everything.

Admittedly, this is meant to be a simplistic example: handing out questionnaires is better than doing nothing! But this example highlights the need to be realistic and practical when dealing with the various components of impact, public engagement and evaluation. What evaluations of public engagement events really cover is outcomes – the immediate short term results of a one-off engagement activity.

Fully assessing the wider public impacts of research requires long-term evaluation studies beyond many project teams’ budgets and time constraints. But distinguishing outcomes and impact is often not made clear in funders’ guidelines regarding what constitutes impact. Often outcomes are being used as proxies for impacts (for the obvious reasons): to take two examples from the REF main panel D guidelines on impact:

·         Informing policy discussion through a joint seminars with policymakers

·         Influencing the design of the school curriculum

I contend we should accept this interpretation, ‘settle’ for evaluating outcomes, and stop pretending that we’re talking about impacts.  In a way, you may think that this reinforces the argument of the anti-impact lobby, which is particularly strong in the arts and humanities. I do not mean to devalue either public engagement or evaluating outcomes: rather, I feel that all of us– researchers, administrators and policymakers – can benefit by being more realistic and honest in distinguishing long-term objectives, and what can be measured in the shorter timescales dictated by project time and budget constraints.

My argument is that thinking about it in these terms – accepting the practical limits on evaluating impacts, and realising that we are measuring and evaluating outcomes not impacts – should lessen researchers’ resistance to impact and encourage a wider recognize of how researchers may personally and professionally benefit while contributing to society more generally. But the benefits of engagement both for researchers and their public audiences will not be demonstrable without a well-written and executed evaluation plan.

Nadine Lewycky, Warwick, UK, 9^th November 2011.

Photo appears courtesy of Jordanhill School D&T Department.