What can those interested in policy engagement learn from the What Works Network’s approach to impact? We summarise four lessons in planning, generating and evaluating impact that we hope are useful for all organisations that broker evidence into policy.
The Wales Centre for Public Policy at Cardiff University received funding from the Economic and Social Research Council (ESRC) to conduct a project examining the impact of the What Works Network. This Network brings together What Works Centres (WWCs) that generate, synthesise and broker evidence to audiences in different policy areas such as homelessness (Centre for Homelessness Impact), health care (National Institute for Health and Care Excellence (NICE)), and educational attainment (Education Endowment Foundation).
WWC outputs include evidence syntheses, toolkits comparing different types of interventions, evidence reviews, implementation support and events which bring together key players on a topic. Their audiences include decision-makers in central, devolved, regional and local governments, as well as practitioners such as teachers, police officers and health care professionals. The crossovers with the audiences and outputs that other policy engagement bodies are trying to influence are manifold. We summarise four lessons that we think are of particular interest.
Lesson 1: Having impact in the field of knowledge brokering is hard
Significant efforts have been made trying to conceptualise and empirically demonstrate the societal impact of research, with many studies questioning how far one can attribute changes in society back to specific research projects. The task is even harder for demonstrating the impact of knowledge brokering which takes place at a boundary connecting research, evidence, knowledge producers and outputs with policy and practice stakeholders.
Impact is different from and goes beyond engagement. Many WWCs began focusing on engagement as it is often a route to developing impact. They have developed Theories of Change and logic models to help them plan for impact, asking themselves what impact they would like to have and working backwards to think about their activities and relationships. WWCs’ impact focuses as much on getting evidence into policy documents and practice, as it does on changing attitudes and behaviours towards an issue.
Lesson 2: Building trusting and in-depth relationships takes time
To understand impact within the Network, we conducted more than 40 interviews with WWCs and their stakeholders. All interviews revealed that building relationships was seen as a key pathway towards impact. One stakeholder explained how impact was intimately linked to relationship building:
“[I]t is about being there. It is about being visible. It is about being in the places where people are talking about these kinds of decisions in order to land it. It is not really about references. Name checks in Parliament are nice, but it is much more relational than that.”
Other stakeholders talked about the importance of working closely with WWCs and of it being an equal relationship. Many WWCs have now started working in partnerships with local partners such as local government. For instance, the Centre for Ageing Better has forged partnerships with Greater Manchester, Leeds and Lincolnshire which help build strong and trusted ties where evidence needs are discussed on a regular basis and outputs are co-produced to respond to those needs. As place is increasingly emerging as an important focus for impact (e.g., Local Policy Innovation Partnerships, Research England Development Fund), these deep local partnerships can constitute a promising route towards impact. Some universities, such as the Nottingham Civic Institute and the Heseltine Institute at Liverpool, have been working in this ‘place’ space for a while.
WWCs highlighted some key factors for successful relationships with decision-makers. These included providing thinking space and time for them to engage with the evidence rather than just read it, and working co-productively to produce outputs which can be used on the ground.
Lesson 3: Planning for impact can take different shapes
We organised an event in November 2024 bringing together WWCs to discuss how they plan, generate and measure impact. This event featured speakers from outside the Network to illustrate the similar and different approaches to this issue that centres could learn from.
There is no need to reinvent the wheel. There are plenty of ready-to-use models available in the literature and elsewhere on framing and thinking about your impact. Here is a link to a summary of approach that we prepared for WWCs. Could also add examples from the Shift and Share session.
Lesson 4: Evaluating and tracking impact can also take different shapes
Once you think you have had impact, it can be difficult to demonstrate it. This is especially the case for knowledge brokers whose work is often behind the scenes and remains hidden in the boundary between academia and policy. Most often, WWCs are working to evidence that they made a significant contribution to the impact, a situation which might be similar to other players working in the policy engagement space.
WWCs have developed different systems to capture their impact, from project management systems such as Asana which links project tasks to overall goals, to using Slack channels to allow anyone from the organisation to log examples of impact. There can often be long causal chains between research, evidence and final impact, and the need to relate your organisation’s work to change is a long process which needs careful evaluation.
Centres have sought to avoid falling in the evaluation by metrics ‘trap’ which might mean sacrificing the focus on systems change to rely on simpler metrics (e.g., number of downloads) and linear impact. WWCs highlighted the difficulty of measuring nuanced impact, or what could be referred to as informal/conceptual impact such as attitude and behaviour changes. The use of participatory mapping workshops is a potentially useful tool to identify as a group impacts sought and then examine how the impact was achieved.
Conclusion: It is important to share good practice and approaches to measuring impact
Our project presents an opportunity for WWCs to share examples of good practice and approaches to impact. WWCs noted that it was important to share what they are doing with each other and the wider world so that others could take inspiration and, in some cases, avoid the same pitfalls. It is also important for WWCs to learn from other organisations such as university policy engagement bodies and our project will aim to foster closer links between the two communities. If you would be interested in hearing more about this project or being part of a future event with WWCs, please do get in touch with us.
