Evaluations of Policy Fellowships demonstrate that they are an increasingly recognised mechanism for supporting mobility and knowledge exchange between researchers and public policy organisations – but how and what do we evaluate? That’s the question UCL Public Policy set out to explore in a recent workshop at the UPEN Annual conference: ‘Creating Feasible and Realistic Evaluation Frameworks to Measure Success.’
Why this, why now?
Co-designed with University of Durham and Nottingham Civic Exchange, and delivered by Alice Tofts, Sinéad Murphy and Olivia Stevenson from the UCL Public Policy team, the workshop design drew on our knowledge and experience of designing and undertaking evaluations for programmes at scale, such as CAPE and UKRI Policy Fellowships.
But the aim wasn’t to replicate those larger-scale models; instead, the goal was to develop an evaluation approach that is proportionate, adaptable and works for organisations of different sizes, especially those without dedicated evaluation provision.
The workshop focused on three practical challenges:
- identifying the stakeholders who are essential to evaluating policy fellowships,
- clarifying what success looks like from different perspectives,
- selecting suitable and low resource-intensive methodological tools and outputs, without compromising rigour.
A Four-Step, Building Block Framework
Participants were introduced to a Building Block Framework that breaks evaluation down into four manageable stages: problem statement (to understand what your stakeholders want to learn about the fellowship); success Indicators (defining what success looks like from their perspective); methodology: (selecting efficient and effective ways to collect the right data); and outputs: (communicating findings in ways that are meaningful to each audience).
What We Heard in the Room
With 14 participants, from across government, think tanks and universities – some of whom were on fellowships, others delivering programmes – we found that the framework sparked valuable conversations. Indeed, as one participant commented that for fellowships its ‘easy to count the cost, but hard to show the value’.
One takeaway was that positionality and institutional context significantly shape how fellowship impact is evaluated. Fellows, host organisations, funders, and universities emphasise different outcomes – ranging from individual learning and career development, to influencing live policy, to driving long-term systemic change. Evaluating that range of outcomes is inherently complex, particularly when moving beyond anecdote to understand broader shifts in organisational culture and evidence use.
Another takeaway was that there is a growing interest in system-level approaches to evaluation, including cohort analysis, cost-benefit comparisons and ethnographic methods. These strategies offer the potential for a more formalised, evidence-based understanding of how fellowships interact with and impact policymaking systems.
Participants also observed that engaging with the building blocks became a facilitation tool, enabling dialogue around academic-policy engagement practices and challenges that ranged beyond evaluation alone. Some also commented on the value of the visual style of the model, highlighting the importance of developing evaluation models which aim to be inclusive to a variety of users.
What’s next?
In future iterations, the framework could be strengthened by integrating additional factors such as budget constraints and project timescales. It would also benefit from regular check-ins against the original goals, problem statement, and success criteria to ensure ongoing alignment throughout the evaluation process.
If you would like a copy of the Building Block Framework, please get in touch with Alice Tofts (alice.tofts@ucl.ac.uk)
Take a look at CAPE’s review of their fellowships: ‘Research to Policy Fellowships: Six Pillars for Optimising Success’
