The need for robust evidence to inform policy has never been greater, as countries around the world grapple with issues of unprecedented complexity. However, it is rare to find individual studies that conclusively resolve a major knowledge gap or controversy, whose findings are consistently reproduced by others. Instead, knowledge tends to accumulate incrementally via successive studies using different methods in different contexts, often leading to apparently contradictory findings. This can make it difficult to identify clear, evidence-informed policy options.
A new approach
Evidence synthesis methods have been developed to resolve these tensions, showing where there is robust evidence across multiple studies and context to inform policy and practice, and where more research is needed, if the evidence is mixed or inconclusive. However, few researchers have the skills to conduct robust evidence synthesis, and there are few post-graduate programmes that provide this training to early career researchers.
For this reason, Policy Leeds and N8 AgriFood teamed up with evidence synthesis methodologist, Dr Gavin Stewart and Professor Mark Reed, a visiting Professor at Leeds and N8 AgriFood Chair at the time (now at SRUC), to deliver a training programme funded by N8 AgriFood and Research England (QR-SPF).
This is the process:
- Identify evidence needs from policy colleagues in a thematic area (food and farming, in our case, via teams in Defra, Natural England, Environment Agency and Food Standards Agency in England and equivalent departments and agencies in devolved administrations)
- Offer training to early career researchers in rapid evidence synthesis and writing policy briefs
- Support trainees to write paper and policy brief
- Policy colleagues get evidence and early career researchers gain new skills and potentially publications.
One study calculated that a typical systematic review costs about Ł100,000 and takes between 6 months and 16 months assuming five co-authors devote 10–20 hours per week to the review, and another estimated 1–2 years for to complete a full systematic review. In contrast to this, despite a number of authors dropping out due to challenges posed by the pandemic, our whole programme worked out at Ł2336 per review, and in the case of the cohort who were able to attend the residential training, the majority completed a first draft of rapid reviews in one week. This included the training costs, the residential element and open access fees for two of articles, but did not include project management costs for N8 AgriFood.
Does it work?
Overall, despite the challenges posed by the pandemic to this programme of work, it was possible to provide a cohort of early career researchers with skills in evidence synthesis that they will be able to use elsewhere in their careers. The fact that the programme led to the publication of eleven evidence syntheses and/or policy briefs addressing evidence gaps identified by the policy community, for such limited resources, represents remarkably good value for money. The potential speed with which this can deliver outputs to the policy community is also important, given that the programme was designed to be completed within three months from identification of policy questions through to production of papers and briefings. Although many participants didn’t produce outputs, they did benefit from the training and skills development (some conducted their review but just didn’t produce a final synthesis paper or briefing). See for yourself by reading the outputs here
Mark Reed is Professor of Rural Entrepreneurship and Director of the Thriving Natural Capital Challenge Centre at Scotland’s Rural College, Research Lead for the International Union for the Conservation of Nature’s UK Peatland Programme, Co-Chair of UNEP’s Global Peatland Initiative Research Working Group, a Commissioner on Scottish Government’s Just Transition Commission and serves on multiple committees and boards for UK governments.
Dr Gavin Stewart is a Senior Lecturer in Evidence Synthesis at Newcastle University. His research focuses on applying evidence-based methods to trans-disciplinary problems relating to sustainable development and food security. He also applies these methods to healthcare, environmental and social science domains. He is an elected member of the Society for Research Synthesis Methods and associate editor of its journal, Research Synthesis Methods. He is also the co-chair and senior editor of the Campbell Collaboration food security group, associate editor for PeerJ, and statistical editor for the Cochrane Pain and Palliative Care Group.
Anthonia James has over 20 years’ experience in research programme management, with a specific emphasis on policy influence and impact. During her career she has managed multiple research projects and 8 research programmes, totalling Ł30m. She began her career managing a research network, building the capacity of primary care researchers across the UK Yorkshire region. She then spent over a decade at the Nuffield Centre for International Health and Development at the University of Leeds working in global health research and international development, focusing on communicable diseases and health care delivery. In 2021, she became Operations Director of the FixOurFood programme, a UKRI Transforming UK Food Systems grant, which aims to transform the Yorkshire food system.
Dr Ged Hall has 20 years of experience at senior levels in the UK HE sector. For the last decade this has focused on research impact. This has included organisational development projects (reporting to University Executive level) and professional development for individual researchers and groups (covering all career stages). Previously he managed researcher development for the University of Nottingham (postgraduate researchers and research staff), which included setting up academic provision in international markets.
