On 13 June UCL kindly hosted a welcome opportunity for UPEN members to discuss policy brokerage with Professor Sir Peter Gluckman, Professor Sir Ian Boyd, Professor Robin Grimes and Dr Kathryn Oliver. Several participants touched on evidence synthesis – a potentially neutral way for researchers and academics to act as brokers to help policymakers understand evidence.
All types of evidence synthesis try to describe objectively what we know, how sure we are, and where we (dis)agree. There is increasing interest from the science and policy communities on how to promote the creation and use of best-practice evidence synthesis activities, in whatever form they might take, structured or unstructured, qualitative or quantitative.
In February 2019 the latest in the series of Oxford Martin Restatements, A restatement of the natural science evidence base on the effects of endocrine disrupting chemicals on wildlife, was published in the Proceedings of the Royal Society B.
These reports contain no new information, contain no surprises for practitioners working in the field, and do not attempt to answer questions about policy choices or resolve scientific debates. Yet Oxford Martin Restatements have been requested from senior members of the Civil Service, recognised by the Government Chief Scientific Advisor and cited by Ministers in Parliament.
The many faces of evidence synthesis
Restatements take their name from a tradition in law: legal restatements are sets of statements, compiled with input from law professors, attorneys and judges, which reflect consensus on an area of law, supported by “reporters’ notes” – detailed discussions of all the cases that went into the principle summarised in a particular section. Scientific restatements take much the same structure: they consist of a list of numbered and graded statements, backed up by an extensive annotated bibliography detailing the sources of the evidence that support those particular statements.
Oxford Martin Restatements are one of many different forms of evidence synthesis. Approaches to evidence synthesis can be highly structured, like the formal systematic reviews commonly used in the health sciences and which often review randomised control trials, the ‘gold standard’ evidence for evaluating the effectiveness of an intervention. Such systematic reviews have a strong emphasis on data (often including quantitative analysis of that data – a meta-analysis); on sometimes-exhaustive coverage of the literature; and on the reproducibility of their results via highly standardised methodologies. Evidence synthesis approaches can also take a more narrative, informal or unstructured form. Narrative reviews tend to rely more heavily on expert opinion, on an interpretative or discursive approach to the evidence base. They can range from the traditional single-expert reflection on a topic, to more systematic qualitative approaches such as meta-ethnographies.
All types of evidence synthesis try to describe objectively what we know, how sure we are, and where we (dis)agree. There is increasing interest from the science and policy communities on how to promote the creation and use of best-practice evidence synthesis activities, in whatever form they might take, structured or unstructured, qualitative or quantitative.
Capturing the expertise, grading the evidence
On a continuum between informal and formal, Oxford Martin Restatements fall somewhere in between. We use a mixed-methods approach that attempts to represent the science evidence base underlying areas of policy concern or controversy by taking the accumulated expertise of a group of well-respected scientists in the field, representing a range of views, and distilling this expert knowledge through a semi-structured and iterative process. We assess the peer-reviewed literature, and simultaneously make efforts to understand the nuances of a thorny subject by engaging an expert group of people with dialogue and deliberation.
Once we have assembled the restatement and annotated bibliography in league with our author experts, we undertake a consultation process with a large number of stakeholders (from academia, practice, and policy), and submit the paper for traditional peer-review for publication in an open-source journal. This ‘thrice-over’ process of review and referral results in what we believe is an impartial, plain language, and richly detailed snapshot of the state of knowledge of the field.
We also ask our experts to grade the evidence behind each statement in the paper. We design a system of grading appropriate for each topic: for instance, the endocrine disrupting chemicals restatement distinguishes between background material like basic chemistry; a substantial evidence base where further information is unlikely to change the current consensus; an evidence base which supports the statement less strongly and where further information might alter current consensus; and expert opinion based on information from related substances or general principles. This allows generalist readers to understand rapidly the depth and quality of the underlying evidence.
As an example, it is highly likely that a class of chemical known as PCBs are behind the severe decline in reproductive capacities of some killer whale populations. Here, the evidence is strong. In situations whereby one examines the multi-factorial effect of a cocktail of chemicals, the effect on wildlife populations is much less well characterised – indeed, such an effect may not exist at all, but the data is simply not known, and very difficult to obtain.
The distillation and grading process, whilst laborious, frequently gets to the nub(s) of a contentious issue: why do scientists disagree? Is it because of uncertainty in existing data, or unavailability of data in the first place? Is it because of a conflicting framework of assumptions, or a different set of values? Is it because of cherry-picking data to fit an a priori argument?
Being an honest broker
Academics who advise policymakers, or who generate material aimed to be read and used by them, can (and should) divide their activity into that which is ‘science advocacy’ and that which is ‘honest brokerage’ (Pielke, 2007). For instance, many Oxford Martin School policy papers clearly make the case for a particular course of policy action based on expert interpretation of the underlying evidence. In writing an Oxford Martin Restatement, we aim, as far as is possible, to be policy-neutral: we act as an ‘honest broker’ to help policymakers understand the evidence without discussing implications for policy. Both ‘science advocate’ and ‘honest broker’ activities are important and valuable – but it is vital to be transparent about which activity is on offer.
Why are honest broker activities so important? The policy world is often characterised by fast-paced decisions, limited resources, high staff turnover and limited corporate memory. In that world, having pre-prepared, policy neutral evidence statements, ideally funded by independent and arms-length bodies, can keep decision-makers informed on the true state of the evidence base. This is just one of the tools that the policymaker will use in her decision-making, but it is a powerful one.
The accurate synthesis of existing information is the most important single offering by academics to the policy process
– Professor Chris Whitty, CSA, DfID/DoH
Honest broking is particularly valuable when people have strong views on a topic, and where there can be multiple and sometimes vocal advocacy groups involved. Sometimes those views can collide across evidence bases that are complex (like low-level ionising radiation) or sparse (like endocrine-disrupting chemicals). Such evidence bases are imperfect, and often they are chronically so. What the Oxford Martin Restatements projects aim to do is bring some delineation to where there is good evidence and where there isn’t. We identify complex scientific questions, without trying to answer those questions. Altogether, we try to help the policymaker make complex policy decisions about sensitive issues, in the face of imperfect evidence.
Further information
The restatements series was founded by Professor Sir Charles Godfray and Professor Dame Angela Mclean. The series is managed by Dr Ella Adlen. To date, the topics have covered the effects of endocrine-disrupting chemicals on wildlife, the impact of low dose ionising radiation on human health, the effectiveness of ‘natural’ flood management approaches, the effects of neonicotinoid pesticides on pollinators, and the control of bovine tuberculosis. Ongoing restatement project topics include the impact of grazing management on soil carbon sequestration, interventions to control campylobacter in chickens, and the link between antibiotic use in agriculture and antimicrobial resistance in human infections. For further information contact ella.adlen@oxfordmartin.ox.ac.uk.
– Find out more about Oxford Martin Restatements
– Find out about The Royal Society’s Evidence Synthesis programme
– Watch Video: ‘Why science matters to policy‘ with UK Government Chief Scientific Adviser Dr Patrick Vallance
An earlier version of this blog was published on oxfordmartin.ox.ac.uk.
Dr Ella Adlen is Research and Programmes Manager at the Oxford Martin School, University of Oxford.
Author: Dr Ella Adlen, Research and Programmes Manager at the Oxford Martin School, University of Oxford|
